ODI JDBC EXCEL

BY ODI EXPERTS

V1.0 DOCUMENTATION

CONTENTS READING AND WRITING EXCEL WITHOUT ODBC ..................................................... 2 LIMITATION OF THIS DRIVER ............................................................................................ 2 PRELIMINARY STEPS .............................................................................................................. 3 REVERSING THE EXCEL FILE (RKM)................................................................................ 6 SINGLE SHEET HEADER PRESENT AT 5TH ROW ................................................... 7 MULTIPLE SHEETS ............................................................................................................... 9 SELECTIVE MULTIPLE SHEETS ................................................................................... 10 LOADING FROM EXCEL TO DB (LKM) .......................................................................... 11 LOADING EXCEL FILE INTO DB ................................................................................... 11 HANDLING EMPTY ROWS .............................................................................................. 13 INTEGRATION INTO EXCEL (IKM) ................................................................................. 14 WRITING INTO 2003 (XLS)............................................................................................ 16 REWRITING INTO EXISTING EXCEL FILE ............................................................... 20 WRITING HUGE DATA INTO EXCEL FILE ................................................................ 22 LOADING DIRECTLY FROM FILE ................................................................................. 24 ODI EXCEL JDBC DATATYPE ............................................................................................. 25 CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE DATATYPE

SCALE LEVEL........................................................................................................................ 25 HANDLING DATE & DATETIME ................................................................................... 26 HANDLING CURRENCY_SYMBOLS .............................................................................. 27 API ................................................................................................................................................ 28 COMMON ERROR/ISSUES .................................................................................................. 31

READING AND WRITING EXCEL WITHOUT ODBC

For a long time, Reading and Writing Excel was a big challenge especially in the Non-windows environment. Today from ODIExperts we are providing a unique solution ODI JDBC EXCEL, which will enable you to Reverse, Load and Integrate (Read/Write) Excel XLS and XLSX files.

LIMITATION OF THIS DRIVER

In ODI Data / View Data on the data store will not work Excel (XLS) format only allows 65,536 Rows per Sheet and Excel (XLSX) format allows 1,048,576 rows per sheet. Although not a limitation but while writing Rows greater than 100,000 should go for XLSX format, as the process will be Faster and File created will be compressed and smaller in size. Excel 2003 (xls) lacks such capabilities.

PRELIMINARY STEPS
Extract the ODI Excel JDBC Zip file and you will find these files KM_IKM_EXCEL_INSERT_V1.xml KM_LKM_EXCEL_to_SQL_V1.xml KM_RKM_EXCEL_V1.xml ODI_JDBC_Excel_V1.jar TECH_ODI_JDBC_Excel.xml

Copy the ODI_JDBC_Excel_v1.jar under your ODI Agents Driver Folder and restart your Agent .For local, please copy in to your respective USERLIB Folder.

Next Import the ODI JDBC Excel Technology in Duplication Mode.

Create a DataServer with just an appropriate Name for it. No Need to provide any other details.

Create a Physical Schema and provide the Directory where the Excel Files are present. [ Note: - You can create n Number of Physical Schema under the same Data Server ]

 Finally create a Logical Schema and link them through the appropriate Context.

Now you are ready to use this Technology.

Login into Designer and Import the RKM, LKM and IKM respectively.

REVERSING THE EXCEL FILE (RKM)

The different options of the RKM are HEADER_ROW By default most of the excel have the first row as the header but there might be chance that header row can be from 3rd or Nth line in that case provide the appropriate Value to it. SHEET_NO If one of the Excel Sheet is to be reversed then specify the Sheet No. By default the Sheet No. To Reverse Multiple Sheets please use %. First sheet will be 1; Second sheet will be 2 and so on... Note: - If different Sheets have different Header Row then please reverse them individually. Empty sheets will not be ignored, while reversing. Model Setup

SINGLE SHEET HEADER PRESENT AT 5TH ROW

SAMPLE FILE

SELECT REVERSE ENGINEER CUSTOMIZED KNOWLEDGE MODULE RKM EXCEL_V1

MASK - << File Name with Extension>> SHEET_NO 1, Specify the sheet No. HEADER_ROW 5 [Row No ,where the header is present, as you can see in the above screenshot ] Once reversed the datastore will be similar as shown below,

 Name - << File Name with Extension >> Resource Name Excel File Name [Note Please dont modify anything in the First line of the description]

Data type is detected depending on the content.

MULTIPLE SHEETS
SAMPLE FILE

Provide the Options depending on your requirement

 MASK - << File Name with Extension>> SHEET_NO % HEADER_ROW 1 [NOTE: - The same header row Value will be used for all the Sheets. In case if different sheets have different values please reverse individually] As you can see from the below image, all the sheets are reversed.

SELECTIVE MULTIPLE SHEETS

In case you wish to reverse selective multiple sheets you can provide the Sheet-No separated by comma. A sample screenshot is shown below.

LOADING FROM EXCEL TO DB (LKM)

The different options of LKM are TRIM_COLUMNS: If true the columns will be trimmed while loading into C$ table else column will retain spaces as present in File. DELETE_TEMPORARY_OBJECTS: As present in all the KM options, this option is to retain C$. Restrictions Filters has to be only on the Staging Joins between two Sheets also have to be on Staging Any SQL Operation on source has to be performed on the Staging or Target side only.

LOADING EXCEL FILE INTO DB

The below shown sample data will be Inserted into the Target Table.

Map the column accordingly.

Note: - All the columns need to be mapped to staging only. Column mapped on Source is not supported and will throw error. Select the LKM Excel to SQL_V1 and appropriate IKM depending on your Target Technology.

Save and Execute and ones the Execution is successful. Please look into the Target Table for Data. Operator Session Execution of the above Interface

Data on the Target Table.

HANDLING EMPTY ROWS

The LKM automatically takes care of the empty rows while loading into the Target. As you can see from the below screenshot, the lines where the row is empty is not loaded into target

INTEGRATION INTO EXCEL (IKM)

This IKM is Multi Technology IKM and so does not create I$ table but load directly from Source or C$ table depending the Source. Enable the Option Staging Area Different from Target and select the appropriate Source Logical Schema.

The different options of the IKM are HEADER_ALIGNMENT: You can set the Header Text Alignment .The valid values are LEFT, RIGHT and CENTER. By default LEFT is used. HEADER_BOLD: If true the Header will be bold else Normal text. HEADER_COLOR: You can set the Header color here. By default the plum color is used. The valid color values are AQUA, AUTOMATIC, BLACK, BLUE, BLUE_GREY, BRIGHT_GREEN, BROWN, CORAL, CORNFLOWER_BLUE, DARK_BLUE, DARK_GREEN, DARK_RED, DARK_TEAL, DARK_YELLOW, GOLD, GREEN, GREY_25_PERCENT, GREY_40_PERCENT, GREY_50_PERCENT, GREY_80_PERCENT, INDIGO, LAVENDER, LEMON_CHIFFON, LIGHT_BLUE, LIGHT_CORNFLOWER_BLUE, LIGHT_GREEN, LIGHT_ORANGE, LIGHT_TURQUOISE, LIGHT_YELLOW, LIME, MAROON, OLIVE_GREEN, ORANGE, ORCHID, PALE_BLUE, PINK, PLUM, RED, ROSE, ROYAL_BLUE, SEA_GREEN, SKY_BLUE, TAN, TEAL, TURQUOISE, VIOLET, WHITE, YELLOW REWRITE_EXISTING_FILE: Enable this option if you want to rewrite existing Excel Sheet. WRITE_LARGE_XLSX: If you are loading huge data then enable this option as then JDBC driver uses a different algorithm to load faster and with less memory. DATE_FORMAT: Date format while writing into Excel.

d- Day m - Month y - Year Some of the Valid Format ----------------------------- m/d/yy d-mmm-yy d-mmm mmm-yy DATETIME_FORMAT: Date time format while writing into Excel d - Day m - Month y - Year s - Second h - Hour mm - Minutes Some of the Valid Format ------------------------------ h:mm AM/PM h:mm:ss AM/PM h:mm h:mm:ss m/d/yy h:mm

CURRENCY_SYMBOL: The currency symbol to be used if the datatype is Currency. The default currency is $.

WRITING INTO 2003 (XLS)

You will need to manually Create Target datastore with appropriate Column Name and Datatype. Data Store Name : <<Sheet Name>>. The Sheets created in Excel will have data Store Name. Resource Name : <<File Name with Extensions>> Make sure you provide the file with appropriate Extension xls / xlsx. Note: - Without Extension IKM will throw error.

Select the appropriate Datatype and dont worry about the logical length but make sure you select the appropriate Scale as Excel created will use that for creating required decimals data.

Map the Column accordingly.

Sample Source data to be loaded into Excel

Sample Target data for the above source data. Since my Scale for Salary and com_pct was 2, the decimals were added accordingly.

Now the Scale level is increased to 3

As you can see the Decimal are created with 3 Decimal points.

If Decimal points are not entered then data is create is with no Decimal Values. Make sure you provide the Options else Excel file will be created using default options.

REWRITING INTO EXISTING EXCEL FILE

The below example illustrates rewriting an existing File. The Driver will automatically create a new Sheet in the File and Start Writing into it. Note: - The Present driver cannot rewrite into an existing Sheet, but only create a new sheet in the existing File. Also rewriting does not work if WRITE_LARGE_XLSX is enabled This is a sample file with one Sheet .Now we will Insert data into this File.

Create a Datastore and datatype accordingly.

In the KM options change the option REWRITE_EXISTING_FILE to True

As you can see from the below screenshot, the New sheet is created next to exiting Sheet in the same file.

WRITING HUGE DATA INTO EXCEL FILE

The below example illustrates writing huge data into an Excel File. Note: - Its highly recommended to use only XLSX format to write huge file and enable option WRITING_LARGE_XLSX to be true which uses a different algorithm to write huge data in less time and memory .

WRITING_LARGE_XLSX will purge the existing data in the File. We are going to load the below table data (around 3.6 million) into an Excel file.

In the IKM Options change the options WRITE_LARGE_XLSX to True

As you can see from the screenshot,

it only took 96 seconds to write around 3.6 Million Rows.

1048575 * 3 Sheets + 506247 = 3,651,972 Rows which matches the above total Count. [Note: - In above calculation Header row is not included, thats why its 506247]

LOADING DIRECTLY FROM FILE

The IKM can load directly from a delimited file too, but none of the Header styling will work. Sample output

In the Interface just select the Source File logical schema as Staging Area

ODI EXCEL JDBC DATATYPE

CURRENCY DATE DATETIME NUMERIC PERCENTAGE SCIENTIFIC VARCHAR

CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE DATATYPE SCALE LEVEL

Depending on the scale and the datatype the data is displayed accordingly. For currency we have scale-4, so currency displayed with 4 decimal points. For Numeric we have scale-3, so the numbers is displayed with 3 decimal points. For Percentage the scale -2, so the percentage is displayed with 2 decimal points.

In case if the decimal value in the database is more than the scale specified in the Excel the data is displayed accordingly. For example now all the scale is changed to 2.

Now as you can see from the screenshot, the decimal values are displayed to 2 points. Even though the original data is of different scale, but the data is displayed accordingly and no data is truncated as you can see in the formula bar. In case you wish to display the data as it is in the Database, then have the datatype as VARCHAR, which is shown in the below screenshot

While loading data from Excel (LKM), all the above datatype are read as NUMBER and accordingly loaded as number/Integer into Staging or Target Side.

HANDLING DATE & DATETIME

Date datatype displays only Day, Month and Year and does not support time. For both Date and Time support please use the DATETIME datatype. Note: - If you have trouble loading from the Excel when the datatype is DATE/DATETIME, please use VARCHAR and convert accordingly at the staging or target side.

HANDLING CURRENCY_SYMBOLS
For any currency symbol other than $ please provide the appropriate currency symbol in the options as shown in the below examples.

API
UnloadTables You can use this API to unload multiple tables into same or different Excel File. The method is as follows

UnloadTables(String filename,Connection conn,String schema, String tableNames)

In the Command on Source Specify the Technology and the Schema. In the Command on Target Select the Technology Jython. Here is an example to unload multiple tables of HR Schema.
import api.api as Excelapi conn=odiRef.getJDBCConnection("SRC") fileName='C:/HR_TABLES.xlsx' schema='HR' tableNames='COUNTRIES,DEPARTMENTS,EMPLOYEES,JOB_HISTORY,JOBS,LOCATIO NS,REGIONS' Excelapi.UnloadTables(fileName,conn,schema,tableNames)

Sample Output

In case you wish to have different schema / different excel File; you can repeat the steps in single or different Command.

COMMON ERROR/ISSUES
We have documented some of the common error/issues which can occur due to invalid entry/options in KM options or Excel File. FileNotFoundException: - This exception will occur when you provided an invalid File Name or File without correct Extension.

Sheet does not display in the Operator : - While trying to Reverse multiple or Individual Sheets and if the session in the operator is successful, yet you dont see the Sheet being reversed there is high chance that Sheet is empty or you have provide an Header Row which is empty . Sheet index is out of range: - If you have provided a sheet No which does not exists in Excel File Sheet index out of range error will occur. Cannot get a numeric value from a text cell: - In case if the Excel data type is Numeric, Currency, Percentage and Scientific but some of the data contains Non numeric values or String in nature this Error can occur. Change the datatype of the Excel to VARCHAR and try reloading the Content. Data does not get loaded from Excel (LKM):- While using LKM Excel to SQL, if your interface is successful yet you dont see any data in target, it means you have set the column mapped to Source. Please set the appropriate column mapping to Staging and try again Loading data into Excel is taking too much time: - If you are loading huge data and taking too much. Try enabling WRITING_LARGE_XLSX .

CONTACT US
We hope that this documentation has helped in the initial steps of using and understanding the ODI JDBC EXCEL driver and KMs. In case if you still have issues or question please feel free to contact us at www.odiexperts.com.