Informatica PowerCenter 8.

6
Pushdown Optimization

Agenda
Pushdown Optimization Overview Running Pushdown Optimization
Running Source-Side Pushdown Optimization Running Target-Side Pushdown Optimization Running Full Pushdown Optimization

Working With Databases

Using ODBC drivers

Working With Expressions

Operators Variables Functions

25 May 2012

Agenda
Working With Transformations
Aggregator transformation Expression Transformation Filter Transformation Joiner Transformation Lookup Transformation Sorter Transformation Union Transformation Source Qualifier transformation Target

Working With Sessions

Working With partition Working With Target Loads Rules Error Handling, Logging, and Recovery

25 May 2012

Agenda
Working With SQL Overrides
Views Troubleshooting Orphaned Views Rules and Guidelines

Using the $$PushdownConfig Mapping Parameter Viewing Pushdown Groups Rules and Guidelines

25 May 2012

Pushdown Optimization Overview

25 May 2012

Pushdown Optimization Overview

Use the Pushdown Optimization Viewer to preview the SQL statements ,pushdown messages and mapping logic that is push to the source or target database

Pushes the transformation logic to the source or target database The Integration Service analyzes the mapping and writes one or more SQL statements based on the mapping transformation logic, mapping, session and pushdown configuration
25 May 2012 6

Running Pushdown Optimization

25 May 2012

Running Source-Side Pushdown Optimization

All valid transformation logic are pushed to DB

For SQL or Lookup override views are generated. Select statement runs against the view

Integration Service generates SELECT statement based on the transformation logic for each transformation it can push to the database

The Integration Service analyzes the mapping from the source to the target or until it reaches a downstream transformation it cannot push to the database
25 May 2012 8

Running Target-Side Pushdown Optimization

Processes transformation logic up to the point that it can push the transformation logic to the target database

The Integration Service analyzes the mapping from the target to the source or until it reaches an upstream transformation it cannot push to the database

Integration Service generates INSERT, DELETE, or UPDATE statement based on the transformation logic

25 May 2012

Running Full Pushdown Optimization

INSERT SELECT statement that is run on the database Incorporating the whole of transformation logic

When a session runs for full pushdown optimization, the database must run a long transaction if the session contains a large quantity of data

Only when Source and Target are on same Database

25 May 2012

10

Working With Databases

25 May 2012

Working With Databases

Pushdown optimization can be configured for the following databases : Oracle 9.x and above IBM DB2 Teradata Microsoft SQL Server Sybase ASE Databases that use ODBC drivers

25 May 2012

Working With Databases

When native drivers are used, the Integration Service generates SQL statements using native database SQL When ODBC drivers are used, the Integration Service generates SQL statements using ANSI SQL The Integration Service can generate a greater variety of functions when it generates SQL statements using native language instead of ANSI SQL

25 May 2012

Working With Databases

The database may produce different output than the Integration Service when the following settings and conversions are different: Nulls treated as the highest or lowest value Sort order Case sensitivity Numeric values converted to character values. Date values converted to character values Precision SYSDATE system variable

25 May 2012

Working With Databases

Nulls treated as the highest or lowest value

Null value is treated as low in the sorter transformation

Oracle treats null values as the highest value in the sort order

Returned results therefore will differ as null values are treated differently

25 May 2012

Working With Databases

Sort Order

The sort order is case sensitive

Sort Order in Microsoft SQL Server database is not case sensitive Returned results therefore will differ as sort orders are different.

25 May 2012

Working With Databases

Case Sensitivity

The filter condition will return the values where col_varchar2 = CA

Database which are not case sensitive will return rows that match the values Ca, ca, cA, and CA

Therefore the results will differ in case of pushdown optimization

25 May 2012

Working With Databases

Numeric values converted to character values

The IN_EMPLOYEE_ID = 1234567890 when converted to character will be 1234567890

Database might convert the number to 1.2E9

Returned results may differ because of different conversion format

25 May 2012

Working With Databases

Date values converted to character values

The ROUND function on a date, stores the date value in a character column, using the format MM/DD/YYYY HH:MI:SS.US Oracle stores the date value using default format as DD-MON-YY

Disable pushdown optimization if the date format required is MM/DD/YYYY HH:MI:SS.US

25 May 2012

Working With Databases

Precision

A transformation Decimal datatype has a precision of 1-28

Teradata Decimal datatype has a precision of 1-18

Therefore the results will differ as database uses a different precision than the Integration Service

25 May 2012

Working With Databases

SYSDATE system variable

The current date and time for the node running the service process is returned

On Pushdown optimization, the SYSDATE variable returns the current date and time for the machine hosting the database HH:MI:SS.US

If the time zone of the machine hosting the database is not same as the time zone of the machine running the Integration Service process, the results can vary

25 May 2012

Using ODBC Drivers

When ODBC drivers are used, the Integration Service cannot detect the database type and must generates ANSI SQL

In some cases, ANSI SQL is not compatible with the syntax required for a particular database
Different databases encounters different types of problems

25 May 2012

Using ODBC Drivers

IBM DB2 A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort and the one of the sort keys is a string datatype. A session containing a Lookup transformation fails for source-side or full pushdown optimization. A session that requires type casting fails if the casting is from x to date/time or from float/double to string, or if it requires any other type casting that IBM DB2 databases disallow

25 May 2012

Using ODBC Drivers

Microsoft SQL Server A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort Sybase ASE When Sybase ASE 12.5 or below is used, sessions that perform datatype conversions fail. The session fails if Joiner transformation used is configured for a full outer join.

25 May 2012

Using ODBC Drivers

Teradata Teradata sessions fail if the session requires a conversion to a numeric datatype and the precision is greater than 18. Teradata sessions fail when full pushdown optimization is used for a session containing a Sorter transformation A sort on a distinct key may give inconsistent results if the sort is not case sensitive and one port is a character port A session containing an Aggregator transformation may produce different results from PowerCenter if the group by port is a string datatype and it is not case-sensitive A session containing a Lookup transformation fails if it is configured for target-side pushdown optimization A session that requires type casting fails if the casting is from x to date/time. A session that contains a date to string conversion fails.

25 May 2012

Working With Expressions

25 May 2012

Working With Expressions

When pushdown optimization is used, the Integration Service converts the expression in the transformation or in the workflow link by determining equivalent operators, variables, and functions in the database. If there is no equivalent operator, variable, or function, the Integration Service processes the transformation logic.

25 May 2012

Working With Expressions - Operators

Operator

Oracle

IBM DB2

Teradata

Sybase ASE X X

Microsoft SQL Server X X

ODBC

+-*/ = > < >= <= <> != ^= not and or || %

X X

X X

X X

X X

X X X X X

X X X Source X

X X X Source X

X X X Source X

X X X Source X

X X X -

25 May 2012

Working With Expressions - Variables

Variable

Oracle

IBM DB2
X X -

Teradata

SybaseASE

Microsoft SQL Server

X X -

ODBC

SESSSTARTTIME SYSDATE WORKFLOWSTARTTIME

X X -

X X -

X X -

X -

25 May 2012

Working With Expressions - Functions

Function IBM Microsoft Oracle Sybase DB2 SQL ASE Server Teradata ODBC Exceptions

ABS() X ABORT() ADD_TO_DATE() X

X
S

X
X

X
S

X
X

X
Cannot push transformation logic to a Teradata database for a transformation that uses an expression containing ADD_TO_DATE to change days, hours, minutes, or seconds.

AES_DECRYPT() AES_ENCRYPT() ASCII() X AVG() X CEIL() X CHOOSE() CHR() X CHRCODE()

X X X X

X X X X

X X X X

X S

25 May 2012

Working With Expressions - Functions

Function

IBM DB2 Microsoft Oracle SQL Server

Sybase ASE

Teradata

ODBC

COMPRESS() CONCAT() COS() COST() COSH() COUNT() CRC32() CUME() DATE_COMPARE() DATE_DIFF() DECODE()

X S X X X X X X S X X

X S X X S X X X S X X

X X X X X X X X S X X

X S X X S X X X S X X

X S X X X X X S S X

X X X X X S X

25 May 2012

Working With Expressions - Functions

Function DECODE_BASE64() DECOMPRESS() ENCODE_BASE64() ERROR() EXP() FIRST() FLOOR() FV() GET_DATE_PART() GREATEST() IIF() IN() INDEXOF() INITCAP() INSTR() IS_DATE()

IBM DB2 X S X X X X X X X X X S

Microsoft SQL Server X S X X X X X X X

Oracle X X X X X X X X X

Sybase ASE X S X X X X X X X

Teradata X S X X X S X X X

ODBC X X X X

X X

25 May 2012

Working With Expressions - Functions

Function IBM DB2 Microsoft Oracle Sybase Teradata ODBC Exceptions SQL ASE Server

IS_NUMBER() IS_SPACES() ISNULL() LAST()

X Oracle returns the date up to the second. If the input date contains subseconds, Oracle trims the date to the second

LAST_DAY() LEAST() LENGTH() LOWER() LPAD() LTRIM()

X X X X X X

X X

X X

X X

X X

If LTRIM in transformation logic, is used the database treats the argument (' ') as NULL, but the Integration Service treats the argument (' ') as spaces

LOG()

S
25 May 2012

Note: For details on remaining functions refer to the Workflow Administration Guide

Working With Transformations

25 May 2012

Working With Transformations

The Integration Service can push transformation logic to the source, target, or both. Some transformations cannot be pushed to the database, and additional rules apply to each transformation that can be pushed to the database The Integration Service processes the transformation logic if any of the following conditions are true: The transformation logic updates a mapping variable and saves it to the repository database The transformation contains a variable port Override default values for input or output ports

25 May 2012

Working With Transformations

Summary of Mapping Objects Valid for Pushdown Optimization

Transformation

Source-Side and Full

Target-Side

Aggregator

Application Source Qualifier

Custom Expression External Procedure Filter HTTP Java Joiner Lookup Normalizer Rank

X X X X -

X X 25 May 2012

Working With Transformations

Summary of Mapping Objects Valid for Pushdown Optimization

Transformation Router Sequence Generator Sorter Source Qualifier

Source-Side and Full X X

Target-Side -

SQL
Stored Procedure Target Transaction Control Union Update Strategy XML Generator XML Parser XML Source Qualifier

X X -

25 May 2012

Working With Transformations Aggregator Transformation

The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true
A session is configured for incremental aggregation

Transformation logic for the first Aggregator is pushed to the database, and the Integration Service processes the second Aggregator transformation

TARGET_ITEM S_ORDER1 (Or acle)

SQ_TARGET_I TEMS_ORDER 1

SOR_OrderID

AGG_OrderID

AGG_ITEMID
25 May 2012

TARGET_ITEM S_ORDER2 (Or acle)

38

Working With Transformations Aggregator Transformation

The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true
An output port is not an aggregate or a part of the group by port

Nested aggregate function or Conditional aggregate clause is used

Any one of the aggregate functions FIRST(), LAST(), MEDIAN(), or PERCENTILE() is used

25 May 2012

Working With Transformations Expression and Filter Transformation

ORDERS (Oracl e)

SQ_ORDERS

EXP_Transform Data

order12 (Flat File )

All the valid databases for the pushdown optimization can use all three types of Pushdown Optimization when Expression Transformation is used

ITEMS (Oracle)

SQ_ITEMS

FIL_DisContFlag

ORDER_ITEMS (Oracle)

SQ_ORDER_IT EMS

JOIN_items

TARGET_ITEM S_ORDER (DB2 )

All the valid databases for the pushdown optimization can use all Source Side and Full Pushdown Optimization Type when Filter Transformation is used
25 May 2012

Working With Transformations Joiner Transformation

The Integration Service cannot push the Joiner transformation logic to the database if any of the following conditions are true:
If the Aggregator transformation is placed upstream from a Joiner transformation in the pipeline

SQL cannot be generated to represent an outer join combined with a multi-table join that joins greater than two tables

The Joiner is configured for a full outer join and attempt to push transformation logic to a Sybase database

25 May 2012

Working With Transformations Lookup Transformation

Guidelines for Lookup transformations in Pushdown Optimization Unconnected Lookup Transformation can be used Lookup table and source table must be on the same database for source-side or target-side pushdown optimization Lookup should not be configured to Dynamic Cache or Policy on multiple match as First or Last Value To use pushdown optimization, you must configure the Lookup transformation to report an error on multiple matches Mapping should not contain Lookup transformation downstream from an Aggregator transformation Database does not uses Powercenter cache during Pushdown Optimization

25 May 2012

Working With Transformations Lookup Transformation (Snapshot)

Only one branch of the pipeline is processed when Lookup transformation exists in two branches of a pipeline that are joined by a Joiner or Union transformation

When session requires data type conversion target-side pushdown optimization fails

Target-side pushdown optimization fails when lookup table connected to ODBC Connection or Sybase or Microsoft SQL Server database

25 May 2012

Working With Transformations Lookup Transformation

Use the following rules and guidelines if the lookup query in a Lookup transformation is overridden : Configure the session for pushdown optimization with a view. The Integration Service creates a view, runs an SQL query against the view, and then drops the view. Need to manually remove the views that the Integration Service creates. To use a lookup override, should have necessary database privileges to create a view. Cannot append a custom ORDER BY clause to the SQL statement in the lookup override. The order of the columns in the lookup override must match the order of the ports in the Lookup transformation.

25 May 2012

Working With Transformations Sorter Transformation

Guidelines for Sorter transformations in Pushdown Optimization

On distinct sort, the Integration Service pushes the Sorter transformation to the database and processes downstream transformations

When a Sorter transformation is downstream from a Union transformation, the sort key on the Union transformation must be a connected port.

25 May 2012

Working With Transformations Union Transformation

Guidelines for Union transformations in Pushdown Optimization

A distinct key port should be connected

Input groups must originate from the same source

All the input groups should be pushed to source database

25 May 2012

Working With Sessions

25 May 2012

Working With Sessions

The Integration Service can push transformation logic to the source or target database when it does not conflict with the following session configurations: Partitioning : When you configure pushdown optimization, the Integration Service merges all the rows into the first partition and passes empty data for each subsequent partition. The Integration Service cannot merge partitions for all partition types. Target load rules: Some target load rules affect pushdown optimization. The full pushdown optimization cannot be used when some target load rules are configured. Error handling, logging, and recovery : When the Integration Service pushes transformation logic to the database, it handles errors differently from when it runs a full session.

25 May 2012

Working With Sessions

The Integration Service processes the transformation logic for the session if any of the following conditions are true:

Run a data profiling or debugging session

Use an external loader and configure the session for target-side pushdown optimization. The Integration Service can push transformation logic to the source database if you use an external loader Enable row error logging

25 May 2012

Working With Sessions Partitioning

When a session configured for pushdown optimization contains multiple partitions, the Integration Service can push the session to the database in the following situations:

If a transformation uses pass-through partitioning, the Integration Service can push transformation logic to the source or target database, or both.
If a transformation uses a key range partition in the Source Qualifier transformation and hash auto-keys partitioning in downstream partition points, the Integration Service can push transformation logic to the source database by merging all rows into the first partition. To merge all rows into the first partition, the end key range for each partition must equal the start range for the next partition, and it cannot overlap with the next partition.

25 May 2012

Working With Sessions Partitioning

Sorter transformation with hash auto-keys partitioning

The first key range is 1313 3340, and the second key range is 3340 - 9354

The SQL statement selects items 1313 through 9354, which includes all values in the key range and merges the data from both partitions into the first partition

The SQL statement for the second partition passes empty data

25 May 2012

Working With Sessions Target Load Rules

Pushdown Optimization with Target Load Options Target Option
Insert Delete Update as update Update as insert

Source-Side Optimization
X X X X

Target-Side Optimization
X X X X

Full Optimization
X* X** X*

Update else insert

X* = If source rows as datadriven are treated, the Integration Service cannot perform full pushdown optimization. It can push transformation logic upstream or downstream from the Update Strategy or Custom transformation

X** =Although full pushdown optimization can be used when source rows as delete are treated, performance gains might not be achieved. If not, use partial pushdown optimization. Also, full pushdown optimization cannot be used and treat source rows as delete if the session includes a Union transformation and the Integration Service pushes transformation logic to a Sybase database0

25 May 2012

Working With Sessions Error Handling

When the Integration Service pushes transformation logic to the database, it cannot track errors that occur in the database. As a result, it handles errors differently from when it runs the full session. When the Integration Service runs a session configured for full pushdown optimization and an error occurs, the database handles the errors. When the database handles errors, the Integration Service does not write reject rows to the reject file, and it treats the error threshold as though it were set to 1.

25 May 2012

Working With Sessions Logging

When transformation logic is pushed to the database, the Integration Service performs the following logging functionality:

The session log does not contain details for transformations processed on the database.
The Integration Service does not write the thread busy percentage to the log for a session configured for full pushdown optimization. The Integration Service writes the number of loaded rows to the log for source-side, targetside, and full pushdown optimization. When the Integration Service pushes all transformation logic to the database, the Integration Service does not write the number of rows read from the source to the log.

When the Integration Service pushes transformation logic to the source, the Integration Service writes the number of rows read for each source to the log. However, the number may differ from statistics for the same session run by the Integration Service.

25 May 2012

Working With Sessions Recovery

When a session is configured for full pushdown optimization, the Integration Service runs the session on the database. As a result, it cannot perform incremental recovery if the session fails. Instead the database rolls back transactions. If the database server fails, it rolls back transactions when it restarts. If the Integration Service fails, the database server rolls back the transaction. When recovery for sessions is performed that contain SQL overrides or lookup overrides, the Integration Service must drop and recreate views

25 May 2012

Working With SQL Overrides

25 May 2012

Working With SQL Overrides

When an SQL override for a Source Qualifier transformation is used in a session configured for source or full pushdown optimization with a view, the Integration Service creates a view in the source database based on the override After it creates the view in the database, the Integration Service generates an SQL query that it can push to the database The Integration Service runs the SQL query against the view to perform pushdown optimization To use an SQL override with pushdown optimization, the session must be configured for pushdown optimization with a view.

25 May 2012

Working With SQL Overrides - Views

The session for pushdown optimization when configured with views, the Integration Service completes the following tasks:

Creates a view: The Integration Service generates the view by incorporating the SQL override query in a view definition. The Integration Service does not parse or validate the SQL override, so the SQL override should be tested before running the session.
Runs an SQL query against the view: After the Integration Service generates a view, the Integration Service runs an SQL query to push the transformation logic to the source. It runs this query against the view created in the database. Drops the view: When the transaction completes, the Integration Service drops the view it created to run the SQL override query.

25 May 2012

Working With SQL Overrides - Views

A mapping that searches for 94117 zip codes in a customer database
After the session completes, the Integration Service drops the view

After the view is created, the Integration Service runs an SQL query to perform the transformation logic in the mapping

SQL override which return customers information whose names match variations of the name Johnson, including names such as Johnsen, Jonssen and Jonson

The Integration Service pushes transformation logic for this session to the database, it runs the SQL statement to create a view in the source database

To create a unique view name, the Integration Service appends PM_V to a value generated by a hash function

25 May 2012

Working With SQL Overrides - Troubleshooting Orphaned Views

The Integration Service might be unable to drop a view if the Integration Service, session, or connectivity fails. The orphaned view does not affect performance.

Complete the following steps to remove extra views from the database:
View the session log: While running the session, the Integration Service writes an event log when it creates the view, and it also writes an event log when it drops the view. If an Integration Service, session, or connection fails when running a session, check the session log to see if Integration Service dropped the view. Run a query: If the Integration Service did not drop views, run a query to locate all views created by the Integration Service. Views created by the Integration Service use the same prefix. Drop the view: After locating the views created by the Integration Service, run an SQL statement to manually drop the views from the database.

25 May 2012

Working With SQL Overrides - Rules and Guidelines

Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:

Do not use an order by clause in the SQL override

When a custom SQL query is created, the SELECT statement must list the port names in the order in which they appear in the transformation. Use ANSI outer join syntax in the SQL override. Do not use a Sequence Generator transformation. If a Source Qualifier transformation is configured for a distinct sort and contains an SQL override, the Integration Service ignores the distinct sort configuration.

25 May 2012

Working With SQL Overrides - Rules and Guidelines

Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:

If the Source Qualifier contains multiple partitions, specify the SQL override for all partitions.
If a Source Qualifier transformation contains Informatica outer join syntax in the SQL override, the Integration Service processes the Source Qualifier transformation logic. PowerCenter does not validate the override SQL syntax, so test the SQL override query before it is pushed to database. When SQL override is created, ensure that the SQL syntax is compatible with the source database.

25 May 2012

Using the $$PushdownConfig Mapping Parameter

25 May 2012

Using the $$PushdownConfig Mapping Parameter

To use different pushdown optimization configurations at different times, use the $$PushdownConfig mapping parameter. The parameter runs the same session using the different types of pushdown optimization.

25 May 2012

Using the $$PushdownConfig Mapping Parameter

Complete the following steps to configure the mapping parameter: Create $$PushdownConfig in the Mapping Designer. When the $$PushdownConfig mapping parameter is added in the Mapping Designer, use the following values: Field Value Name Type $$PushdownConfig Parameter

Datatype
Precision or Scale Aggregation Initial Value Description

String
10 n/a None Optional

When the session is configured, choose $$PushdownConfig for the Pushdown Optimization attribute`

25 May 2012

Using the $$PushdownConfig Mapping Parameter

Define the parameter in the parameter file Enter one of the following values for $$PushdownConfig in the parameter file: None: The Integration Service processes all transformation logic for the session. Source: The Integration Service pushes part of the transformation logic to the source database. Source with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push part of the transformation logic to the source database. Target: The Integration Service pushes part of the transformation logic to the target database. Full: The Integration Service pushes all transformation logic to the database. Full with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push part of the transformation logic to the source database. The Integration Service pushes any remaining transformation logic to the target database.

25 May 2012

Viewing Pushdown Groups

25 May 2012

Viewing Pushdown Groups

The group of transformations that can be processed as one SQL statement is called a pushdown group The Integration Service creates multiple pushdown groups depending on the number of pipelines, sources, targets, and the type of pushdown optimization used. If the session has multiple partitions, the Integration Service executes an SQL statement for each partition in the group.

Pushdown groups can be viewed using the Pushdown Optimization Viewer

From Pushdown Optimization Viewer , the transformations that can be pushed to the database can be identified and those that the Integration Service processes

The messages in the Viewer can be used to determine how to edit transformations or mappings to push more transformation logic to the database
The Pushdown Optimization Viewer cannot display the SQL that runs in the session if variables is used or if the session is configured to run on a grid.
25 May 2012

Viewing Pushdown Groups

Pipeline1

Pipeline2

Pushdown Group 1

Pushdown Group 2

* ->Transformation is part of both pushdown groups

25 May 2012

Viewing Pushdown Groups

In the Workflow Manager, open a session configured for pushdown optimization Select the type of optimization.

Select pushdown groups to view

In the Mapping tab, select View Pushdown Optimization

Select a value for the connection variable

Number indicates the pushdown group.

The pushdown option in the viewer does not affect the optimization that occurs at run time. To change pushdown optimization for a session, edit the session properties

SQL statement Generated fo the pushdown g

25 May 2012

Configuring Sessions for Pushdown Optimization

25 May 2012

Configuring Sessions for Pushdown Optimization

Review the information in the Pushdown Optimizer Viewer to determine if edition in the mapping is needed

Select to configure pushdown optimization. Select one of these None, To Source, To Source with View, To Target, Full, Full with View, $$PushdownConfig

25 May 2012

25 May 2012

73

25 May 2012

74