Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Informatica Powercenter Debugger Tracing level and Performance Tuning

AUTHOR Amaranadh Palappareddi

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Revision History Document Reference Author Reason for issue Effecti ve Date Approved y

Amaranadh Palappareddi

Table of !ontent

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Informatica Debugger....................................4
1. Overview....................................................................................................................4 2. Debugger Sessions......................................................................................................4 3. Debug Process.............................................................................................................4 4. Privileges Required.....................................................................................................5 1. Normal:....................................................................................................................... 2. !erbose ini"iali#a"ion:................................................................................................. 3. !erbose da"a: .............................................................................................................. 4. $erse:...........................................................................................................................% 1. &''ec"ive (a))ing Design..........................................................................................* 1.1 Descri)"ion ...........................................................................................................* 1.2 +eneral Sugges"ions 'or O)"imi#ing ....................................................................* 1.2 ,oo-u) $rans'orma"ion O)"imi#ing $i)s ..........................................................1. 1.3 O)era"ions and &/)ression O)"imi#ing $i)s .....................................................11 2. Per'ormance $uning a" 0or-'lows...........................................................................11 2.1 Overview.............................................................................................................11 2.2 1den"i'2ing wor-'low bo""lenec-s.......................................................................12 3. O)"imi#ing 0or-'low 'or be""er )er'ormance 3session4...........................................12

Informatica Tracing Levels..............................6

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Informatica Debugger
"# $verview
To gain troubleshooting information, we can debug a valid mapping We configure and run the Debugger from within the !apping "esigner The "ebugger uses a session to run the mapping on the Integration %ervice When we run the "ebugger, it pauses at brea#points and we can view and edit transformation output data We can configure the "ebugger in two situations$ efore running a session& After we save a mapping, we can have some initial tests on the mapping After running a session& %f the session fails, we can configure the Debugger against the session and find out the reason for failure or une&pected results

'# Debugger %essions

While configuring a "ebugger, we can select the following different debugger session t'pes$ E(isting non)reusable session& Uses e(isting source, target, and non(reusable session configuration properties The "ebugger does not suspend on error E(isting reusable session& Uses e&isting source, target, and session configuration properties When we run the "ebugger, it creates and runs a debug wor#flow for the reusable session !reate a debug session instance& We can create source, target and session configuration properties on our own through Debugger Wi*ard %t creates and runs a debug wor#flow for the session

+# Debug Process
Here are the steps to debug a mapping$ !reate brea,points& We can create brea#points in a mapping where we want to evaluate data and error conditions !onfigure the Debugger& Using the "ebugger Wi)ard configure the Debugger and select a session When we create a debug session, we can select sub sessions li#e sources and targets under debugger wi)ard

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Run the Debugger& When we run the debugger, the debugger connects to %ntegration *ervice Integration service runs the session and connected wor,flow %t reads the brea#points and pause the debugger when condition evaluates to true -onitor the Debugger& While running the "ebugger we can view the transformation, target and mapping output data, debug log and session log There are few windows we can view target data and transformation data etc$ o o o Debug log& +iew messages from the "ebugger Target window& +iew target data Instance window& +iew transformation data

-odify data and brea,points& We can modify the data when the "ebugger pauses and see the effect on transformations, mappings and targets as data moves through pipeline

.ote& "esigner saves the brea#point information in the wor#space files We can cop' this information and use this in another mapping

/# Privileges Re0uired
There is a set of privileges re,uired to wor# with "ebugger, To create or edit brea,points1 Use Designer privilege with read permission for the folder %uper 2ser privilege

To run the Debugger1 Use Designer and Use Wor,flow !anager privilege with read and e(ecute permission on the folder Use Designer and Wor,flow Operator privilege with read permission on the folder %uper 2ser privilege

To run the Debugger using an Integration %ervice enabled in safe mode1 Admin %ntegration *ervice, Use Designer, and Use Wor,flow !anager privilege with read and e(ecute permission on the folder Admin %ntegration *ervice, Use Designer, and Wor,flow Operator privilege with read permission on the folder %uper 2ser privilege

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Informatica Tracing Levels

%nformatica server creates session log file for each session %t writes information about session into log files such as initiali)ation process,

Creation of *-. commands for reader and /rrors encountered and load summar' *tart

writer threads

and end times for target loading

/rrors encountered during the session and general information /&ecution of post(session commands .oad summar' of reader, writer, and DTM statistics Integration Service version and build number

The amount of detail in session log file depends on the tracing level that is set at *ession Properties There are different types of tracing levels, they are "# '# +# /# .ormal 3erbose initiali*ation 3erbose data Terse

"# .ormal&

Power center server logs initiali)ation and status %nformation, skipped rows and summari)ation of the success rows and target rows due to transformation errors

'# 3erbose initiali*ation&

%n addition to 0ormal tracing, the session log file contains location of the data cache files and index cache files that are treated and detailed transformation statistics for each and ever' transformation within the mapping

+# 3erbose data&

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Along with verbose initiali)ation records it logs

each and every record debugging purpose

processed by the informatica

+erbose initiali)ation and verbose data are used for

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Advantages&

1inding out the error where e&actl' %ntegration *ervice truncates string data to fit the precision of a column and provides detailed transformation statistics %t writes errors to both Session log and Error log when enable row error logging +erbose initiali)ation and verbose data are used for debugging purpose

Disadvantages&

%t will decrease the performance due to debugging the records on row b' row basis The space consumed b' the cache files is more when compared to other tracing levels

/# Terse&
To

reduce the amount of time spent writing to the session log file; need to

set the tracing level to terse. At this the power center server does
not write /rror !essages or row(level information for Rejected data dvantages!

of mapping execution the tracing level should be specified as TERSE.

1or better performance

Disadvantages!
It can not find out properly where e(actly where the record is getting truncated and re4ected

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

Informatica Performance Tuning

"# Effective -apping Design

"#" Description
Although Power Center environments var' widel', most sessions and2or mappings can benefit from the implementation of common ob3ects and optimi)ation procedures 1ollow these procedures and rules of thumb when creating mappings to help ensure optimi)ation

"#' 5eneral %uggestions for $ptimi*ing

"# '# +# Reduce the number of transformations There is alwa's overhead involved in moving data between transformations !onsider more shared memory for large number of transformations *ession shared memor' between 45!6 and 78!6 should suffice !alculate once1 use many times o Avoid calculating or testing the same value over and over o Calculate it once in an e&pression, and set a True21alse flag o Within an e&pression, use variable ports to calculate a value than can be used multiple times within that transformation $nly connect what is used o Delete unnecessar' lin,s between transformations to minimi)e the amount of data moved, particularl' in the *ource -ualifier o This is also helpful for maintenance %f a transformation needs to be reconnected, it is best to onl' have necessar' ports set as input and output to reconnect o %n loo,up transformations, change unused ports to be neither input nor output This ma#es the transformations cleaner loo#ing %t also ma#es the generated *-. override as small as possible, which cuts down on the amount of cache necessar' and thereb' improves performance Watch the data types o The engine automatically converts compatible t'pes o *ometimes data conversion is e&cessive "ata t'pes are automaticall' converted when t'pes are different between connected ports -inimi*e data t'pe changes between transformations b' planning data flow prior to developing the mapping 8acilitate reuse o Plan for reusable transformations upfront o Use variables Use both mapping variables as well as ports that are variables +ariable ports are especiall' beneficial when the' can be used to calculate a comple( e&pression or perform a disconnected loo#up call onl' once instead of multiple times o Use mapplets to encapsulate multiple reusable transformations o Use mapplets to leverage the wor# of critical developers and minimi)e mista#es when performing similar functions $nly manipulate data that needs to be moved and transformed

/#

6#

7#

9#

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

;#

Reduce the number of non(essential records that are passed through the entire mapping o Use active transformations that reduce the number of records as earl' in the mapping as possible 9i e , placing filters, aggregators as close to source as possible: o *elect appropriate driving:master table while using 3oins The table with the lesser number of rows should be the driving2master table for a faster 3oin 2tili*e single)pass reads# o Redesign mappings to utili)e one %ource <ualifier to populate multiple targets This wa' the server reads this source onl' once %f 'ou have different %ource <ualifiers for the same source 9e g , one for delete and one for update2insert:, the server reads the source for each *ource -ualifier o Remove or reduce field(level stored procedures %f 'ou use field(level stored procedures, the Power Center server has to ma#e a call to that stored procedure for ever' row, slowing performance

"#' =oo,up Transformation $ptimi*ing Tips

4 5 When 'our source is large, cache loo#up table columns for those loo#up tables of ;88,888 rows or less This t'picall' improves performance b' "> to '> percent The rule of thumb is not to cache an' table over ;88,888 rows This is onl' true if the standard row b'te count is 4,857 or less %f the row b'te count is more than 4,857, then the ;88# rows will have to be ad3usted down as the number of b'tes increase 9i e , a 5,87< b'te row can drop the cache row count to between 5;8= and >88=, so the loo#up table should not be cached in this case: This is 3ust a general rule though Tr' running the session with a large loo#up cached and not cached Caching is often still faster on ver' large loo#up tables When using a .oo#up Table Transformation, improve loo#up performance b' placing all conditions that use the e,ualit' operator ? first in the list of conditions under the condition tab Cache onl' loo#up tables if the number of loo#up calls is more than 48 to 58 percent of the loo#up table rows 1or fewer number of loo#up calls, do not cache if the number of loo#up table rows is large 1or small loo#up tables9i e , less than ;,888 rows:, cache for more than ; to 48 loo#up calls Replace loo#up with decode or II8 9for small sets of values: %f caching loo#ups and performance is poor, consider replacing with an unconnected, uncached loo#up 1or overl' large loo#up tables, use dynamic caching along with a persistent cache Cache the entire table to a persistent file on the first run, enable the update else insert option on the d'namic cache and the engine will never have to go bac# to the database to read data from this table Aou can also partition this persistent cache at run time for further performance gains Review comple& e&pressions /&amine mappings via Repositor' Reporting and "ependenc' Reporting within the mapping -inimi*e aggregate function calls Replace Aggregate Transformation ob3ect with an /&pression Transformation ob3ect and an Update *trateg' Transformation for certain t'pes of Aggregations

> 7

; ? @

<

1.

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

"#+ $perations and E(pression $ptimi*ing Tips

4 5 > 7 ; ? @ 0umeric operations are faster than string operations Optimi)e char(varchar comparisons 9i e , trim spaces before comparing: Operators are faster than functions 9i e , BB vs CO0CAT: Optimi)e II8 e&pressions Avoid date comparisons in loo#upC replace with string Test e&pression timing b' replacing with constant Use flat files Using flat files located on the server machine loads faster than a database located in the server machine 8i(ed)width files are faster to load than delimited files because delimited files re,uire e(tra parsing %f processing intricate transformations, consider loading first to a source flat file into a relational database, which allows the Power Center mappings to access the data in an optimi*ed fashion b' using filters and custom %<= *elects where appropriate %f wor#ing with data that is not able to return sorted data 9e g , Web .ogs:, consider using the %orter Advanced E(ternal Procedure Use a Router Transformation to separate data flows instead of multiple 8ilter Transformations Use a %orter Transformation or hash)auto #e's partitioning before an Aggregator Transformation to optimi)e the aggregate With a *orter Transformation, the *orted Ports option can be used, even if the original source cannot be ordered Use a .ormali*er Transformation to pivot rows rather than multiple instances of the same target Re3ected rows from an update strateg' are logged to the bad file Consider filtering before the update strateg' if retaining these rows is not critical because logging causes e&tra overhead on the engine Choose the option in the update strateg' to discard re4ected rows When using a @oiner Transformation, be sure to ma#e the source with the smallest amount of data the -aster source %f an update override is necessar' in a load, consider using a =oo,up transformation 3ust in front of the target to retrieve the primary ,ey The primar' #e' update will be much faster than the non)inde(ed loo#up override

< D 48 44 45

4> 47

'# Performance Tuning at Wor,flows

'#" $verview 4: The first step in performance tuning is to identif' performance bottlenec#s *ession tas# is one of the places where we can encounter the bottlenec# 5: The strateg' is to identif' a performance bottlenec#, eliminate it, and then identif' the ne&t performance bottlenec# until 'ou are satisfied with the performance To tune session performance, 'ou can use the test load option to run sessions

11

Wipro Confidential

Wipro Technologies

Informatica Powercenter debugger Tracing level and performance tuning

'#' Identifying wor,flow bottlenec,s 4: %f 'ou do not have a source, target, or mapping bottlenec#, 'ou ma' have a session bottlenec# 5: To identif' a session bottlenec#, use the performance details The %ntegration *ervice creates performance details when 'ou enable Collect Performance "ata in the Performance settings on the session properties >: *mall cache si*e, low buffer memory, and small commit intervals can cause session bottlenec#s 7: /ncountering deadloc,s can slow session performance +# $ptimi*ing Wor,flow for better performance AsessionB

4 5 > 7 ; ? @ < D

Use a grid to balance the %ntegration *ervice wor#load Aou can run independent sessions and wor#flows concurrentl' to improve session and wor#flow performance Aou can increase the buffer memory allocation for sources and targets that re,uire additional memor' bloc#s %f the %ntegration *ervice cannot allocate enough memor' bloc#s to hold the data, it fails the session *et the optimal location and si*e for the caches Increase the commit interval# /ach time the %ntegration *ervice commits changes to the target, performance slows %ncrease the interval at which the %ntegration *ervice commits changes Disable high precision# Performance slows when the %ntegration *ervice reads and manipulates data with the high precision data t'pe Reduce errors tracing# Reduce the error tracing level, which reduces the number of log events generated b' the %ntegration *ervice Remove staging areas# When 'ou use a staging area, the %ntegration *ervice performs multiple passes on the data Running wor#flows and sessions on the nodes of a grid provides the following performance gains$ balances the %ntegration *ervice wor,load Processes concurrent sessions faster Processes partitions faster

48 When 'ou run a wor#flow on a grid, the %ntegration *ervice loads memor' and CPU resources on nodes without re,uiring coordination between the nodes 44 When 'ou configure the %ntegration *ervice with high availabilit', the %ntegration *ervice recovers wor#flows and sessions that ma' fail because of temporary networ, or machine failures To recover from a wor#flow or session, the %ntegration *ervice writes the states of each wor#flow and session to temporar' files in a shared director' This ma' decrease performance

12

Wipro Confidential