7/16/2017 InformaticaDynamicLookupCache

InformaticaDynamicLookupCache
dwbi.org /etl/informatica/138dynamiclookupcache
WrittenbySauravMitra

ALookUpcachedoesnotchangeitsdataoncebuilt.Butwhatiftheunderlyingtableuponwhichlookupwasdone
changesthedataafterthelookupcacheiscreated?Isthereawaysothatthecachealwaysremainuptodate
eveniftheunderlyingtablechanges?

WhydoweneedDynamicLookupCache?

Let'sthinkaboutthisscenario.Youareloadingyourtargettablethroughamapping.Insidethemappingyouhave
aLookupandintheLookup,youareactuallylookingupthesametargettableyouareloading.

Youmayaskme,"So?What'sthebigdeal?Wealldoitquiteoften...".Andyesyouareright.

Thereisno"bigdeal"becauseInformatica(generally)cachesthelookuptableintheverybeginningofthe
mapping,sowhateverrecordgettinginsertedtothetargettablethroughthemapping,willhavenoeffectonthe
Lookupcache.

Thelookupwillstillholdthepreviouslycacheddata,eveniftheunderlyingtargettableischanging.

ButwhatifyouwantyourInformaticaLookupcachetogetupdatedasandwhenthedataintheunderlyingtarget
tablechanges?

Whatifyouwantyourlookupcachetoalwaysshowtheexactsnapshotofthedatainyourtargettableatthat
pointintime?Clearlythisrequirementwillnotbefullfilledincaseyouuseastaticcache.Youwillneedadynamic
cachetohandlethis.

Butinwhichscenariowillsomeoneneedtouseadynamiccache?Tounderstandthis,let'sfirstunderstanda
staticcachescenario.

StaticLookupCacheScenario

Let'ssupposeyourunaretailbusinessandmaintainallyourcustomerinformationinacustomermastertable
(RDBMStable).Everynight,allthecustomersfromyourcustomermastertableisloadedintoaCustomer
Dimensiontableinyourdatawarehouse.Yoursourcecustomertableisatransactionsystemtable,probablyin
3rdnormalform,anddoesnotstorehistory.Meaning,ifacustomerchangeshisaddress,theoldaddressis
updatedwiththenewaddress.

Butyourdatawarehousetablestoresthehistory(maybeintheformofSCDTypeII).Thereisamapthatloads
yourdatawarehousetablefromthesourcetable.TypicallyyoudoaLookupontarget(staticcache)andcheck
witheveryincomingcustomerrecordtodetermineifthecustomerisalreadyexistingintargetornot.Ifthe
customerisnotalreadyexistingintarget,youconcludethecustomerisnewandINSERTtherecordwhereasif
thecustomerisalreadyexisting,youmaywanttoupdatethetargetrecordwiththisnewrecord(iftherecordis
updated).Thisscenariocommonlyknownas'UPSERT'(updateelseinsert)scenarioisillustratedbelow.

AstaticLookupCachetodetermineifasourcerecordisneworupdatable
7/16/2017 InformaticaDynamicLookupCache

Youdon'tneeddynamicLookupcachefortheabovetypeofscenario.

DynamicLookupCacheScenario

NoticeinthepreviousexampleImentionedthatyoursourcetableisanRDBMStable.Generallyspeaking,this
ensuresthatyoursourcetabledoesnothaveanyduplicaterecord.

But,Whatifyouhadaflatfileassourcewithmanyduplicaterecordsinthesamebunchofdatathatyouaretrying
toload?(OrevenaRDBMStablemayalsocontainduplicaterecords)

WouldthescenariobesameifthebunchofdataIamloadingcontainsduplicate?

UnfortunatelyNot.Let'sunderstandwhyfromthebelowillustration.Asyoucanseebelow,thenewcustomer
"Linda"hasbeenenteredtwiceinthesourcesystemmostlikelymistakenly.Thecustomer"Linda"isnotpresent
inyourtargetsystemandhencedoesnotexistinthetargetsidelookupcache.

Whenyoutrytoloadthetargettable,Informaticaprocessesrow3andinsertsittotargetascustomer"Linda"
doesnotexistintarget.ThenInformaticaprocessesrow4andagaininserts"Linda"intotargetsinceInformatica
lookup'sstaticcachecannotdetectthatthecustomer"Linda"hasalreadybeeninserted.Thisresultsinto
duplicaterowsintarget.

Theproblemarisingfromabovescenariocanberesolvedbyusingdynamiclookupcache

Herearesomemoreexampleswhenyoumayconsiderusingdynamiclookup,

Updatingamastercustomertablewithbothnewandupdatedcustomerinformationcomingtogetheras
shownabove
Loadingdataintoaslowlychangingdimensiontableandafacttableatthesametime.Remember,you
typicallylookupthedimensionwhileloadingtofact.Soyouloaddimensiontablebeforeloadingfacttable.
Butusingdynamiclookup,youcanloadbothsimultaneously.
Loadingdatafromafilewithmanyduplicaterecordsandtoeliminateduplicaterecordsintargetby
updatingaduplicaterowi.e.keepingthemostrecentrowortheinitialrow
Loadingthesamedatafrommultiplesourcesusingasinglemapping.JustconsiderthepreviousRetail
businessexample.IfyouhavemorethanoneshopsandLindahasvisitedtwoofyourshopsforthefirst
time,customerrecordLindawillcometwiceduringthesameload.

Howdoesdynamiclookupcachework

Onceyouhaveconfiguredyourlookuptousedynamiccache(wewillseebelowhowtodothat),whenIntegration
Servicereadsarowfromthesource,itupdatesthelookupcachebyperformingoneofthefollowingactions:

Insertstherowintothecache:Iftheincomingrowisnotinthecache,theIntegrationServiceinsertsthe
rowinthecachebasedoninputportsorgeneratedSequenceID.TheIntegrationServiceflagstherowas
insert.
Updatestherowinthecache:Iftherowexistsinthecache,theIntegrationServiceupdatestherowin
thecachebasedontheinputports.TheIntegrationServiceflagstherowasupdate.
7/16/2017 InformaticaDynamicLookupCache

Makesnochangetothecache:Thishappenswhentherowexistsinthecacheandthelookupis
configuredorspecifiedToInsertNewRowsonlyor,therowisnotinthecacheandlookupisconfiguredto
updateexistingrowsonlyor,therowisinthecache,butbasedonthelookupcondition,nothingchanges.
TheIntegrationServiceflagstherowasunchanged.

NoticethatIntegrationServiceactuallyflagstherowsbasedontheabovethreeconditions.

Andthat'sagreatthing,because,ifyouknowtheflagyoucanactuallyreroutetherowtoachievedifferentlogic.

Fortunately,assoonasyoucreateadynamiclookupInformaticaaddsoneextraporttothelookup.Thisnewport
iscalled:

NewLookupRow

Usingthevalueofthisport,therowscanberoutedforinsert,updateortodonothing.Youjustneedtousea
RouterorFiltertransformationfollowedbyanUpdateStrategy.

Oh,forgottotellyoutheactualvaluesthatyoucanexpectinNewLookupRowportare:

0=IntegrationServicedoesnotupdateorinserttherowinthecache.
1=IntegrationServiceinsertstherowintothecache.
2=IntegrationServiceupdatestherowinthecache.

WhentheIntegrationServicereadsarow,itchangesthelookupcachedependingontheresultsofthelookup
queryandtheLookuptransformationpropertiesyoudefine.Itassignsthevalue0,1,or2totheNewLookupRow
porttoindicateifitinsertsorupdatestherowinthecache,ormakesnochange.

ConfiguringaDynamicLookupMappingExample

Ok,IdesignamappingforyoutoshowDynamiclookupimplementation.Ihavegivenafullscreenshotofthe
mapping.Sincethescreenshotisslightlybigger,soIlinkitbelow.Justclicktoexpandtheimage.
7/16/2017 InformaticaDynamicLookupCache

Ifyoucheckthemappingscreenshot,thereIhaveusedaroutertoreroutetheINSERTgroupandUPDATE
group.Therouterscreenshotisalsogivenbelow.NewrecordsareroutedtotheINSERTgroupandexisting
recordsareroutedtotheUPDATEgroup.

DynamicLookupSequenceID

Whileusingadynamiclookupcache,wemustassociateeachlookup/outputportwithaninput/outputportora
sequenceID.TheIntegrationServiceusesthedataintheassociatedporttoinsertorupdaterowsinthelookup
cache.TheDesignerassociatestheinput/outputportswiththelookup/outputportsusedinthelookupcondition.
7/16/2017 InformaticaDynamicLookupCache

WhenweselectSequenceIDintheAssociatedPortcolumn,theIntegrationServicegeneratesasequenceIDfor
eachrowitinsertsintothelookupcache.

WhentheIntegrationServicecreatesthedynamiclookupcache,ittrackstherangeofvaluesinthecache
associatedwithanyportusingasequenceIDanditgeneratesakeyfortheportbyincrementingthegreatest
sequenceIDexistingvaluebyone,whentheinsertinganewrowofdataintothecache.

WhentheIntegrationServicereachesthemaximumnumberforageneratedsequenceID,itstartsoveratone
andincrementseachsequenceIDbyoneuntilitreachesthesmallestexistingvalueminusone.IftheIntegration
ServicerunsoutofuniquesequenceIDnumbers,thesessionfails.

DynamicLookupPorts

Thelookup/outputportoutputvaluedependsonwhetherwechoosetooutputoldornewvalueswhenthe
IntegrationServiceupdatesarow:

Outputoldvaluesonupdate:TheIntegrationServiceoutputsthevaluethatexistedinthecachebefore
itupdatedtherow.
Outputnewvaluesonupdate:TheIntegrationServiceoutputstheupdatedvaluethatitwritesinthe
cache.Thelookup/outputportvaluematchestheinput/outputportvalue.

Note:WecanconfiguretooutputoldornewvaluesusingtheOutputOldValueOnUpdatetransformation
property.

HandlingNULLindynamicLookUp

IftheinputvalueisNULLandweselecttheIgnoreNullinputsforUpdatepropertyfortheassociatedinputport,
theinputvaluedoesnotequalthelookupvalueorthevalueoutoftheinput/outputport.Whenyouselectthe
IgnoreNullproperty,thelookupcacheandthetargettablemightbecomeunsynchronizedifyoupassnullvalues
tothetarget.Youmustverifythatyoudonotpassnullvaluestothetarget.

Whenyouupdateadynamiclookupcacheandtargettable,thesourcedatamightcontainsomenullvalues.The
IntegrationServicecanhandlethenullvaluesinthefollowingways:

Insertnullvalues:TheIntegrationServiceusesnullvaluesfromthesourceandupdatesthelookup
cacheandtargettableusingallvaluesfromthesource.
IgnoreNullinputsforUpdateproperty:TheIntegrationServiceignoresthenullvaluesinthesource
andupdatesthelookupcacheandtargettableusingonlythenotnullvaluesfromthesource.

Ifweknowthesourcedatacontainsnullvalues,andwedonotwanttheIntegrationServicetoupdatethelookup
cacheortargetwithnullvalues,thenweneedtochecktheIgnoreNullpropertyforthecorresponding
lookup/outputport.

WhenwechoosetoignoreNULLs,wemustverifythatweoutputthesamevaluestothetargetthatthe
IntegrationServicewritestothelookupcache.WecanConfigurethemappingbasedonthevaluewewantthe
IntegrationServicetooutputfromthelookup/outputportswhenitupdatesarowinthecache,sothatlookup
cacheandthetargettablemightnotbecomeunsynchronized.

Newvalues.Connectonlylookup/outputportsfromtheLookuptransformationtothetarget.
Oldvalues.AddanExpressiontransformationaftertheLookuptransformationandbeforetheFilteror
Routertransformation.AddoutputportsintheExpressiontransformationforeachportinthetargettable
andcreateexpressionstoensurethatwedonotoutputnullinputvaluestothetarget.

SomeotherdetailsaboutDynamicLookup
7/16/2017 InformaticaDynamicLookupCache

Whenwerunasessionthatusesadynamiclookupcache,theIntegrationServicecomparesthevaluesinall
lookupportswiththevaluesintheirassociatedinputportsbydefault.

Itcomparesthevaluestodeterminewhetherornottoupdatetherowinthelookupcache.Whenavalueinan
inputportdiffersfromthevalueinthelookupport,theIntegrationServiceupdatestherowinthecache.

Butwhatifwedon'twanttocompareallports?

WecanchoosetheportswewanttheIntegrationServicetoignorewhenitcomparesports.TheDesigneronly
enablesthispropertyforlookup/outputportswhentheportisnotusedinthelookupcondition.Wecanimprove
performancebyignoringsomeportsduringcomparison.(Learnhowtoimproveperformanceoflookup
transformationhere)

Wemightwanttodothiswhenthesourcedataincludesacolumnthatindicateswhetherornottherowcontains
dataweneedtoupdate.SelecttheIgnoreinComparisonpropertyforalllookupportsexcepttheportthat
indicateswhetherornottoupdatetherowinthecacheandtargettable.

Note:WemustconfiguretheLookuptransformationtocompareatleastoneportelsetheIntegrationServicefails
thesessionwhenweignoreallports.