ETL Testing Strategy - Migration Projects

By Nishikant Mathapati

Index

ETL Process Test Strategy

Challenges / Risks
Solution for Risk Benefits
ETL Testing Strategy - Migration Project

ETL Process
The ETL process is being used to perform migration of data from various legacy source systems to the target system. Different ETL stages are -

Source

Extracted Data Delta Stage 1

Transformed Data - Delta stage 2

Data Loaded to Target tables

ETL Testing Strategy - Migration Project

ETL Process Contd

Sample architecture

ETL Testing Strategy - Migration Project

ETL Process Contd

ETL process can be implemented using various tools like Cognos, Informatica, Control-M etc. Required fact and dimension tables will be created, mock / existing production history like data will be acquired from business for testing purpose. Data cleansing will be done on data provided by business. Workflows will be designed which will perform all requisite operations on data and then data will be loaded to target system.
ETL Testing Strategy - Migration Project

Test Strategy

Thorough testing for Migration process is important The ETL process can be effectively validated using checks as specified on next slide

ETL Testing Strategy - Migration Project

Test Strategy

Contd

ETL Testing Strategy - Migration Project

Challenges / Risks faced

Common issues which could be faced in every ETL project implementations are

1. Workflow failures with reasons such as Database drivers not found Unique constraint violation Not enough disk space for delta stage tables 2. Data rejections due to reasons such as Data cleansing not done for fact tables Production look a like data not provided by business Revised data not uploaded to facts tables

ETL Testing Strategy - Migration Project

Solutions

Liaise with development team and create a checklist for all common failures which are repeated and can be avoided Include this list as part of entry criteria to enter in to any testing phase

Smoke test any new build delivered considering the prepared checklist E.g. Run the workflow which will extract and run very large amount of data in all delta stages and in target system

ETL Testing Strategy - Migration Project

Benefits

A good confidence can be established on to the data tested By avoiding common failures one can spend more time in development and testing related activities Data quality will be improved A common practice can be implemented in similar projects where Data Migration / Integration is implemented through ETL process

10

ETL Testing Strategy - Migration Project

Q&A

Thank you all !!!

Questions Please

11

ETL Testing Strategy - Migration Project