UNCLASSIFIED

IBM InfoSphere DataStage Demo

Phuoc Kha & Chintan Parekh
Document ID // Rev 0.1 // 15/04/2016

This document is provided for information purposes only. This document is subject to the information classification set out on each page. If
no information classification has been included, this document must be treated as UNCLASSIFIED, SENSITIVE and must not be disclosed
other than with the consent of nbn co. The recipient (including third parties) must make and rely on their own inquiries as to the currency,
accuracy and completeness of the information contained herein and must not use this document other than with the consent of nbn co.
2015 nbn co ltd. nbn, bring it on, and the Aurora device are trademarks of nbn co ltd | ABN 86 136 533 741

Agenda
1. Logging
DataStage Director (10 mins)
Operations Console (10 mins)
Reporting Requirements (10 mins)
2. Processing Time (LocOps)
PL/SQL LocOps Operation (10 min)
Datastage Execution Analysis (10 min)
Recommendation (10 min)

2
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Prototype DataStage Job

3
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

DataStage Director

Logging of Job flow, success and Failure is done inside DataStage

Director logs. These logs are available to application support to
access the success and failure of the physical job run, and provide
Error, Warning and informational messages about the jobs execution
Requires IBM InfoSphere client to be installed

4
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

DataStage Director

5
Document ID // Rev 0.1 // 10/02/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

DataStage Director

6
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

DataStage Director

7
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Operations Console

Provides engine-wide information about job runs, system resources,

workload management queues, and engine status
Browser-base
Need to ensure EngMonApp process is running

8
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Operations Console

9
Document ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Operations Console

10 Document ID // Rev 0.1 // 15/04/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Operations Console

11 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Prototype DataStage Job:

Write Reject Log File

12 Document ID // Rev 0.1 // 15/04/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Rejects output to file

13 Document ID // Rev 0.1 // 15/04/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Reject Log File

14 Document ID // Rev 0.1 // 15/04/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Job/Process Log vs Telstra Data

Reports

Why you need process log?

Why you need 7 Telstra Data Reports?
1.
2.
3.
4.
5.
6.
7.

Polygon Wash Progressive Report

Merge Progressive Report
Street PID Lookup Progressive Report
Pologon Wash Detail Output Report
Merge Detail Output Report
Street PID Lookup Detail Output Report
Street Matching Data Report

15 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Current Tasks of LocOps

(addAddressAlias)
1.

Determine the location type (e.g. base address or subaddress)

2.

Find existing location address by using FindAddress function

3.

Create base address and base address location

4.

Adding audit information

16 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Current Tasks of LocOps

(editAddressAlias)
1.

Determine the location type (e.g. base address or subaddress)

2.

Find existing location address by using FindAddress function

3.

End Date existing address

4.

Create new alias with new details.

5.

Add Audit Information

17 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Execution Analysis
Based on our Analysis of LocOps and Datastage jobs, we found
following key behaviour
1.LocOps addAddress alias took 36 hours to complete 25,000
records
2.When we disable the findAddress function call and compared
DataStage jobs with LocOps below are the timings of execution
Task

LocOps Time

Datastage Time

Insert

3.5 hrs/100,000

25mins/100,000

Update

13 hrs/100,000

40mins/100,000

18 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Datastage Log

19 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Datastage Log

20 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd

Recommendations
Step

LocOps

DataStage

1. Determine Loc Type

Disable

Create reusable module

2. Find Existing Address

Disable

Create reusable module

3. Insert/Update/Retire

Use LocOps

Call LocOps

21 Document ID // Rev 0.1 // 10/02/2016

UNCLASSIFIED

2015 nbn co ltd. nbn is a trade mark of nbn co ltd