Ab Initio Training

Ab Initio
TCS, Kolkata
1
Confidential
Asias Largest Global Software & Services Company
Ab Initio
Ab Initio >
DAY 1
2
Confidential
Ab Initio
INTRODUCTION
Data processing tool from Ab Initio software corporation (

http://www.abinitio.com)
Latin for from the beginning
Designed to support largest and most complex business
applications
Graphical, intuitive, and fits the way your business
workstext
Confidential
Ab Initio
Product Constituents
Graphical Development Environment

(GDE)
FTP
TELNET
REXEC
RSH
DCOM
Co-operating System
On a typical installation, the Co-operating system is
installed on a Unix or Windows NT server while the
GDE is installed on a Pentium PC.
4
Confidential
Ab Initio
Product Architecture
GDE
Host Machine 1
Supplies parameter values to underlying

programs through arguments and
environment variables
Ability to graphically design batch

programs comprising Ab Initio
components, connected by pipes
Ability to test run the graphical

design and monitor its progress
Controls the flow of data through pipes
Usually generated using the GDE
Ability to generate a shell script or

batch file from the graphical design
Unix Shell Script or NT Batch File
Co>Operating System
Ab Initio Built-in
Component Programs
(Partitions, Transforms etc)
Host Machine 2
User
Programs
Co-Operating
System
User
Programs
Operating System
( Unix , Windows NT )
Operating System
5
Confidential
Ab Initio
CO>Operating System
Layered on the top of the operating system
Unites a network of computing resources CPUs, storage disks, programs, datasets into a data-processing system with scalable performance
Co>Operating system runs on
IBM
AIX 4.1,4.2,4.3.
Sun
Solaris 2.5.1,2.6,2.7.
HP
HPUX 10, 11.
Sequent
DYNIX/ptx 4.4.1, 4.4.2.
Alpha
Digital Unix 4.0D, 4.4.2.
Pyramid
Reliant Unix 5.43 .
Intel
Windows NT 4.0
6
Confidential
Ab Initio
CO>Operating System contd
Co>Operating System Services

Parallel and distributed application execution
Control
Data Transport
Transactional semantics at the application level

Checkpointing
Monitoring and Debugging
Parallel file management
Metadata-driven components
7
Confidential
Graphical Development Environment
Ab Initio
The GDE
can talk to the Co-operating system using several protocols like Telnet,
Rexec and FTP
is GUI for building applications in Ab Initio
and the Co-operating system have different release mechanisms, making
Co-operating system upgradation possible without change in the GDE
release
Note: During deployment, GDE sets AB_COMPATIBILITY to the Co>Operating System version number. So, a change in the
Co>Operating System release may require a re-deployment
8
Confidential
Ab Initio
The Graph Model
A Graph
is the logical modular unit of an application.
consists of several components that forms the building blocks of
an Ab Initio application
A Component
is a program that does a specific type of job and can be controlled
by its parameter settings.
A Component Organizer
groups all components under different categories.
9
Confidential
Ab Initio
The Graph Model: Naming the pieces
A Sample Graph
Datasets
Dataset
Components
L1
L1
L1*
L1*
Score
Select
Customers
out*
deselect*
Good
Customers
L1
Flows
Other
Customers
10
Confidential
Ab Initio
The Graph Model: A closer look
A Sample Graph
Expression Metadata
Record format metadata
Ports
Layout
11
Confidential
Parts of typical graph
Ab Initio
Files
Formats
Components
Flows
Layouts
Building with mp job
Building with mp run
12
Confidential
Structural Components of a Graph
Ab Initio
Setup Command
Ab Initio Host (AIH) file
Builds up the environment to run an Ab Initio application.
Start Script (Host Setup)

Local to the Graph
Graph
End Script
Local to the Graph
13
Confidential
Ab Initio
Runtime Environment
A graph, after development, is deployed to the back-end

server as a Unix shell script or Windows NT batch file.
This becomes the executable to run at the back-end
with the help of the Co-operating system.
The execution can be done from the GDE itself or
manually from the back-end
Ab Initio runtime environment is different from the
development environment.
14
Confidential
A sample graph
Ab Initio
15
Confidential
A sample korn shell script
Ab Initio
16
Confidential
Ab Initio
DMLs and XFRs
DML
Ab Initio stores metadata in the form of record formats.
Metadata can be embedded within a component or can be
stored external to the graph in a file with a .dml extension.
XFR
Data can be transformed with the help of transform functions.
Transform functions can be embedded within a component or
can be stored external to the graph in a file with a .xfr
extension.
17
Confidential
Ab Initio
Anatomy of a Running Job
Host Process Creation

Pushing Run button generates script
Script is transmitted to Host node
Script is invoked,creating Host process
CPU2
Memory
BUS
CPU1
Disk
HOST
GDE
CLIENT
HOST
PROCESSING NODES
18
Confidential
Ab Initio
Agent Process Creation

Host process spawns Agent process
HOST
GDE
Agent
CLIENT
HOST
Agent
PROCESSING NODES
19
Confidential
Ab Initio
Component Process Creation

Agent process creates Component processes on each processing
nodes
HOST
GDE
Agent
CLIENT
HOST
Agent
PROCESSING NODES
20
Confidential
Ab Initio
Component Process Execution

Component processes do their jobs.
Component processes communicate directly with datasets and
each other to move data around
HOST
GDE
Agent
CLIENT
HOST
Agent
PROCESSING NODES
21
Confidential
Ab Initio
Successful Component Process Termination

As each Component process finishes with its data,it exits with
success status
HOST
GDE
Agent
CLIENT
HOST
Agent
PROCESSING NODES
22
Confidential
Ab Initio
Successful Agent Process Termination

When all of an Agents Component processes exit,the Agent
informs the Host process that those component are finished.
The Agent process then exits.
HOST
GDE
CLIENT
HOST
PROCESSING NODES
23
Confidential
Ab Initio
Successful Host Process Termination

When all have exited,the Host process informs the GDE that the job
is complete.
The Host process then exits
HOST
GDE
CLIENT
HOST
PROCESSING NODES
24
Confidential
Ab Initio
Abnormal Component Process Termination

When an error occurs in a Component process,it exits with error
status.
The Agent then informs the Host
HOST
GDE
Agent
CLIENT
HOST
Agent
PROCESSING NODES
25
Confidential
Ab Initio
Abnormal Component Process Termination

The Host tells each Agent to kill its Component processes
HOST
GDE
Agent
Agent
CLIENT
HOST
PROCESSING NODES
26
Confidential
Ab Initio
Agent Process Termination

When every Component process of an Agent have been killed, the Agent
informs the Host process that those components are finished.
The Agent process then exits
HOST
GDE
CLIENT
HOST
PROCESSING NODES
27
Confidential
Ab Initio
Host Process Termination

When all Agents have exited, the Host process informs the GDE that the job failed
The Host process then exits
HOST
GDE
CLIENT
HOST
PROCESSING NODES
28
Confidential
Ab Initio
Ab Initio >
DAY 2
29
Confidential
Ab Initio
Data Metadata Language or DML
Record Format Metadata in DML
Field Names
Data Types
0345John
Smith
0212Sam
Spade
0322Elvis
Jones
0492Sue
West
0221William
Black
record
decimal(4) id;
DML BLOCK
string(6)
first_name;
string(6)
last_name;
end
30
Confidential
Ab Initio
DML Syntax
Record types begin with record and end with end
Fields are declared: data_type(len) field_name;
Field names consist of letters(az,AZ),digits(09) and
underscores(_) and are Case sensitive
Keywords/Reserved words are record, end, date.
31
Confidential
Ab Initio
Data Types
String
Decimal
Integer
Storing Data in binary form
Date and Datetime

EBCDIC and ASCII records
32
Confidential
Ab Initio
More DML Types

0345,01-09-02,1000.00John,Smith
Delimiters
0212,05-07-03, 950.00Sam,Spade
0322,17-01-00, 890.50Elvis,Jones
0492,25-12-02,1000.00Sue,West
0221,28-02-03, 500.00William,Black
record
decimal(,) id;
date(DD-MM-YY)(,) join_date;
decimal(7,2) salary_per_day;
Precision
& Scale
string(,)
first_name;
string(\n)
last_name;
end
33
Confidential
Ab Initio
NULL in Ab Initio
Null in Oracle Missing Information

Null in Ab Initio Non-existence of coulmns
Ab Initio uses Conditional DML and is_defined function
in order to take care of NULL values.
Records
Table Data
Header
Header NULL NULL
Body
NULL Body
Trailer
NULL NULL Trailer
NULL
34
Confidential
Ab Initio
Data Transformation
record
0345,090297John,Smith;
Drop
decimal(7) id;
date(MMDDYY) join_date;
string(,)
first_name;
string(;)
last_name;
end
Reformat
Reformat
Reorder
id+1000000
record
decimal(7) id;
string(8)
last_name;
date(DD-MM-YY)(,) join_date;
end
1000345,Smith
1997-09-02
35
Confidential
Transform Functions : XFRs
Ab Initio
User-defined function producing one or more output from one or more

input
Associated with transform components
Rules that computes expression from input values and local variable and
assigns the result to output objects
Syntax
Functions :
output-records : : function-name (input-records) =
begin
assignments
end;
Assignments :
output-records.field : : expression;
36
Confidential
Basic Components
Ab Initio
Filter by Expression
Reformat
Redefine Format
Sort
Join
Replicate
Dedup
Aggregate
Rollup
Scan
37
Confidential
Ab Initio
Filter by Expression
1. Reads record from input port
2. Evaluate the select_expr
3. If result is true, record written to out port
4. If result is false, record written to deselect port
Input port
expr
true?
Yes
Out port
No
Deselect
38
Confidential
Ab Initio
Diagnostic Ports
REJECT
Input records that caused error
ERROR
Associated error message
LOG
Logging records
Note: Every transform component has got diagnostic ports
39
Confidential
Ab Initio
Reformat
1. Reads record from input port
2. Record passes as argument to transform function or xfr
3. Records written to out ports, if the function returns a success

status
4. Records written to reject ports, if the function returns a failure
status
40
Confidential
Reformat
Ab Initio
Parameters of Reformat Component

Count
Transform (Xfr) Function
Reject-Threshold
Abort
Never Abort
Use Limit & Ramp
Limit
Ramp
41
Confidential
Ab Initio
Instrumentation Parameters
Limit
Number of errors to tolerate
Ramp
Scale of errors to tolerate per input
Tolerance value=limit + ramp*total number of records read

Typical Limit and Ramp settings . .
Limit = 0 Ramp = 0.0 Abort on any error
Limit = 50 Ramp = 0.0 Abort after 50 errors
Limit = 1 Ramp = 0.01 Abort if more than 1 in 100 records causes error
Limit = 1 Ramp = 1 Never Abort
42
Confidential
Ab Initio
Sort
Keys
A key identifies a field or set of fields to organize a dataset
Single Field: employee_number
Multiple field or Composite key: (last_name; first_name)
Modifiers: employee_number descending
A surrogate key is a substitution for the natural primary key.

It is just a unique identifier or number for each record like ROWID of an Oracle table
Sort Component
Reads records from input port, sorts them by key, writes result to output port
Parameters
Key
Max-core
43
Confidential
Ab Initio
Join
1. Reads records from multiple input ports
2. Operates on records with matching keys using a multi-input transform

function
3. Writes result to the output port
PORTS
PARAMETERS
in
count
out
key
unused
override key
reject (optional)
transform
error (optional)
limit
log (optional)
ramp
44
Confidential
Ab Initio
Join
Join Types
Inner
Outer
Explicit
Join Methods
Merge Join
Using sorted inputs
Hash Join
Using in-memory hash tables to group input
45
Confidential
Ab Initio
Priority Assignment
The Priority is the order of evaluation of rules in a

transform function.
An example
A join component may have a transform function with
prioritized rules as
out.ssn :1: in1.ssn;
out.ssn :2: in2.ssn;
out.ssn :3: "999999999";
46
Confidential
Ab Initio
Multistage Transform
Data transformation in multiple stages following several sets of

rules
Each set of rule form one transform function
Information is passed across stages by temporary variables
Stages include initialization, iteration, finalization and more
Few multistage components are aggregate,rollup,scan
Aggregate/Rollup/Scan
Generates summary records for group of input records
47
Confidential
Ab Initio
Components contd..
Name
Description
Normalize
Generates multiple data records from each input data record

Separate a data record with a vector field into several individual records, each
containing one element of the vector.
Denormalize
Sorted
Consolidates groups of related data records into a single output record with a vector
field for each group
Requires Grouped Input
Validate
Records
Separates valid data records from invalid data records
Check Order
Tests whether data records are sorted according to a key-specifier.
Compare
Records
Compares data records from two flows one by one
Generate
Records
Generates a specified number of data records with fields of specified lengths and types.
Gather Logs
Collects the output from the log ports of components for analysis of a graph after
execution
Sample
Selects a specified number of data records at random from one or multiple input flows
48
Confidential
ROLLUP
Ab Initio
ROLLUP EXAMPLE:
customer_id dt
amount
C002142
1994.03.23 52.20
C002142
1994.06.22 22.25
C003213
1993.02.12 47.95
C003213
1994.11.05 221.24
C003213
1995.12.11 17.42
C004221
1994.08.15 25.25
C008231
1993.10.22 122.00
C008231
1995.12.10 52.10
customer_id total_amount
C002142
74.45
C003213
286.61
C004221
25.25
C008231
174.10
49
Confidential
Ab Initio
ROLLUP
type temporary_type =
record
decimal(8.2) total_amount;
end;
out::initialize(in) =
begin
out.total_amount :: 0;
end;
out::rollup(tmp, in) =
begin
out.total_amount :: temp.total_amount + in.amount;
end;
out::finalize(tmp, in) =
begin
out.customer_id :: in.customer_id;
out.total_amount :: tmp.total_amount;
end;
50
Confidential
Ab Initio
SCAN
type temporary_type =
record
decimal(8.2) amount_to_date;
end;
temp :: initialize(in) =
begin
temp.amount_to_date :: 0;
end;
out :: scan(temp, in) =
begin
out.amount_to_date :: temp.amount_to_date + in.amount;
end;
out :: finalize(temp, in) =
begin
out.customer_id :: in.customer_id;
out.dt :: in.dt;
out.amount_to_date :: temp.amount_to_date;
51
Confidential
Ab Initio
AGGREGATE
record
string(20) customer_id;
decimal(8.2) purchase_amount;
end;
Suppose the output record format is:
record
string(20) customer_id;
decimal(8.2) total;
end;
You can sum the purchases with the following transform function:
out :: agg(temp, in) =
begin
out.total :1: temp.total + in.purchase_amount;
out.total : : in.purchase_amount;
out.customer_id : : in.customer_id;
end;
52
Confidential
Ab Initio
Ab Initio >
DAY 3
53
Confidential
Ab Initio
Built-in Functions
Ab Initio built-in functions are DML expressions that
can manipulate strings, dates, and numbers
access system properties
Function categories
Date functions : now(), today(), date_to_int(), ..
Inquiry and error functions: is_defined(), is_valid(), force_error(), ..
Lookup functions: lookup(), lookup_local(), ..
Math functions: ceiling(), floor(), ..
Miscellaneous functions:decimal_round(), hash_value(), ..
String functions: string_substring(), is_blank(), ..
54
Confidential
Ab Initio
Regular Expression Functions
Perform pattern matching operations on strings using regular expressions

Name
Function
Example
re_get_match()
Returns the first string that

matches a regular expression.
re_get_match("man on the moon", "oo") oo

re_get_match("1234 Milldale Ave.", "[a-zA-Z]+")
"Milldale"
re_get_match("1234 Milldale Ave.", "^[0-9]+")
"1234
re_get_match("Fourteen 35th St.", "^[0-9]+") NULL
re_index()
Returns the index of the first

character of a string matching
a regular expression.
re_index("man on the moon", "oo") 13

re_index("1234 Milldale Ave.", "[a-zA-Z]+") 6
re_index("1234 Milldale Ave.", "^[0-9]+") 1
re_index("Fourteen 35th St.", "^[0-9]+") 0
re_replace()
Returns a string after

replacing all the substrings
matching a regular expression
re_replace("man on the moon", "m[aeiou]+n", "X")

X on X
re_replace_first()
Returns a string after

replacing the first substring
matching a regular
expression.
re_replace_first("man on the moon", "m[aeiou]+n", "X")

"X on the moon"
55
Confidential
Ab Initio
Database Components
db_config_utility : Generate interface file to the database
Input Table
unloads data records from a database into an Ab Initio graph
Source : DB table or SQL statement to SELECT from table
Output Table
loads data records into a database
Destination : DB table or SQL statement to INSERT into table
Update Table
executes UPDATE or INSERT statements in embedded SQL format to modify a
DB table
56
Confidential
Database Components
Ab Initio
Truncate Table
deletes all the rows in a specified DB table
Run SQL
executes SQL statements in a DB
57
Confidential
Ab Initio
Parallelism
Parallel Runtime Environment
Where some or all of the components of an application datasets and
processing modules are replicated into a number of partitions, each
spawning a process.
Ab Initio can process data in parallel runtime environment

Forms of Parallelism
Component Parallelism
Pipeline Parallelism
Inherent in Ab Initio
Data Parallelism
58
Confidential
Ab Initio
Component Parallelism
When different instances of same component run on separate data sets
Sorting Customers
Sorting Transactions
59
Confidential
Ab Initio
Pipeline Parallelism
When multiple components run on same data set
Processing Record 100
Processing Record 99
60
Confidential
Ab Initio
Data Parallelism
When data is divided into segments or partitions and processes run simultaneously on each
partition
Expanded View
s
on
i
it
t
r
Pa
Global View
Multifile
NOTE : # of processes per component = # of partitions
61
Confidential
Data Parallelism
Ab Initio
Multifiles
A global view of a set of ordinary files called partitions usually
located on different disks or systems
Ab Initio provides shell level utilities called m_ commands for
handling multifiles (copy, delete,move etc.)
Multifiles reside on Multidirectories
Each is represented using URL notation with mfile as the protocol
part:
mfile://pluto.us.com/usr/ed/mfs1/new.dat
62
Confidential
Ab Initio
A Multidirectory
A directory spanning across partitions on different hosts
mfile://host1/u/jo/mfs/mydir
//host1/u1/jo/mfs
//host1/vol4/pA/mydir
//host2/vol3/pB/mydir
//host3/vol7/pC/mydir
Data
Partition
on Host2
Data
Partition
on Host3
<.mdir>
Control
Partition
Data
Partition
on Host1
63
Confidential
Ab Initio
A Multifile
A file spanning across partitions on different hosts
mfile://host1/u/jo/mfs/mydir/myfile.dat
//host1/u1/jo/mfs/mydir
/myfile.dat
Control
Partition
//host1/vol4/pA/mydir
/myfile.dat
Data
Partition
on Host1
//host2/vol3/pB/mydir
/myfile.dat
//host3/vol7/pC/mydir
/myfile.dat
Data
Partition
on Host2
Data
Partition
on Host3
64
Confidential
Ab Initio
A sample multifile system

Host Node
Agent Nodes
A multidirectory
A multifile
Control file
Partitions (Serial Files)

Multidirectories dat, s95
Multifiles t.out, new.dat
65
Confidential
Ab Initio
Data parallelism features
Data parallelism scales with data and requires data partitioning

Data can be partitioned using different partitioning methods.
The actual way of working in a parallel runtime environment is
transparent to the application developer.
It can be decided at runtime whether to work in serial or in parallel,
as well as to determine the degree of parallelism
66
Confidential
Ab Initio
Ab Initio >
DAY 4
67
Confidential
Data Partitioning Components
Ab Initio
Data can be partitioned using

Partition by Round-robin
Broadcast
Partition by Key
Partition by Expression
Partition by Range
Partition by Percentage
Partition by Load Balance
68
Confidential
Ab Initio
Partition by Roundrobin
Writes records to the flow partitions in
round-robin way, with block-size
records going into one partition before
moving on to the next
Partition0
A
D
C
G
A
E
Partition1
B
E
D
B
D
A
Partition2
C
F
B
A
F
D
A
B
C
D
E
F
C
D
B
G
B
A
A
D
F
E
A
D
Partition0 Partition1 Partition2

A
B
C
D
E
F
C
D
B
G
B
A
A
D
F
E
A
D
69
Confidential
Ab Initio
Partition by Key
Key dependent data parallelism requires data

partitioning based on key value
Partition by key
Generates a hash code based on the key
Hash code determines which partition a record will go into
Records with same key value goes into the same partition
Partition by Key is also called Hash Partition
70
Confidential
Ab Initio
Partition by Key
Partition0
A
Partition1
Partition2
Partition0
Partition1
Partition2
Hash + Sort = Parallel Grouping
Partition0
Partition1
Partition2
71
Confidential
Ab Initio
Keys, Layouts and Flows
In key-dependent data parallelism,identical key values must

be
in the same partition
Layouts determine the degree of parallelism
Fan-out and All-to-All flows with respect to data partitioning
72
Confidential
Ab Initio
Partitioning Components contd ..

Partition by Range
partitions according to the ranges of key values specified for each

partition
Splitters or split port
Partition by Range + Sort Global Ordering
Partition by Percentage
distributes a specified percentage of the total number of input data
records to each output flow
Pct port
73
Confidential
Ab Initio
Partitioning Components contd ..

Partition by Expression
partitions according to a specified hash function or DML expression
Partition by Load Balance

distributes data records to its output flow partitions, writing more
records to the flow partitions that consume records faster
Broadcast
Acts like a partitioning component when the layout changes
74
Confidential
Ab Initio
Summary of Partitioning Methods
Method
Key-Based
Balancing
Uses
Roundrobin
No
Good
Record-independent
parallelism
Hash
Yes
Good
Key-dependent
parallelism
Function
Yes
Depends on data and

function
Application specific
Range
Yes
Depends on splitters
Key-dependent
parallelism, Global
Ordering
Load-level
No
Depends on load
Record-independent
parallelism
75
Confidential
Ab Initio
Partitioning Performance : SKEW
A
B
C
D
A
D
C
D
B
C
B
A
A
A
A
B
B
B
C
C
C
D
D
D
p0
p1
p2
A
A
C
D
A
D
D
D
B
D
B
A
p3
Partitions evenly balanced
Partitions skewed
A
A
A
A
A
p0
B
B
p1
p2
D
D
D
D
D
p3
Skew determines the distribution of data on different partitions.

Skew for a data storage partition is defined as: (N Average)/Max
where N is the number of bytes in that partition
AVERAGE is the total number of bytes in all the partitions divided by number of partitions
MAX is the number of bytes in the partition with the most bytes.
Common Source of Skew large key groups
76
Confidential
Ab Initio
Departitioning Components
Data can be de-partitioned using
Gather
Concatenate
Merge
Global View
Interleave
Expanded View
77
Confidential
Ab Initio
Gather
Reads data records from the flows connected to the input port
Combines the records arbitrarily and writes to the output
Concatenate
Concatenate appends multiple flow partitions of data records one after another
Merge
Combines data records from multiple flow partitions that have been sorted on
a key
Maintains the sort order
78
Confidential
Ab Initio
Summary of Departitioning Methods

Method
Key-Based Ordering
Uses
Concatenate
No
Global
Creating serial flows

from partitioned data
Interleave
No
Inverse of Round
Robin partition
Creating serial flows

from partitioned data
Merge
Yes
Sorted
Creating ordered
serial flows
Gather
No
Arbitrary
Unordered
departitioning
79
Confidential
Lookup File
Ab Initio
Serial or Multifiles
Held in main memory
Searching and Retrieval is key-based and faster as compared to files stored on
disks
associates key values with corresponding data values to index records and
retrieve them
Lookup parameters
Key
Record Format
80
Confidential
Ab Initio
Lookup File
Storage Methods
Serial lookup : lookup()
whole file replicated to each partition
Parallel lookup : lookup_local()

file partitions held separately
Lookup Functions
Name
Arguments
Purpose
lookup()
File Label and Expression.
Returns a data record from a Lookup File which matches with the
values of the expression argument
lookup_count()
- do -
Returns the number of matching data records in a Lookup File.
lookup_next()
File Label
Returns successive data records from a Lookup File.
lookup_local
File Label and Expression.
Returns a data record from a partition of a Lookup File.
lookup_count_local()
- do -
Same as lookup_count but for a single partition
lookup_next_local()
File Label
Same as lookup_count but for a single partition
NOTE: Data needs to be partitioned on same key before using lookup local functions
81
Confidential
Generating Unique Keys
Ab Initio
In Parallel Environment unique keys are generated using the functions

next_in_sequence()
this_partition()
number_of_partitions()
Key = number_of_partitions() * next_in_sequence() + this_partition() +
last_generated_key
82
Confidential
Ab Initio
Deadlock
A Deadlock happens when
when the graph stops progressing because of mutual dependency of data among
components
Identified when record count in does NOT change over a period of time
Deadlock can be avoided using
Phasing
Checkpointing
Flow Buffering
Components likely to cause deadlock
Join
Concatenate
Compare records
Merge
Interleave
Compare Checksums
NOTE : Too many all-to-all flows likely to cause deadlocks
83
Confidential
Deadlock
Ab Initio
Blocking on read
Blocking on write
84
Confidential
Ab Initio
Ab Initio >
DAY 5
Progressing to the Next Level:
Performance
85
Confidential
Factors: Data Volume
Ab Initio
Data Volume : Files

Source/Target files
DB Tables, Flat files
Temporary files
Phases
Checkpoints
Sort, Merge etc.
Buffered Flows
Data Volume : Skew

reflects data load balance among partitions
86
Confidential
Factors:Data Volume contd..
Ab Initio
Data Volume: I/O

reading from/writing to files on disk
datasets
phases
checkpoints
loading/unloading tables from database

"spilling" to disk
In-memory join or rollup
sort
87
Confidential
Ab Initio
Factors:Data Volume contd..

Data Volume: .WORK
Temp files in parallel layouts are written to .WORK subdirectories of

the multifile system associated with the layout
These files are cleaned up on successful job completion or after
rolling back the job with m_rollback -d.
WARNING: .WORK contains hidden files and should never be
manually removed and recreated.
88
Confidential
Ab Initio
Factors: CPU
CPU: Number of processes
Processes are faster as long as resources are available
Overdriving machine
Resource intensive applications
Symptoms
Increased processes
Increased system calls
Increased paging of memory
Strange error messages
RULE OF THUMB: GO parallel as soon as possible

NOTE: In Data parallelism # processes per component = # partitions
89
Confidential
Factors: Memory
Ab Initio
Memory: Consumers
Lookup Tables
Serial lookup: lookup()
whole table replicated into each partition
Parallel lookup: lookup_local()

table partitions held separately
In-memory components
Rollup, scan, normalize
(temporary data) x (number of key groups)
Join
non-driving input(s) loaded into memory
90
Confidential
Ab Initio
Factors: Memory
Memory: max_core
Memory Allocated by component to hold data
Exceeding max-core
Disk spilling if memory is used up writing to disks
Graph aborts if memory not available
Performance bottleneck: slower execution
Issues
Changing data volume over time
Rule of thumb : available max-core = (total memory)/(2* # partitions)

shared among components that allocate max-core in one phase
91
Confidential
Ab Initio
Factors: Parallel Computing

Rule of Thumb: GO parallel as soon as possible
Processing data serially causes I/O bottleneck
Rule of Thumb: Minimize Serial reads

Reading multiple serial reads simultaneously is faster than reading them
separately
Rule of Thumb: Minimize Database access

Unload DB tables once (if possible) SQL Loader is slow
If multiple graphs require same database table, unload to a file and replicate.
92
Confidential
Ab Initio
Factors: Phases
Purpose:Controlling # simultaneous processes
Performance Enhancements:
CPU Usage:Optimum resource utilization - limit the number of active
components and thus the number of processes.
Memory usage:
Allocate max-core across components in phase
Place components which need large max-core in separate phases
- trade-off: writing to disk at phase boundary
Sort: Set max-core to 100 MB

Safety Cushion: DO NOT over allocate memory in a phase
93
Confidential
Ab Initio
Factors: Phases
Performance Enhancements
Communication bottleneck : ALL-to-ALL flows
N-way to N-way: Uses N network resources
Limit the number of All-to-All flows by using phases
Safety cushion: <=4 per phase
Alternative: Use Two-Stage-Communication
Uses 2Nx(N) channels of communication
applicable if depth is >= 30
94
Confidential
Ab Initio
Factors: Phases
Rule of thumb: Placement of phases
DO NOT put phases after Replicates, across all-to-all flows, after temp
files, after sorts
Data on all flows crossing the current phase boundary is written to
disk, so place phases to minimize writing of data to disks
95
Confidential
Factors: Checkpoint
Ab Initio
Purpose:
Provide same functionality as phase
Additional: Provide restart capability
How does it work ?

At job start, output datasets are copied to temporary files (in .WORK dirs)
At checkpoint completion, intermediate datasets and job state are stored in
temporary files
Recovery information is stored in host and vnode directories of /var/abinitio
96
Confidential
Factors: Checkpoint
Ab Initio
Rule of thumb : Placement of Checkpoint

Same as phase
Rule of thumb : Minimize the amount of data to be held on temporary storage

Rule of thumb : Recovery files
Should be explicitly deleted/rolled back when a checkpointed graph is aborted
m_rollback d graphname.rec
WARNING: rolling back old .rec files will restore the output files to old state
97
Confidential
Ab Initio
Performance: Debugging MP file

insert temporary files to check/capture intermediate data
set reject to Never Abort, capture and inspect rejected records

attach Logger component and inspect log messages
advanced: use write_to_log() / validate to debug dml code
advanced: use hash_value() to check partitioning
Use check order component to check sorting
Using m_dump command to view data
98
Confidential
Performance: Debugging Log file
Ab Initio
A sample log file ..
99
Confidential
Ab Initio
Reading the Log : CPU

CPU time: total processing for component
Status: [ Running : Finished ]
Skew: among CPU times of each partition
Vertex: component
Reading the Log : DATA

Data bytes: # processed
Records: # processed
Status: [ unopened : opened : closed ]
Skew: among data bytes in partitions
100
Confidential
Ab Initio

Reading the Log : DATA
Flow: link between components
data tracking info is displayed on flows in GDE
Vertex: component
Port: of component
Interpreting the log

Compute data bytes/sec through component, in each partition
Look for serialization: effective CPU = (cpu time)/(elapsed time)
compare open vs. closed partitions:serialized when some partitions remain
open long after others have closed data skew
Deadlock:no change in record counts over couple of intervals
101
Confidential
Performance: In a nutshell ..
Ab Initio
Avoid Sorts
Use Lookups
Use In-memory Join/Rollup
Assign Driving Port of Join correctly
Allocate memory correctly
Phasing
102
Confidential
Ab Initio
THANK YOU
103
Confidential

Ab Initio Training

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Ab Initio Training

Enviado por

Direitos autorais:

Formatos disponíveis

Ab Initio

Asias Largest Global Software & Services Company

Asias Largest Global Software & Services Company

Data processing tool from Ab Initio software corporation (

Asias Largest Global Software & Services Company

Graphical Development Environment

Asias Largest Global Software & Services Company

Supplies parameter values to underlying

Ability to graphically design batch

Ability to test run the graphical

Controls the flow of data through pipes

Usually generated using the GDE

Ability to generate a shell script or

Unix Shell Script or NT Batch File

Asias Largest Global Software & Services Company

HPUX 10, 11.

DYNIX/ptx 4.4.1, 4.4.2.

Digital Unix 4.0D, 4.4.2.

Reliant Unix 5.43 .

Asias Largest Global Software & Services Company

CO>Operating System contd

Co>Operating System Services

Transactional semantics at the application level

Asias Largest Global Software & Services Company

Graphical Development Environment

Asias Largest Global Software & Services Company

The Graph Model

Asias Largest Global Software & Services Company

The Graph Model: Naming the pieces

Asias Largest Global Software & Services Company

The Graph Model: A closer look

Asias Largest Global Software & Services Company

Parts of typical graph

Asias Largest Global Software & Services Company

Structural Components of a Graph

Start Script (Host Setup)

Asias Largest Global Software & Services Company

A graph, after development, is deployed to the back-end

Asias Largest Global Software & Services Company

Asias Largest Global Software & Services Company

A sample korn shell script

Asias Largest Global Software & Services Company

DMLs and XFRs

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Host Process Creation

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Agent Process Creation

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Component Process Creation

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Component Process Execution

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Successful Component Process Termination

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Successful Agent Process Termination

Asias Largest Global Software & Services Company

Anatomy of a Running Job

Successful Host Process Termination