Proactive PowerCenter Monitoring

Operational Monitoring and Governance Solution

Kurt McDonal Product Specialist

Agenda
Introduction
Why did we build this? Becoming Proactive

Customer Pains Technical Overview Demonstration

Why Did we Build this?

PowerCenter: 15,000 implementations 4,000 customers Common Customer Challenges surfaced in 25+ Customer Visits
People struggle with comprehensive Operational Monitoring of PowerCenter Severe Consequences when PowerCenter Workflows fail No Proactive way to Enforce Best Practices Difficult to manage Development Quality Across Teams On and Offshore
3

The Goal of Proactive PowerCenter Monitoring

Proactive Issue(s) Problem(s)

Reactive Loss(es)

Event(s)

Time
Workflow is started Changes are made Login to Repository Workflows are Named Paths are hardcoded Workflows run long Schedules are Developers make missed many changes No rows are loaded Source rows rejected Zombie Workflows Development Practices slip to Production Workflows Fail Phone Calls At 3am Business doesnt get reports

We can help you avoid many issuesBEFORE they occur.

4

Who Benefits from Proactive PowerCenter Monitoring?

CIO and Management like the fact that it reduces TCO.

Developers benefit from Active Best Practices and Reduced Time Spent Monitoring

BI Users benefits from more reliable data and decreases in delays to loading issues

Offshore Manager and team benefit from Active Best Practice Enforcement Operations team benefits from Reduced Time Spent Monitoring and Less issues and Failures Helpdesk benefits from Reduced Time Spent Monitoring and Less issues and Failures

Anyone who accesses operational systems that are loaded by PowerCenter

Where we seeing it being used

Development

Actionable Best Practices Enforcement Reduce Training of New Developers

Production Environments

Proactive PowerCenter Monitoring

Testing

Operational Monitoring Automate Code Review Performance Testing

Code Review

Consequences of Being Reactive

Failed Workflows Operational down time Extreme work hours Erroneous reports Dependent systems, people, processes fail Missed SLAs Lower developer efficiency Slow remediation Security threats Failed governance policies
7

What Are Some Customer Pains?

We invest a lot on FTEs to monitor PowerCenter and they still miss many issues We have tools to report on performance after an issue occurs, but nothing that helps us prevent it I want out of the custom scripting business for monitoring.. Not scalable and not effective anymore One missed SLA can have significant business impact and damages the reputation of our group

What is Proactive PowerCenter Monitoring

Add-on to Informatica PowerCenter Delivers Alerts to people,
dashboards and systems Application with pre-built

rules/

patterns logic
Flexibility for Self-Service to easily modify rules or create new ones (from simple to very complex)

Extensible

Product Evolution
Q2- Q3 Added 35+ Customers

2010 -Conducted over 25 Customer Site Visits to Gather Requirements Q4 2011

2009 Agent Logic- CEP Technology

Q1 2010 Released Proactive Monitoring for PowerCenter

Added 200 + New Best Practices Rules Will Add Environmental Monitoring (CPU,disk, etc..)

What else can it be used for?... Schema changes, Trigger BI, Monitor other ETL processes

10

Solution Overview
Proactive PowerCenter Monitoring

Custom:
Environmental
(CPU, Memory..)

DB (SQL) JMS Web Services

Alert Enrichment
DBs, Web Services and more
Alert Editing, Creation and Management

Operational Design Time

Start/Stop Workflows Create Help Desk Ticket Update Control Tables

11

In Action
PowerCenter SDK
Real time Workflow and Session Data Workflow Controls

Environment & Correlation Enrichment

(other monitors, etc) OS (CPU, Memory, Disk) DB (Tablespace, Listener)

PowerCenter Repository

Workflow Statistics

Proactive Monitoring for PowerCenter

(Source Feeds, Rules/Templates, Watchlists, Alerts)

Monitor & Alert

Alerts (Dashboards, e-mail, DBs, other systems, etc)

12

Prepackaged Alerts
Operational Monitoring
1 2 3 4 5 6 7 8 9 10 11 12
Workflow/session was successful but zero (0) records loaded Rejected Records in Session Disproportionate elapsed time between Workflow and Session Session/Workflow Failures Correlate session changes to workflow failures SLA Violations: Workflow/Folder/Repository Level Significant increase in Workflow/Session Elapsed Times Significant decrease in applied row counts Workflows missing schedules Too many concurrent workflows Automatic Restart of Zombie Workflows Ping Domain/Integration Service/Repository Service

Governance & Best Practices

13 14 15 16 17 18 19 20 21 22Naming Conventions: Mappings/Sessions/ Workflows Invalidated Mappings/Sessions/Workflows Transforms saved with Default Values Saving objects without comments Frequent Changes to Mappings/Sessions Hardcoded Paths in Object definitions Illegal Command tasks used in Workflows Duplicate Mappings/Sessions/Workflows Customizable Session Configuration alerts License allows for limitless rules

13

Getting the Right Amount of Alerts

Personas for Alerts
Administrators Operation Developers Architects

Set Alerts on
Repository Domain Workflow Session Transforms
14

Expanded Alerting on Attributes for: Transformation, Session, Workflow

Examples of Configuration Flags that Can be Monitored
Additional Concurrent Pipelines for Lookup Cache Creation Cache LOOKUP() function Constraint based load ordering Custom Properties Data Column Delimiter DateTime Format String Default buffer block size Dynamic Partitioning Enable Recovery Error Log DB Connection Error Log File Directory Error Log File Name Error Log Table Name Prefix Error Log Type Is Enabled Line Sequential buffer length Log Row Data Log Source Row Data Maximum Memory Allowed For Auto Memory Attributes Maximum Partial Session Log Files Maximum Percentage of Total Memory Allowed For Auto Memory Attributes Multiplication Factor Number of Partitions On Pre-Post SQL error On Pre-session command task error On Stored Procedure error Optimization Level Override tracing Partitioning Options Pre 85 Timestamp Compatibility Pre-build lookup cache Save session log by Save session log for these runs Session Log File Max Size Session Log File Max Time Period Session on Grid Stop on errors Writer Commit Statistics Log Frequency Writer Commit Statistics Log Interval

15

Demonstration Scenarios
Workflow Successful but returns no rows Workflow runs longer than normal & SLA Violation Design Time Best Practices Enforcement Governance on PowerCenter Options Build a New Rule with Wizard

16

Proactive PowerCenter Monitoring vs. w Monitoring Tools and Scripts

Workflow Management Tools (CA, BMC, etc.) Workflows Miss Schedules Workflows fail or Succeed

Proactive PowerCenter Monitoring

Workflows running longer or shorter than normal Workflows return more or less data than normal Workflow to Session Correlate Concurrent Workflow Alerts SLA Violations: Workflow/Folder/Repository Level Automatic restart of Zombie Workflows Best Practices Enforcement (Governance) Enforce comments when editing Naming Conventions Hard coded paths Reserved Words Framework to Build New Rules such as Throughput
17

Demonstration

18

Prepackaged Alerts
Operational Monitoring
1 2 3 4 5 6 7 8 9 10 11 12
Workflow/session was successful but zero (0) records loaded Rejected Records in Session Disproportionate elapsed time between Workflow and Session Session/Workflow Failures Correlate session changes to workflow failures SLA Violations: Workflow/Folder/Repository Level Significant increase in Workflow/Session Elapsed Times Significant decrease in applied row counts Workflows missing schedules Too many concurrent workflows Automatic Restart of Zombie Workflows Ping Domain/Integration Service/Repository Service

Governance & Best Practices

13 14 15 16 17 18 19 20 21 22Naming Conventions: Mappings/Sessions/ Workflows Invalidated Mappings/Sessions/Workflows Transforms saved with Default Values Saving objects without comments Frequent Changes to Mappings/Sessions Hardcoded Paths in Object definitions Illegal Command tasks used in Workflows Duplicate Mappings/Sessions/Workflows Customizable Session Configuration alerts License allows for limitless rules

19

Benefits
Stakeholders Administrator/ Operator Impacts & Benefits

24/7 Vigilance (reduce personnel needs for monitoring) Avoid failed jobs Data Architect Higher operational uptime Reduce excess/extreme work hours Integration Architects Avoid erroneous reports Deliver on SLAs & Developers Improved developer efficiency Analyst/Subject Faster remediation Matter Expert Protect from security threats
We procured the Proactive PowerCenter Monitoring option because it solves a big problem for us This not only monitors and alerts on a wide variety of issues, but an extra benefit is that it also helps us enforce some best practices in support of wider governance.

20

Additional Information
Overview white paper available New ROI white paper and calculator developed
Example: Medium PowerCenter deployment 200 workflows, 10 developers $350K+ annual costs/impacts Conservative estimates of 2-year positive ROI

Upcoming Deep Dive Webinar

21

Thank You
Q&A Send additional questions or requests for information to:
Scott Fingerhut sfingerhut@informatica.com

22

Proactive Monitoring for PowerCenter

Impact on PowerCenter Performance

23

Impact of PCPM on PowerCenter Workflow Elapsed Times

No. 1 2 3 4 5 6

Workflow Operation Database Table -> Flatfile Flatfile -> Database Table Database Table -> 3 Database Tables Database Table -> Database Table Flatfile -> Database Table (Zero Rows) Flatfile -> Database Table and Flatfile

Average Workflow Elapsed Minutes with PCPM source scheduler settings set at various levels (seconds) No PCPM 300 60 30 20 10 5

00:01:45 00:01:42 00:01:33 00:01:20 00:01:49 00:02:17 00:02:03 00:02:11 00:01:52 00:01:38 00:01:41 00:02:30 00:02:31 00:02:42 00:02:11 00:01:06 00:01:17 00:01:44 00:02:05 00:02:06 00:02:26 00:04:32 00:05:20 00:05:26 00:05:06 00:07:06 00:06:54 00:10:01 00:03:04 00:02:43 00:03:00 00:02:33 00:01:57 00:02:14 00:03:32 00:04:13 00:04:38 00:05:23 00:04:48 00:07:04 00:08:13 00:09:33

24

Impact of PCPM on PowerCenter Workflow Elapsed Times

00:11:31 00:10:05 00:08:38 00:07:12 00:05:46 00:04:19 00:02:53 00:01:26 00:00:00 No PCPM 300 60 30 20 10 5 1 Database Table -> Flatfile 2 Flatfile -> Database Table 3 Database Table -> 3 Database Tables 4 Database Table -> Database Table 5 Flatfile -> Database Table (Zero Rows) 6 Flatfile -> Database Table and Flatfile

25

Reducing the TCO of PowerCenter

Development

Development Benefits
Actionable Best Practices Enforcement

Production Environments

Proactive PowerCenter Monitoring

Testing

Reduce Training of New Developers Provides Metrics to Monitor Development Quality

Code Review

Enforce Proper Documentation Practices

26

Reducing the TCO of PowerCenter

Development

Testing Benefits
Track Run Times of Workflows

Production Environments

Proactive PowerCenter Monitoring

Testing

Correlate changes to errors Track and Monitor Throughput Statistics Make sure variables are constant (commit, cache, etc..)

Code Review

27

Reducing the TCO of PowerCenter

Development

Code Review
Catch Errors before they go to code review

Production Environments

Proactive PowerCenter Monitoring

Testing

Track over 200 Parameters as you perform migrations Track and Monitor Throughput Statistics

Code Review

Enforce Proper Documentation Practices

28

Proactive Monitoring for PowerCenter & Best Practices Enforcement Solution

Discover Issues Before They Become Problems

Louis Hausle Product Specialist Josh Crane Account Manager

29

Agenda
What is Proactive Monitoring for PowerCenter? Why Did Informatica Build It? Technical Overview Demonstration
We procured the Proactive PowerCenter Monitoring option because it solves a big problem for us. This not only monitors and alerts on a wide variety of issues, but an extra benefit is that it also helps us enforce some best practices in support of wider governance.
Gary Policastro
Coordinator, Data Services Fairfax County (VA) Public Schools

30

Proactive Monitoring for PowerCenter Why This, Now?

PowerCenter: 15,000 implementations 4,000 customers Massive Variability:
Onshore and Offshore Teams How mappings are made When sessions/workflows scheduled Which run concurrently or dependently Best practices adopted how enforced What resources used DW Load or Operational? Size/skills of teams

Better than traditional, manual monitoring enforces Governance

31

How Real are the Problems it solves? Quotes from Customers & Prospects
We have 15 full time people monitoring PowerCenter and they still miss many issues.
-Major TelcoCompany

We procured the Proactive PowerCenter Monitoring option because it solves a big problem for us. This not only monitors and alerts on a wide variety of issues, but an extra benefit is that it also helps us enforce years ago we in 2 some best practices support of widerspent over $300,00 governance.

We have spent a great deal of time and money creating and documenting best practices and realize we have no way to enforce them
- Major Technology Company

to write Policastro Gary Coordinator, Data Services monitoring scripts Fairfax County (VA) Public Schools and they only cover a fraction of this solution.
- Investment Advisory Firm

32

The Goal of Proactive Monitoring PowerCenter Example

Proactive Issue(s) Problem(s)

Reactive Loss(es)

Event(s)

Time
Workflow is started Changes are made Workflows are Named Paths are hardcoded Workflows run long Developers make many changes Source rows rejected Schedules are missed No rows are loaded Zombie Workflows Workflows Fail

Phone Calls At 3am Business doesnt get reports

33

Solution Overview
Proactive PowerCenter Monitoring License-Limited + Rule Pack

34

Component Snapshot
Real time Workflow and Session Data

PowerCenter SDK

PowerCenter Run Time Statistics

Environment & Correlation Enrichment

(other monitors, etc)

Workflow Controls

PowerCenter Repository

Workflow Statistics

Proactive Monitoring for PowerCenter

(Source Feeds, Rules/Templates, Watchlists, Alerts)

Monitor & Alert

Alerts (Dashboards, e-mail, DBs, other systems, etc)

35

Prepackaged Scenarios and Alerts

Operational Monitoring
1 2 3 4 5 6 7 8 9 10 11 12
Workflow/session was successful but zero (0) records loaded Rejected Records in Session Disproportionate elapsed time between Workflow and Session Session/Workflow Failures Correlate session changes to workflow failures SLA Violations: Workflow/Folder/Repository Level Significant increase in Workflow/Session Elapsed Times Significant decrease in applied row counts Workflows missing schedules Too many concurrent workflows Automatic Restart of Zombie Workflows Ping Domain/Integration Service/Repository Service

Governance & Best Practices

13 14 15 16 17 18 19 20 21Naming Conventions: Mappings/Sessions/ Workflows Invalidated Mappings/Sessions/Workflows Transforms saved with Default Values Saving objects without comments Frequent Changes to Mappings/Sessions Hardcoded Paths in Object definitions Illegal Command tasks used in Workflows Duplicate Mappings/Sessions/Workflows License allows for limitless rules

36

Other Proactive Monitoring Use Cases.

Proactive Customer Engagement

Customer Service Up Sell & Cross Sell

Proactive Control & Compliance

Corporate Spend Safety & Security Large fines for non-compliance

Proactive Monitoring
IT Processes DI processes Data Quality

Informatica: Internal and Channel Use Only

37

Demonstration Scenarios
Workflow successful but returns Zero rows SLA Violation Workflow runs longer than X Design Time Best Practice Enforcement
Naming Conventions Hard Coded Paths Comments, etc..

Build a Brand New Rule with Wizard

38

The Key to Prevention

Growing levels of complexity require new solutions to help ensure uptime and reliability

39

Session Configuration Flags that Can be Monitored

Additional Concurrent Pipelines for Lookup Cache Creation Cache LOOKUP() function Constraint based load ordering Custom Properties Data Column Delimiter DateTime Format String Default buffer block size Dynamic Partitioning Enable Recovery Error Log DB Connection Error Log File Directory Error Log File Name Error Log Table Name Prefix Error Log Type Is Enabled Line Sequential buffer length Log Row Data Log Source Row Data Maximum Memory Allowed For Auto Memory Attributes Maximum Partial Session Log Files Maximum Percentage of Total Memory Allowed For Auto Memory Attributes Multiplication Factor Number of Partitions On Pre-Post SQL error On Pre-session command task error On Stored Procedure error Optimization Level Override tracing Partitioning Options Pre 85 Timestamp Compatibility Pre-build lookup cache Save session log by Save session log for these runs Session Log File Max Size Session Log File Max Time Period Session on Grid Stop on errors Writer Commit Statistics Log Frequency Writer Commit Statistics Log Interval

40