Strategies

for a Successful
E2E Systems Integration Test

Fiona Charles
Let s Test
May 9, 2012
 This session
Describes key project management strategies I
have used to manage large-scale Systems
Integration Tests

Oers practical tips for implementing those
strategies on your next SIT

The focus is primarily on large-scale integrations,

but the strategies and tips can be scaled for any
size of test.
Strategies for a Successful E2E SIT 2
1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 3

 A Typical Systems Integration Project

Merchandising New POS Accounting

systems systems

Point of Sale
New Loyalty registers &
systems Controllers

Supply Chain Management

systems Information
systems

Strategies for a Successful E2E SIT 4

 The Test Manager s Job

Develop a test strategy that addresses the major

risks within the time available

Build the team you need for success

Build the right controlled data tests and plan

Manage testing and evaluation

Ensure acceptance by stakeholders

Sounds just like any other test - right???

Strategies for a Successful E2E SIT 5

The test management challenge is at least as
great as the testing challenge

A project in itself

Always highly visible, and often controversial

Requirements for success often misunderstood

Your job is to hold to the big picture

E2E Systems Integration Testing requires the skills
of an experienced test project manager

Strategies for a Successful E2E SIT 6

1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 7

 It s a Test Project Management job
Much of the complexity in managing a Systems
Integration Test comes from the sheer number of
things involved:

Applications
Platforms
Environments
Data
Application readiness dates
Business processes or functions
People

Managing these is a logistical, rather than

test, challenge.
Strategies for a Successful E2E SIT 8
The numbers can seem overwhelming!

Strategies for a Successful E2E SIT 9

But it s a nightmare only if you don t
manage it
Divide and conquer

Have a strategy for each type of thing, and a project

plan that says when you are going to deal with it

Keep your priorities current

Don t confuse the urgent with the merely important

Delegate where you can

Use project management tools and techniques

Strategies for a Successful E2E SIT 10
Be sure to level-set in your own head

Large Vehicles
Need More Room

And then in everyone else s head

Strategies for a Successful E2E SIT 11
 Make sure you are positioned correctly
Senior role, working on equal footing with:
Project managers
Architects

No barriers to information
Pulse of the program
When each project is delivering (really)
Risks, issues, delays, individual test strategies, etc.

Authority
SIT entry criteria
Requirements to make SIT work

Reporting to the overall Project/Program Manager

Strategies for a Successful E2E SIT 12
1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 13

The applications will determine test scope
Start with the Architecture/integration diagrams
Use whatever you can nd
If they don t exist, draw your own
Validate what you use

Keep an open mind about the test boundaries
Received belief may be wrong
Focus on whole system outcomes
Identify all incoming data that could inuence your results
Ask what the end point should really be
What s the last application or report the critical data impacts?
Who needs to sign o?
You may need to include unchanged systems

Strategies for a Successful E2E SIT 14

 System Context Diagram
APP

APP APP
Teller
APP
(Authenticated)

Perform CSR Transactions:

Retrieve Client Profile APP Retrieve Client Existing CTP Objects for:
Credit,
Perform Client Search Profile Retrieve Client Profile
Debit,
Withdraw Funds, Search Client Perform Client Search
DDA Inquiry,
Receive Cash, Retrieve Account Retrieve Account List and
Bank book updates
Deposit Funds (PDA), List Balances (if no client card
Acct Details,
Negotiate Cheque/Money Order, Account Details or for a full account list)
Check SECAF Acct History
Transfer between accounts, Retrieve & Update Offer &
Purchase Drafts/ Money Orders, Entitlements Opportunities
Get a VISA cash advance,
Make VISA payment,
Reverse VISA Payment Negotiate/Accept
Reverse VISA Cash Advance, System priced APP
Purchase Foreign Exchange Deals
Purchase Traveler Cheques or gift
cheques,
Make Bill Payment, Teller Application Authenticate Client (PIN Verification),
Authenticate Client,
Perform CIS Authorization
Deposit Funds (Current Account)
Perform CIS Inquiry(2,4,5,6 or 7),
Perform Close Day and Extended
Retrieve Client Information, CIS Limits
hour Processes
and Account List and Balances (Uses APP
Batch Off
CIS 11 as primary path if the user has
Switch Workstation
a client card)
Query Client Session
Update Bank book Get Fee
Get Currency List
Insert Client Session Detail,
Get Denominations
Query Client/Transaction
(XML Files from Web
Session,
Current Account Server)
Retrieve end of day data by
branch - Cash and VISA Cash Advance , APP
merchant Retrieve Transit Make VISA Payment,
Post GL Entries envelope List VISA Cash Advance
(batch interface) transactions. Reversals,
Get Traveler (batch interface) VISA Payment Reversal
Cheque and
Currency Txn
Details
(batch interface)
APP
Journal APP APP APP

APP
General
Extract for EDW
Ledger
Strategies for a Successful E2E SIT 15
1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 16

You will be dealing with LOTS of people
Strategies for a Successful E2E SIT 17
The rest of the project management team
Project or Program Manager
Architects
Project Control

People associated with each application, e.g.,

Project leads
Project test leads and testers
Programmers/Architects

Management
People outside the project who will want information

QA, auditors, business sponsors, acceptors

Your own test team

Strategies for a Successful E2E SIT 18
 Make sure you have a strategy for each group
The rest of the project management Build relationships - lunch
team Status reports, regular status
meetings
People associated with each Test working group or forum
application get input & buy-in for strategy
SIT progress broadcasts
Management 1 on 1 s
Occasional status/strategy
presentations
QA, auditors, business sponsors, Approach them know who they
acceptors are and what they need

Strategies for a Successful E2E SIT 19

 Success depends on building relationships
Be aware of cultural dierences
Dierent technology groups may have very dierent processes
Dierent groups may have varied risk tolerances: e.g., front-line
business vs. nance or audit
Distributed teams in dierent countries

Understand what each group wants/needs from your tests

Work to get early buy-in for everything signicant in your plan
Develop your test strategy in a lightweight medium, and walk it
around ( socialize it)
If you have to publish a detailed document, try to get consensus
before you write it

Strategies for a Successful E2E SIT 20

Sta your own team appropriately

Strategies for a Successful E2E SIT 21

You won t have time to be hands-on!

Strong second-in-command
Day-to-day SIT test team management

Environment and data manager

Work with all participating teams

Small team of seasoned testers

Design tests; prepare test materials

Toolsmith(s)
Any automation you can do

Strategies for a Successful E2E SIT 22

1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 23

Establish the ground rules (Entry & exit criteria)
Environments ready, connected and tested

Coordinated data

Project-level tests complete, e.g.,

System and usability test
Performance
Near-neighbor interfaces

All must-x bugs, and all integration bugs of any severity,

resolved

Teams assigned (and set up in defect tracking system)

Communicate requirements and get early buy-in

Strategies for a Successful E2E SIT 24
 Track readiness
Your own team s deliverables
Test materials (test cases and scenarios)
Environments
Data
Initial setup of defect tracking system

System readiness
All participating systems
Readiness to meet SIT entry criteria

Report on readiness weekly using an easy-to-

understand graphic
Strategies for a Successful E2E SIT 25
A simple timeline for system readiness

Aug
May

Nov
Sep

Dec
Jun
Mar

Apr

Jan
Oct
Jul
4 11 18 25 1 8 15 22 29 6 13 20 27 3 10 17 24 1 8 15 22 29 5 12 19 26 2 9 16 23 30 7 14 21 28 4 11 18 25 2 9 16 23 30 6 13 20 27

Incremental Partial E2E Test with Main E2E Regression Test

Program E2E SIT Test integration App v0.4, then full test with Main App v1.0 (final
& pre-test with v0.9 (complete) fixes)

Main App v0.4 Main App v0.7 Main App v0.9 Main App v1.0 (final)
Proj 1 App 0
Proj 1 App 1

Proj 1 App 2 R6 Proj 1 App 2 R7

Proj 1 App 3

Proj 2 App1

Proj 2 App 2

Proj 3 App 1
Proj 3 App 2

Proj 3 App 3

Proj 3 App 4

Proj 3 App 5

Proj 3 Special Infrastructure

Proj 4 all other Business Apps

Proj 4 Enablement

Proj 4 Billing

Proj 5 App 1 Proj 5 App 2 Corp Billing GL Accounting

NOT YET PLANNED
Proj 6 App 1

Proj 6 App 2 Proj 6 App Proj 6 App 4

3
Proj 6 App 6
Proj 6 App 5

Proj 6 Proj 6
App 7 App 8

Legend: Completed - Ready for E2E Planned date Date at risk Late - new date not yet available Not yet planned

Assumptions Risks
1) Main App v0.9 will be full-featured and stable. 1) Required modifications to GL Journalling and back-end financial accounting have not yet
2) Project 3 Apps 2 & 3 will be E2E-ready no later than 12-Oct. been finalized or planned. (Change Request #29)
3) Starting E2E test without Project 3 Apps 4 & 5 will be feasible and will provide E2E test value.

Strategies for a Successful E2E SIT 26

 Verify readiness

Schedule a handover meeting with each team before

scheduled entry to SIT

Ask to see evidence (give advance notice of what

that should be)

Give teams a standard checklist to populate

Review the checklist and evidence with each team

when you meet

Strategies for a Successful E2E SIT 27

 Readiness Documentation Examples
Test requirement Criteria Sample Documentation

System/functional test Evidence of testing covering the Test cases & traceability matrix
functionality delivered in the Business acceptor sign-o on
release test coverage
IDs & detail for outstanding bugs
Exploratory test documentation
(charters, logs. Etc.)
IDs & detail for outstanding bugs

Regression test of systems Completed regression test Evidence that the required
currently in production and covering critical production testing has been done (risk
modied for this project functionality. assessment, test completion
checklist/report, etc.)
Sign-o by required IT and/or
business acceptors.

Strategies for a Successful E2E SIT 28

Readiness Documentation Examples
Test requirement Criteria Documentation

UAT (or Agile Acceptance Test) Completion of (User) Acceptance Business acceptor sign-o
Test covering business OR
acceptance criteria . Conditional acceptance by
business acceptors, pending nal
acceptance at the end of SIT.
All project test levels Outstanding bugs: Summary report showing
No outstanding must-x current status of all bugs logged
Review outstanding bugs with during testing
impacts on downstream data or Detailed bug reports for
function outstanding bugs, with analysis of
downstream impacts,
documented workarounds, and
resolution plans

Strategies for a Successful E2E SIT 29

1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 30

 Keep yourself informed
Take advantage of all the communication
channels

Make sure you are in the key program/project

level status meeting

Ask regularly for updates on system readiness

Review the program/project risk and issue logs

for anything that might aect your test

Chat to project managers and test leads

Strategies for a Successful E2E SIT 31
Build your network

Strategies for a Successful E2E SIT 32

 Outward communication
Make regular presentations to all the stakeholder
groups
SIT strategy
Progress reports

Set up a notice-board to post daily status (you ll
need this during E2E SIT execution)
Intranet, etc.

Set up a test question and answer forum for your
team s questions and share it on the notice-board
Strategies for a Successful E2E SIT 33
1 Project management
2 Applications
3 People
4 Test readiness
5 Information & communication
6 Environments & data
7 Questions & discussion

Strategies for a Successful E2E SIT 34

Strategies for a Successful E2E SIT 35
Some of your most dicult issues are here
Multiple platforms (hardware, OS, DBMS)

Other infrastructure, like batch schedulers and

message transports

Excusive use for your test of adequate test

environments

Data, co-ordinated across participating applications

Conguration management

Co-ordinated defect management

You need an Environment/Data Manager!
Strategies for a Successful E2E SIT 36
 Environment/Data Manager
Inventory all applications and infrastructure
elements

Work with you and SIT test team on data

strategy

Work with each team to develop plans

Environment plan
Data plan

Develop conguration management process

Administer conguration management process

during E2E test execution Strategies for a Successful E2E SIT 37
 Test Environments
Virtualize if you can but you may not be able to

Physical environments may not exist for every

application

There may not be connectivity where you need it

You may have to negotiate for exclusive use

If setup/purchase is required, there will be long lead

times

Start early! (Get that Environment Manager on it!)
Strategies for a Successful E2E SIT 38
 Coordinated Test Bed Data

This can be very dicult to achieve

A showstopper if you don t have it

You need a strategy and a plan

Start early! (Get that Environment Manager on it!)

Strategies for a Successful E2E SIT 39
 Conguration Management
Can be complex, with Production xes and other projects
going on at the same time as yours

You need an exact picture at all times

A system solution across platforms will probably not

work

Design a simple logical release process:

Start with an inventory of all the systems, platforms,
infrastructure elements participating in the test
Names, versions, etc.
Logical Release 1 is the starting inventory
Any time ANYTHING changes: application for defect resolution,
infrastructure upgrade, etc., update the inventory, and you re
now on Release 2

Strategies for a Successful E2E SIT 40

 Defect Management
Essential to have one defect management system for the
entire integration

Set it up so you can track by application /project/ vendor

Set up a SWAT team to handle defect diagnosis and

assignment
Neutral lead, e.g., overall Project Architect
Representative from each team

Make the Lead a regular attendee at your daily checkpoint

during test execution

Strategies for a Successful E2E SIT 41

Use project management to go from this

Strategies for a Successful E2E SIT 42

To something like this

Strategies for a Successful E2E SIT 43

Questions & discussion

Strategies for a Successful E2E SIT 44

 Fiona Charles
ona.charles@quality-intelligence.com
www.quality-intelligence.com
Twitter: @FionaCCharles

Images and text Fiona Charles 2012