Test Management Approach (TMap)

Chris Hampton, PMP Practice Manager, Software Control & Testing Sogeti USA September 2007
1

Agenda:
The State of, and Business Case for, Software Quality What is testing? Sogetis Test Management Approach (Tmap) Sogetis Test Process Improvement (TPI) Methodology

Poor State of Software Delivery

Software failure costs the American economy $59.5 billion annually.
Source: NIST (2002)

90% of projects are delivered late

66% of projects rated unsuccessful 54% of projects over budget
30% of projects cancelled early
Source: Standish 2003

The Business Case For QA

Compliance
Basel II, Sarbanes Oxley, HIPAA, etc Reduced liability from defective software and defective process

Cost Savings
Quality processes now reduce support costs later Lost revenue

Satisfaction
Higher User satisfaction Improved business alignment

Growth
Better quality and measurements allows development optimization Increase ability to deliver

Where Quality Problems Originate

7% 10%

56%

27%

Code Other Design Requirements

Source: James Martin

6

How Quality Provides Tremendous Savings

Source: Cutter Consortium/Forrester/Sogeti Planning Analysis System Design $140

Production $14000

Code Unit Test Unit $1000 Test

COST TO REPAIR DEFECT

Integration Code Test $2500

Certification Test

System Test $4500

$7000

Time cost and resource effort increase exponentially later in the lifecycle
November 7, 2013 7

What is testing?
Testing is a process aimed at:
Finding defects in a controlled manner Detecting the level of quality of the test object Demonstrating the gap between specifications and the

actual product
Demonstrating that the end product functions as called for in the requirements

Testing is not . . . .
Testing is not:
Implementation Acceptance Defect repair

A phase after development although testing includes a phase after development

Training on a new system

Why do we test ?
To mitigate business risks that have been identified:
Validate software quality Verify integration of business processes Reduce time-to-market Competitive purposes

To ensure business usability of the software:

Release low-error/known-error software

Move high quality software (meets or exceeds quality expectations) into production
Demonstrate the usefulness of new technology

10

Testing of information systems

What do we test?
Application Software Hardware System Software Procedures Documentation Functionality Continuity Performance Usability Interoperability (between
different applications and systems)

11

Structured Software Testing

Testing everything is impractical
Funding limitations Limited time and resources Diminishing value after a certain point

Isnt there a more effective way? Yes! Structured software testing: a risk-based, qualitycentric approach to testing

12

Test Management Approach (TMap)

Sogetis methodology for structured testing of software products.

TMap has evolved to be the standard for software testing in Europe and quickly gaining traction in the US.
It is being used by more than three hundred Dutch, Belgium and German organizations. Industry adoption includes :
Financial Services Insurance Government Consumer Electronics Telecommunications Medical systems
13

4 Essentials of TMap
Business-driven test management (BDTM) approach Structured test process Complete tool box Adaptive test method

BDTM puts focus on risk-based testingonly testing what has impacts to the organization.

14

Business Driven Test Management

Choices must be made in what is tested and how thoroughly. This is an economic decision based on:
The risks that an organization thinks it will incur The availability of time and resources The results the customer wishes to achieve.

The choices based on risks, result, time and cost constitutes the basis for the BDTM approach.
15

The BDTM Process

1. Formulate the test goals. 2. Determine the risk class for each combination of characteristic and object part. 3. Determining the depth of testing required.
Based on risk/resource trade off

4. Develop the test estimate and test plan. 5. Allocate the appropriate test techniques. 6. Throughout the test process, the test manager provides the stakeholders with insight into and control options over test process and test object.
16 16

Risk Definition
Testing should be based on the mitigation of risk and validation of expected quality defined by the business requirements Definition of risk
A risk is a chance of a failure occurring, related to expected damage to the project (should the failure occur)

Risk Categories
Risks identified in different categories such as business risks, project risks, product risks, process risks

Risk Ranking
Risk are ranked in criticality relative to one another, by instituting a method of risk ranking (or risk rating)

17

How Risk Ranking Works

1. Assemble a team of people representing various roles from the project
PMs, Business Leads/SMEs, Test Leads, Tech Leads

2. Create an initial list of risks 3. Team assigns a numeric value to each risk = the probability of occurrence of each risk 4. Team assigns a second value to each risk, representing the impact on the project/organization 5. Multiply the two values together (the probability of occurrence X the impact) 6. The result is a relative value for each risk 7. Order the risks by their relative risk values
risk rating or risk ranking Ranking helps manage the most critical risks, especially those falling in the middle tier of the ranking

8. Periodically update the list of risks

18

Risks quality characteristics matrix

Multi-Level Matrix
High Priority/ Medium Risk High Priority/ High Risk

High Priority

High Priority/ Low Risk

Quality Characteristics

Medium Medium Priority/ Medium Priority/ Medium Priority/ Medium Risk High Risk Priority Low Risk

Low Priority

Low Priority/ Low Risk

Low Priority/ Medium Risk

Low Priority/ High Risk

Low

Medium

High

Risks
19

Ten steps for creating a test case

Step 10 Step 9 Step 8

CREATE TEST CASES Select suitable test techniques

Associate test scenarios with risks Create test scenarios for each subsystem

Facilitated Sessions
Step 4 Step 3

Step 7 Step 6 Step 5

Create subsystem / quality characteristic pairings Rank relevant quality characteristics

Agree on relevant quality characteristics Rank risks

Step 2
Step 1

Identify risks Identify sub-systems / sub-processes

FUNCTIONAL AREA

20

Structured Test Process

Master Test Plan, managing the total test process
Acceptance and System Tests Developmental Test Supporting Processes

Lifecycle Model

21

Master Test Plan

Developed by the test manager in partnership with the appropriate stakeholders Jointly define:
What will be tested for each test level, When it will be tested To what degree of thoroughness.

This plan constitutes the basis for the test plans for the separate test levels.
Unit Testing Systems Integration Testing Acceptance Testing Developmental Testing
22

Life Cycle Model

The life cycle model is a generic model.
Can be applied to all test levels and test types Used in parallel with the life cycle models for system development.

Test activities are divided across seven phases:

Planning Control Setting up and maintaining infrastructure Preparation, Specification Execution Completion

23

TMap - Structured testing lifecycle

PREPARATION -Testability review -Allocate techniques -Requirements review -Data cycle test -Process cycle test -Other tests -Specify test cases -Create/assess infrastructure -Test specification -Data flow test EXECUTION -Pretest -(Re)test -Checking -Assessing -Debugger -Record and playback -Monitoring -Preserve testware -Evaluate process

SPECIFICATION -Create test scripts -Create infrastructure

S P&C

PLANNING & -Inspection and study -Develop test strategy oRisk analysis oTest Estimation (TPA) -Setup organization -Prepare test plan -Management and control

CONTROL -Test strategy -Test estimation -Reports -Management tools -Budgeting tools -Defect management tool

COMPLETION -Preserve testware for future use -Evaluate process

24

Complete Tool Kit

Techniques
Many techniques can be used in the test process. A test technique is a combination of actions to produce a test product in a universal manner.

TMap provides techniques for the following:

Test estimation Defect management Creating metrics Product risk analysis Test design Product evaluation.

25

Adaptive Test Method

Complete And Flexible Works with any SDLC
Iterative RAD Agile Waterfall RUP Mainframe Web ERP SOA

Works with any technology

26

27

TMap Differentiators
Business-Driven Test Management Risk-based test strategy Efficient testingcoverage without overlap Testing off of critical path as much as possible Criteria and metrics about production readiness Management of testing to project timelines Compliments industry tools

Proven methodology applied in hundreds of companies internationally over the past 14 years!
28

Benefits of Structure Testing

Defects are found early, costing less time/money to reach production delivering a higher quality product

Required quality of the various test objects is tested for and validated, by focusing on testing for quality as a risk mitigation strategy
By keeping a larger percentage of the testing process/effort off the critical path, faster time-to-market results Structured software testing is more cost-effective and efficient than non-structured testing approaches Sound test coverage is provided, without the need to overlap phases Establishes a test organization that is prepared and efficient Delivers a repeatable process
29

Test Process Improvement (TPI)

The Sogeti TPI Model was developed to facilitate a stepwise improvement of the testing process.
Developed from practical experience from 30+ years of Sogeti work in testing. Offers a frame of reference to determine the strengths and weaknesses of your current testing process. Covers 20 key areas that need specific improvement to achieve a well define testing process.

Key TPI Areas:

Test Strategy Life-Cycle Model Moment of Involvement Estimating and Planning Test Specification Techniques Static Test Techniques Metrics Test Automation Test Environment Office Environment Commitment and Motivation Test Functions and Training Scope of Methodology Communication Reporting Defect Management Testware Management Test Process Management Evaluation and Low-Level Testing
30

Typical TPI Process Steps

1. Interview key people 2. Describe the As-Is process including Strengths & Weaknesses 3. Describe the To-Be process 4. Document process improvement actions 5. Define expected benefits in terms of Objectives 6. Plan for Implementation of the Improvement Actions (short-term and long-term)

31

TPI - Test Maturity Matrix

Key area / Scale 1 2 3 4 5 6 7 8 9 Test strategy Life-cycle model Moment of involvement Estimating and planning Test specification techniques Static test techniques Metrics Test automation Test environment A A A A A A A A A A A B A A B C B B B B B C C C C D B A 0 1 A A A A B A A B B C C C D B B C C C D B B C B D 2 3 4 5 6 B 7 8 9 1 0 C 1 1 1 2 D 1 3

10 Office environment

11 Commitment and motivation 12 Test functions and training 13 Scope of methodology 14 Communication 15 Reporting 16 Defect management 17 Testware management 18 Test process management 19 Evaluation 20 Low-level testing

Increasing Maturity
B C B

32

Resources
Web Sites
Sogeti USA Web Site: www.us.sogeti.com TMap Web Site: http://eng.tmap.net/Home/

TMap certification (Netherlands)

Foundation TMap professional Advanced/Expert TMap management Advanced/Expert

Books

33

Sogeti SCT Services

Managed Testing Services (MTS) Quality Management Services Test Automation Services Testing Professional Services SAP Testing Services 3000+ people worldwide in SCT space 30+ yrs experience

34

Sogetis Geographical Presence

A workforce of over 18,000 staff spread over 200 locations
USA

Sweden

NL Belux

Ireland

UK

Denmark Germany Switzerland

France Spain India

35

Thank you!
Chris Hampton, PMP Practice Manager, Software Control & Testing Sogeti USA Chris.Hampton@us.sogeti.com Cell: (214) 549-7613 Sogeti Office: (972) 892-3400
36