Istqb TFC v2.3

03-23-05
Version 2.3
EDS ISTQB Testing Foundation Course
Presentation by Paul Weymouth, UK Testing ADU
Slide 1 • EDS Internal

How to Navigate the Course
• Click on the symbol to go to the next slide.
• Click on the symbol to go to the previous slide.
• Click on the symbol to go to the main Areas Covered slide.
• Click on the symbol to move on to associated slides.
• Click on the symbol to go back to the previous menu.
• You may also see the following symbols displayed :-
EDS EDS specific information available – used where an EDS exceptions exist.
If this appears top right, it means animation applies to the slide. It will also appear
bottom right after the final animation mouse click. For Tutor’s use only.
EDS ISTQB Testing Foundation Course Version 2.3 Slide 2 • EDS Internal
Other Symbols
Refers to the ISTQB Glossary Of Terms.

You will see this symbol appear alongside the
definition of a term found in this Glossary
These key terms are important to learn.
This is a ‘Pearl of Wisdom’. Typically a quote

from an external testing guru, that helps to re-
enforce what we have just learned. You’ll find
a few of these dotted around the course.
Objectives
• The primary objective of this course is to provide testing foundation

level training to staff involved in testing related activities (I.e.
developers, testers, managers)
• It is anticipated that after taking this course staff will be in a position to
take and pass the external ISTQB Foundation exam in Software Testing
Areas covered
Fundamentals of testing
Testing Throughout the Software Lifecycle
Static Techniques
Test Design Techniques
Test Management
Tool Support for Testing
Index of key terms
03-23-05
Version 2.3
Fundamentals of Testing

Fundamentals of Testing
Areas Covered
Why Testing is Necessary

What is Testing
General Principles of Testing
Fundamental Test Process
The Psychology of Testing
Summary
Software Systems – Some context

• Software Systems are now part of our everyday life
• They are used almost everywhere, for example in:
- Banking and Financial institutions
- Retail industry
- Central and Local Government
- Transport (e.g. Planes, Trains and Automobiles)
- Medicine (Hospitals, research centres)
- Home Entertainment
• We have all experienced Software Systems failing!
Software Systems – When things go wrong

Software System Failures can lead to:
– Human Injury or Death
• e.g. airplanes crashing
– Technological disasters
• e.g. Missile Systems malfunctioning
– Legal action and associated costs
• e.g. failure to meet contractual obligations
– Loss of face for suppliers and/or their
customers;
• e.g. mis-spelling company name on mail shots
Causes of Software Failure

• A Human can make an Error (aka a Mistake)
• An Error is ‘A Human Action that produces an Incorrect Result’
• The Error can cause a Defect (aka a Fault or Bug)
• A Defect is ‘A flaw in a component or system that can cause the
component or system to fail to perform its required function’
• A Defect can be in the Software, System or in a Document
* ISTQB Standard Glossary of Terms
Errors, Defects and Failures

• Defects occur because human beings are fallible
• Also because of:
– time pressure
– complex code
– complex infrastructure
– changed technologies
– and/or many system interactions

• A Defect may result in a Failure
• A Failure is a ‘Deviation of the component or system from its

expected delivery, service or result’
• Failures can be caused by environmental conditions as well
– E.g. radiation, magnetism, electronic fields
– Pollution can cause faults in firmware or influence the execution of
software by changing hardware conditions.
* ISTQB Standard Glossary of Terms

a human action that produces
Error an incorrect result
Can manifest as
A flaw in a component or
Defect system that can cause the
component or system to fail to
perform its required function
May result in
Deviation of the component or

Failure system from its expected
delivery, service or result
The Role of Testing

• Rigorous testing of systems and documentation can:
– reduce the risk of problems occurring in an operational environment
– contribute to the quality of the software system
• How? By finding and correcting defects before the system is released
for operational use
• Software testing may also be required to meet contractual or legal
requirements, or industry-specific standards
Testing and Quality

• Testing can help us measure the Quality of software
• Quality - ‘The degree to which a component, system or process
meets specified requirements and/or user/customer needs and
expectations’
• This is measured in terms of defects found
• Defects covering:
– functional software requirements and characteristics
– and non-functional software requirements and characteristics (e.g.
reliability, usability, efficiency, portability and maintainability)
• Testing can give confidence in the Quality of the software if it finds
few or no defects
Testing and Quality

• A properly designed test that passes, reduces the overall level of Risk
in a system
• Risk – ‘A factor that could result in future negative consequences;
usually expressed as impact and likelihood’
• When testing does find defects, the Quality of the software system
increases when those defects are fixed
• The Quality of systems can be improved through Lessons learned
from previous projects
• Analysis of root causes of defects found in other projects can lead to
Process Improvement
• Process Improvement can prevent those defects reoccurring
• Which in turn, can improve the Quality of future systems
• Testing should be integrated as one of the Quality assurance
activities
Testing Terminology
Testing Pearl of Wisdom
• “Quality is not intangible.

The purpose of testing is to make quality visible.
Testing is the measurement of software quality!"
Bill Hetzel 1988
Why exhaustive testing is impossible
• Exhaustive testing of complex software applications:

– requires enormous resources
– is too expensive
– takes too long
• It is therefore impractical
• Need an alternative that is pragmatic, affordable, timely and
provides results
Testing Terminology
• “In any form of testing it is impossible to achieve

total confidence.
The only exhaustive testing there is ………
is so much testing, that the tester is exhausted!"
Bill Hetzel 1988
Why don’t we test everything ?

System has 20 screens
Average 4 menus / screen
Average 3 options / menu
Average of 10 fields / screen
2 types of input per field
Around 100 possible values
Approximate total for exhaustive testing

20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
Test length = 1 sec then test duration = 17.7 days

Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 mins then test duration = 40 years!
So, How Much Testing is Enough?

• Deciding how much testing is enough should take account
of:
– the level of Risk
– project constraints such as time and budget
• Risks should be evaluated at the Business Level,
Technological Level, Project Level and Testing Level
• Risks are also used to decide where to start testing and
where more testing is needed
• Risk considerations can include:
– financial implication of software being released that is un-
tested (support costs / possible legal action)
– software being delivered late to market
– potential loss of Life (safety critical systems)
– potential loss of face (may have financial implications as
well)

• Risk analysis should be used to determine what to test in each
component and just as importantly what not to test
• For example, an unacceptable risk would say we must test, an
acceptable one perhaps not to test
• Testing is a risk-control activity that provides feedback to the
stakeholders
• With this feedback the stakeholders can make informed decisions
about the release of the software (or system) being tested
• More about Risks later in the Course
Why Testing Is Necessary

• Exit criteria is used to determine when testing at any stage is
complete
The set of generic and specific conditions, agreed upon with the stakeholders,
for permitting a process to be officially completed
• Exit criteria may be defined in terms of :-

– Thoroughness – i.e. coverage or requirements
– cost or time constraints
– percentage of tests run without incident
– number of faults remaining
What is Testing?
A Definition (and a Misconception)

• When asked, people often think that Testing only consists of running
tests, i.e. executing the software
• Testing – ‘The process consisting of all life cycle activities, both static
and dynamic, concerned with planning, preparation and evaluation of
software products and related work products to determine that they
satisfy specified requirements, to demonstrate that they are fit for
purpose and to detect defects.’
• Test execution is only a part of testing, but not all of the testing
activities
• Test activities exist before and after test execution
What is Testing?
A Definition (and a Misconception)

• Activities such as:
– Planning and control
– Choosing test conditions
– Designing test cases
– Checking results
– Evaluating completion criteria
– Reporting on the testing process and system under test
– Finalizing or closure (e.g. after a test phase has been completed)
• Testing also includes reviewing of documents (including source code)
and static analysis
Why Testing is necessary
• “Any activity that is undertaken with the

objective of helping us to evaluate or measure
an attribute of our software should be
considered a testing activity”
Hetzel 1998
What is Testing?
Test Objectives
• There are different test objectives:
– To find defects
– To gain confidence about the level of quality and to provide information
– To prevent defects
• Both dynamic testing and static testing can be used as a means for
achieving these objectives
• They provide information in order to improve:
– The system to be tested
– The development and testing processes
– Live operations (e.g. how long it takes for a process to run)
What is Testing?
Test Objectives
• By designing tests early in the project life cycle it can help to prevent
defects from being introduced into code
• Reviews of documents throughout the lifecycle (e.g. requirements and
design) also help to prevent defects appearing in the code. More
about this when we cover Static techniques
What is Testing?
Test Objectives – cost of fault correction
100
90
80
70
60
Relative 50
Cost
Multiples 40
30
20
10
0
Reqs Des Code Unit Accept Use
What is Testing?
Test Objectives
• The Objectives of testing can vary depending on the stage of testing
being conducted. E.g.:
– Development testing (e.g. component, integration and system testing) -

to cause as many failures as possible so that defects in the software are
identified and can be fixed
– Acceptance testing - to confirm that the system works as expected, to
gain confidence that it has met the requirements
– Maintenance testing - often includes testing that no new errors have been
introduced during development of the changes
– During Operational testing - may be to assess system characteristics such
as reliability or availability
What is Testing?
Test Objectives
• In some cases the main Objective of testing may be to assess the

quality of the software (with no intention of fixing defects), to give
information to stakeholders of the risk of releasing the system at a
given time
• More on these test stages later in the course
Testing Terminology
• “Testing is the process of executing a program or

system with the intent of finding errors”
Myers 1979
• "Testing is any activity aimed at evaluating an
attribute or capability of a program or system and
determining that it meets its required results"
Hetzel 1988
What is Testing?
Testing v Debugging
• Debugging and testing are different:
– Testing can show failures that are caused by
defects
– Debugging identifies the cause of a defect,
repairs the code and checks that the defect
has been fixed correctly
• Testing then ensures that the fix does
indeed resolve the failure
• The responsibility for each activity is very
different, i.e.
– Testers test
– Developers debug
General Testing Principles
The Seven Key Principles
1. Testing shows presence of Defects

2. Exhaustive Testing is Impossible!
3. Early Testing
4. Defect Clustering
5. The Pesticide Paradox
6. Testing is Context Dependent
7. Absence of Errors Fallacy

1. Testing shows the presence of Defects
• We test to find Faults (a.k.a Defects)

• As we find more defects, the probability of undiscovered defects
remaining in a system reduces.
• However Testing cannot prove that there are no defects present
Why Testing is necessary
Testing Pearls of Wisdom
• “The probability of the existence of more errors in a section of

a program is proportional to the number of errors already
found in that program”
• “Do not plan a test effort under the tacit assumption that no
errors will be found”
• “A good test is one that has a high probability of detecting an
as yet undiscovered error”
• “A successful test is one that detects an as-yet undiscovered
error”
Myers 2004

2. Exhaustive Testing is Impossible!
• We have learned that we cannot test everything (i.e. all combinations

of inputs and pre-conditions).
• That is we must Prioritise our testing effort using a Risk Based

Approach.

3. Early testing
• Testing activities should start as early as possible in

the development life cycle
• These activities should be focused on defined
objectives – outlined in the Test Strategy
• Remember from our Definition of Testing, that
Testing doesn’t start once the code has been written!

4. Defect Clustering
• Defects are not evenly spread in a system

• They are ‘clustered’
• In other words, most defects found during testing are usually confined to a small
number of modules
• Similarly, most operational failures of a system are usually confined to a small

number of modules
• An important consideration in test prioritisation!

5. The Pesticide Paradox
• Testing identifies bugs, and programmers respond to fix them

• As bugs are eliminated by the programmers, the software improves
• As software improves the effectiveness of previous tests erodes
• Therefore we must learn, create and use new tests based on new techniques to catch
new bugs
• N.B It's called the "pesticide paradox" after the agricultural phenomenon, where bugs such as the
boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful
pesticides followed by ever-more powerful bugs or an altogether different approach.’ – Beizer 1995

6. Testing is Context Dependent
• Testing is done differently in different contexts

• For example, safety-critical software is tested differently from an e-commerce site
• Whilst, Testing can be 50% of development costs, in NASA's Apollo program it was 80% testing
• 3 to 10 failures per thousand lines of code (KLOC) typical for commercial software
• 1 to 3 failures per KLOC typical for industrial software
• 0.01 failures per KLOC for NASA Shuttle code!
• Also different industries impose different testing standards

7. Absence of Errors Fallacy
• If we build a system and, in doing so, find and fix defects ....
It doesn’t make it a good system
• Even after defects have been resolved it may still be unusable and/or
does not fulfil the users’ needs and expectations
The five stages of the fundamental test process
• Test Planning and Control
• Test Analysis and Design
• Test Implementation and

Execution
• Evaluating Exit Criteria and

Reporting
• Test Closure Activities

• The process always starts with planning and ends with test closure
activities
• Each phase may have to be executed a number of times in order to
fulfil exit or completion criteria
• Although logically sequential, the activities in the process may
overlap or take place concurrently
Test Planning and Control

Test Planning
• Specifies how the test strategy and project Test Plan
A document describing the scope, approach, resources and schedule of
intended test activities
apply to the software under test
• Principally:
– verify the mission
– define the Test objectives
– Specify the Test Activities required to meet the mission and objectives

Test Planning (continued)
• Major Tasks are :-
– Identify the objectives of testing
– Determine Scope
– Determine the Test Approach
– Determine the required test resources
– Implement the test policy and/or the test strategy
– Schedule test analysis and design tasks
– Schedule test implementation, execution and evaluation
– Determine the Exit Criteria
• More on Test Planning in Test Management section

Test Control
• The ongoing activity of comparing actual progress against the plan
• Reporting status, including deviations from the plan
• Taking actions necessary to meet the mission and objectives of the
project
• Test Planning takes into account the feedback from monitoring and
control activities.
• Major Tasks are :-
– measure and analyse results
– Monitor and document progress, test coverage and exit criteria
– initiate corrective actions
– make decisions
• More on Test Control in Test Management section
Test Analysis and Design

• General testing objectives are transformed into tangible Test Conditions
(An item or event of a component or system that could be verified by one or more test cases, e.g.
a function, transaction, feature, quality attribute, or structural element)
and Test Designs (A document specifying the test conditions (coverage items) for a test item,
the detailed test approach and identifying the associated high level test cases).
• Tests should be designed using the test design techniques selected in the
test planning activity
• Major tasks are:
– Review the test basis
– From Analysis of test items, identify Test Conditions/Requirements and
required Test Data
– Design the tests (note – the detail, in the form of a Test Case, is developed in
the next stage)
– Evaluate testability of the requirements and system
– Design the test environment set-up
– Identify any required infrastructure and tools
• “The act of designing tests is one of the most

effective error prevention mechanisms known
... The thought process that must take place to create
useful tests can discover and eliminate problems at
every stage of development.“
Beizer 1983
Test Implementation and Execution

• Test Conditions are transformed into Test Cases and Testware
• The test environment is created
Create Test Cases
– Develop and prioritise Test Cases
– Create test data
– Write test procedures
– Preparing test harnesses
– Write automated test scripts
– Create test suites from the test cases for efficient test execution
Check the Environment
– Verify that the test environment has been set up correctly
• Each Test Case is specified in terms of :-

– its objective
– the initial state of the system
– the input
– the expected result.
• Major tasks (continued):
Execute the Tests

– Execute the Test Cases according to the planned sequence.
– Log the outcome of test execution
– Test execution records should uniquely identify the versions of the
software under test, test tools and Testware
– It should be possible to establish that all testing has been carried out by
reference to the test records.

• Major tasks (continued):
Check the Results

– Compare actual results with expected results
– Report discrepancies as Incidents
– Analyse Incidents to establish Root cause
– Repeat, as necessary, test activities as result of
action taken for each discrepancy
– The test coverage levels achieved for those
measures specified as test completion criteria
should be recorded.
•“Thoroughly inspect the results of each test”
Myers - 2004
Ref: Myers, The Art of Software Testing, J Wiley and Sons, 1979
Evaluating Exit Criteria and Reporting
• Test execution is assessed against the objectives defined in Test

Planning
• This should be done for each Test Level (i.e. test stage)
A group of test activities that are organized and managed together.
– Check test logs against the exit criteria specified in Test Planning
– If the exit criteria has not been met
• Assess if more tests are needed
• Assess which test activities may need to be repeated
– Assess if the exit criteria specified should be changed
– Produce a test summary report for stakeholders review
Test Closure Activities
• Collect data from completed test activities to consolidate experience,

Testware, facts and numbers
– Check which planned deliverables have been delivered
– Check that Incident reports status are up-to-date (e.g. Closed)
– Ensure all Incident reports have associated change records
– Record acceptance of the system
– Finalise and archive Testware, the test environment and the test
infrastructure for later reuse
– Handover Testware to the maintenance organization
– Analyse lessons learned for future releases and projects, and for process
improvement.
5 Phases of the Fundamental Test Process
Fix test design and repeat

Fix component or test cases/scripts
and repeat
Test Test Test Evaluating Exit Test Closure

Planning Analysis Implementation Criteria and Activities
and and Design and Execution Reporting
Control
Fix test design and repeat
Fix component test plan and repeat
Psychology of Testing
The Testing Mindset
• Historically testing was viewed as showing the system meets its requirements
• This has evolved to a stage where testing is performed with the primary aim
of finding faults rather than proving correctness. It is perceived as a
destructive process
• Seeking to find failures (the right mindset) can be viewed as criticism of the
product and/or its author
• But looking for failures is constructive!
– Time can be saved
– Risks reduced
– Costs reduced
– Skills improved
The Testing Mindset
• It is important that the Objectives of testing are clearly understood

as humans will moderate their behaviour accordingly (however sub-
consciously):-
– “If testing is showing the system meets its requirements then I will just
produce tests that show this.”
– “If testing is aimed at finding faults then I will be measured on this so I
will put effort into designing tests that are more likely to find faults.”
• The Testing mindset is different from a Developer’s
• “Testing is an extremely creative and

intellectually challenging task”
• “Tests must be written for invalid and

unexpected, as well as valid and expected,
input conditions”
Myers - 1979
Traits of Good Testers

• A Tester needs:
– good communication skills
– good observation skills
– people handling skills
– Curiosity
– patience
– reliability
– thoroughness
– an inquisitive nature
– attention to detail
– creativity in terms of identifying likely faults
– Experience
• However as with most other disciplines an effective
test team will need a mix of skills so it is difficult to
generalise
Developer vs Tester Relationship
• The relationship between a Developer and a Tester is

not normally an easy one because:-
– testers point out problems with software
– developers like to think their software is perfect
– testers are perceived as delaying the project by
finding faults in the system
– when the development slips testers normally have to
work long hours to test the product, which in turn can
cause resentment.
• It is important that they work together
• It is also important that they have mutual respect
for each other.
• Collaboration is the right approach – we work to a
common goal!
• Communicate findings objectively, not subjectively
Independent testing
• The right mindset could enable Developers to test the code
• However, passing this responsibility to trained and professional
testing resources has many benefits (such as higher defect find
rates)
• Authors tend to bring across assumptions they have made when
developing the software. They are less likely to write tests to show
faults in their own software (human nature)
• With testing performed by independent testers, testing effort is
focused and not compromised by development effort and bias
• It is generally believed that objective independent testing is more
effective
Independent testing
• There are several levels of Independence (from Low to High)
– Tests designed by the person(s) who wrote the software under
test
– Tests designed by another person(s) (e.g. from the
development team).
– Tests designed by a person(s) from a different organizational
group (e.g. an independent test team).
– Tests designed by a person(s) from a different organization or
company (e.g. outsourcing to an in-house or external test
specialist organisation)
Fundamentals of Testing - Summary
• We learned of the possible effects of Defects on people, on the

environment and on companies
• We learned of the causes of a Defect and the results –

Error/Defect/Failure – remember these and alternative terms
• We learned why testing was necessary – how it improves Quality

and reduces Risk of failure
• We learned how root cause analysis leads to process improvement
• We learned why you can’t test everything and when to stop testing
– through Risk Analysis, Prioritisation and use of Exit Criteria
• We learned about Testing:
– its main Objectives, i.e. to find and prevent defects and to gain confidence in
system Quality
– How we meet these Objectives
– How they can vary by Test Level
– That it is conducted before and after the code is delivered
– What activities it comprises
– That Debugging and Testing are different
• We learned of the 7 Key Principles of Testing:
– Testing shows presence of Defects

– Exhaustive Testing is Impossible!
– Early Testing
– Defect Clustering
– The Pesticide Paradox
– Testing is Context Dependent
– Absence of Errors Fallacy
• We learned the Fundamental Test process:
– The Stages:
• Test Planning and Control
• Test Analysis and Design
• Test Implementation and Execution
• Evaluating Exit Criteria and Reporting
• Test Closure Activities
– And the main tasks for each stage of the process
• And we learned about the Psychology of Testing:
– What is the right testing mindset (looking for bugs!)

– Why it is constructive not destructive!
– The skills needed for a good tester
– Tester and Developer relationship issues
– The benefit and types of Independent testing
03-23-05
Version 2.3
Testing Throughout the Software Life-cycle

Testing Throughout the Software Lifecycle
Software Development Models

Test Levels
Test Types – the Targets of Testing
Maintenance Testing
Summary
Software Development Models
The V-Model
Iterative Development Models
Testing within a Lifecycle Model
V-Model
User/Business Acceptance Test Acceptance
Requirements Plan Test
System Test System

System
Plan
Requirements Test
Integration Integration
Technical
Test Plan
Test
Specification
Unit Test Test

Development Program Unit
Plan
Levels Levels
Specification Test
Coding
V-Model
The benefits of the V Model include:
• The testing phases are given the same level of management

attention and commitment as the corresponding development
phases
• The outputs from the development phases are reviewed by the
testing team to ensure their testability
• Verification and validation (and early test design) can be carried out
during the development of the software work products
• The early planning and preliminary design of tests provides
additional review comments on the outputs from the development
phase
V-Model
• The levels of development and testing shown in the model vary

from project to project
• For example, there may additional test levels, such as System
Integration Testing, sitting between System Testing and Acceptance
Testing (more on these test levels later)
• The work products coming out from any one development level may
be utilised in one or more test levels
• For example, whilst the prime source for Acceptance testing is the
Business Requirement, the System Requirements (e.g. Use Cases)
may also be needed to support detailed test design
V Model
• “The V Model provides a powerful tool for managing and

controlling risk within the testing component of a project.
The process of bringing testing planning and design into the

development process as early as possible enables risks to
be identified and strategies for removing or mitigating them
to be put in place in a timely manner. ”
Watkins - 2001
• Rapid Application Development
User
Requirements
Code
Acceptance
Test
Testing within a Lifecycle Model
Characteristics of good testing in any life cycle model:
A Test Level exists for every development Level

Each Test Level has specific objectives
Test analysis and design for each Test Level begins during corresponding development Level
– Early and proactive involvement of testers in reviewing development deliverables – benefits both parties
–
–
–
Test Levels should be adapted depending on Project nature. May be better to combine Test Levels, e.g. with COTS testing.
Test Levels
Component Testing
Integration Testing
System testing
Acceptance Testing
Component Testing
System Test System

System
Plan
Requirements Test
Technical
Test Plan
Test
Specification
Unit Test Test

Development Program Unit/Component
Plan
Levels Levels
Specification Test
Coding
Component Testing
Definition
• Component – A minimal software item that can be tested in isolation.

• Component Testing – The testing of individual software components.
• Sometimes known as Unit Testing, Module Testing or Program Testing
• Component can be tested in isolation – stubs/drivers may be
employed
• Test cases derived from component specification (module/program
spec)
• Functional and Non-Functional testing
• Usually performed by the developer, with debugging tool
• Quick and informal defect fixing
Component Testing
Definition
• Test-First/Test-Driven approach – create the tests to drive the design and

code construction!
• Instead of creating a design to tell you how to structure your code, you
create a test that defines how a small part of the system should function.
• Three steps:
1. Design test that defines how you think a small part of the software should
behave (Incremental development).
2. Make the test run as easily and quickly as you can. Don't be concerned about
the design of code, just get it to work!
3. Clean up the code. Now that the code is working correctly, take a step back
and re-factor to remove any duplication or any other problems that were
introduced to get the test to run.
Russell Gold, Thomas Hammell and Tom Snyder - 2005
Integration Testing
Definition
Component Integration Testing
System Integration Testing
Integration Testing
Definition
• Integration Testing - Testing performed to expose

defects in the interfaces and in the interactions between
integrated components or systems
• Components may be code modules, operating systems,
hardware and even complete systems
• There are 2 levels of Integration Testing
– Component Integration Testing
– System Integration Testing
System Test System

System
Plan
Requirements Test
Technical
Test Plan
Test
Specification
Unit Test Test

Plan
Levels Levels
Test
Specification
Coding
Definition
• component integration testing Testing performed to expose

defects in the interfaces and interaction between integrated
components
• performed by the test team
• usually formal (records of test design and execution are kept)
• all individual components should be integration tested prior to
system testing
Test Planning
• To consider - should the integration testing approach:
– Start from top level components and work down?

– Start from bottom level components and work up?
– Use the big bang method?
– Be based on functional groups?
– Start on critical components first?
– Be based on business sequencing? Maybe suit System Test needs.
• Knowledge of the system architecture is important

• The greater the scope of the integration approach the more difficult
it is to isolate defects
• Non-Functional requirements testing may start here – e.g. early
performance measures
Top-down testing
Component
under test
P
Q R
S T U V
Stubs
Top-down testing
Pro’s Con’s
• provides a limited working system • stubs only provide limited
early in the design process simulations of lower level
• depth first integration demonstrates components and could influence
end-to-end functions early in the spurious results
development process • breadth first means that higher
• early detection of design errors levels of the system must be
through early implementation of the artificially forced to generate
design structure output for test observations
• early testing of major control or
decision points
Bottom-up testing
Component under test
P
P is the driver
for components
Q and R
Same for Q
and R
Q R driving their
components
S T U V
Bottom-up testing
Pro´s Con´s
• using drivers instead of upper • unavailability of a demonstrable
level modules to simulate the system until late in the
environment for lower level development process
modules
• late detection of system
• necessary for critical, low level
system components structure errors
• testing can be observed on the

components under test from an
early stage
Big Bang Approach
Main Menu Function 1 Function 2 Function 3
Component 1 Component 2 Component 3 Component 4 Component 5 Component 6
Not usually the preferred approach
Suggested Integration Testing Methodology
The following testing techniques are appropriate for Integration

Testing:
• Functional Testing using Black Box Testing techniques against the
interfacing requirements for the component under test
• Non-functional Testing (where appropriate, for performance or
reliability testing of the component interfaces, for example)
• We’ll talk about System Integration Testing later.
• For now, we should stick to the sequence of the Test lifecycle.
• Which means System Testing next.
System Testing
Context
Definition
Functional Systems testing
Non-Functional Systems Testing
Good Practices for System Testing
System Testing
System Test System

System
Plan
Requirements Test
Technical
Test Plan
Test
Specification
Unit Test Test

Plan
Levels Levels
Test
Specification
Coding
System Testing
Definition
• System Testing - process of testing an integrated system to

verify that it meets specified requirements
• Concerned with the behaviour of the whole system, not with

the workings of individual components.
• Carried out by the Test Team
Functional System Testing
Definition
Requirements-based functional testing
Business process functional testing
A Functional requirement is (per IEEE):
‘A requirement that specifies a function that a system or

system component must perform’
• A Requirement may exist as a text document and/or a

model
Requirements-based functional testing - Functionality

Accuracy Provision of right or agreed results or effects
Interoperability Ability to interact with specified systems
Compliance Adhere to applicable standards, conventions, regulations or

laws
Auditability Ability to provide adequate and accurate audit data
Suitability Presence and appropriateness of functions for specified

tasks
Requirements based testing
• testing against requirements and specifications

• test procedures and cases derived from:
– detailed user requirements
– system requirements functional specification
– User documentation/instructions
– high level System design
Requirements-based functional testing - techniques
• Starts by using the most appropriate black-box testing techniques

• May support this with white-box techniques (e.g. menu structures,
web page navigation)
• Risk based approach important
Business Process based testing
• test procedures and cases derived from:

– expected user profiles
– Business scenarios
– use cases
• testing should reflect the business environment and processes in
which the system will operate.
• therefore, test cases should be based on real business processes.
• “System testing is the most misunderstood and most

difficult testing process ”
Myers - 2004
Non-functional System Testing
Definition
Non-functional requirements
Non-functional test types
Definition
• testing of those requirements that do not relate to functionality
• Emphasis on non-functional requirements:

– Performance
– Load
– Data volumes
– Storage
– Recovery
– Usability
– Stress
– Security*
• * Note that ISTQB treats this as a Functional test. From the syllabus:
– ‘Security Testing A type of functional testing, security testing, investigates
the functions (e.g. a firewall) relating to detection of threats, such as viruses,
from malicious outsiders.’
• The non-functional aspects of a system are all the attributes other than
business functionality, and are as important as the functional aspects.
These include:
– the look and feel and ease of use of the system
– how quickly the system performs
– how much the system can do for the user
• It is also about:
– how easy and quick the system is to install
– how robust it is
– how quickly the system can recover from a crash
Non-functional test types
• Reliability - The capability of software to maintain its level of

performance under stated conditions for a stated period of time. Is the
software product reliable?
• Usability - Is the software product easy to use, learn and understand
from the user’s perspective?
• Maintainability: The effort needed to make specified modifications. Is the
software product easy to maintain?
• Efficiency: The relationship between the level of performance of the
software and the amount of resources used, under stated conditions. Does
the software product use the hardware, system software and other
resources efficiently? Is the number of resources required by the software
product during use affordable and acceptable?
• Portability: The ability of software to be transferred from one environment
to another. Is the software product portable?
System Testing
Good Practices for System testing
• implement documented procedures for requirements analysis,

control and traceability
• review deliverables to ensure feasible, testable requirements and

associated acceptance criteria
• trace requirements to the design and tests which prove the

requirement has been met
• test data, facilities and documentation must be sufficient to

demonstrate conformance with requirements
• Test environment must closely mirror the target production system
Context
Definition
Objectives
Interfaces to External Systems

Plan System
Requirements Test
Integration
Testing
System Test System
System
Plan
Requirements Test
Technical
Test Plan
Test
Specification
Unit Test Test

Plan
Levels Levels
Test
Specification
Coding
Definition
• System Integration Testing is testing between the ‘System’ and

‘Acceptance’ phases.
• The System has already proven to be functionally correct, what
remains to be tested is how the system reacts to other systems
and/or organisations.
Objectives of Systems Integration Testing

• The objective of System Integration Testing is to provide confidence
that the system or application is able to interoperate successfully with
other specified software systems and does not have an adverse affect
on other systems that may also be present in the live environment, or
vice versa
• It is possible that the testing tasks performed during System Integration

Testing may be combined with System Testing, particularly if the
system or application has little or no requirement to interoperate with
other systems
• In terms of the V Model, Systems Integration Testing corresponds to the

Functional and Technical Specification phases of the software
development lifecycle
Testing Interfaces to External Systems

• Having completed Component integration testing and Systems testing, one
must execute the plan for system-to-system integration
• Infrastructure may need to be transformed in order to feed to an external

system
• Black Box testing techniques used
Acceptance Testing
Context
Definition
User Acceptance Testing
Operational Acceptance Testing
Contract/Regulation acceptance testing
Alpha and Beta testing
Other Acceptance Test terms
Acceptance Testing
System Test System

System
Plan
Requirements Test
Technical
Test Plan
Test
Specification
Unit Test Test

Plan
Levels Levels
Test
Specification
Coding
Acceptance Testing
Definition
• Acceptance testing: Formal testing with respect to user

needs, requirements, and business processes conducted to
determine whether or not a system satisfies the acceptance
criteria and to enable the user, customers or other
authorized entity to determine whether or not to accept the
system.
Acceptance Testing
Definition
• Usually the responsibility of the Customer/End user, though other
stakeholders may be involved. Customer may sub-contract the Acceptance
test to a third party
• Goal is to establish confidence in the system/part-system or specific non-
functional characteristics (e.g. performance)
• Usually for ensuring the system is ready for deployment into production
• May also occur at other stages, e.g.
– Acceptance testing of a COTS product before System Testing commences
– Acceptance testing a component’s usability during Component testing
– Acceptance testing a new significant functional enhancement/middleware release
prior to deployment into System Test environment.
Acceptance Testing
User Acceptance Testing (UAT)

• Usually the final stage of validation
• conducted by or visible to the end user and customer
• testing is based on the defined user requirements
• Often uses the ‘Thread Testing’ approach:
• ‘A testing technique used to test the business functionality or business
logic of the application in an end-to-end manner, in much the same way a
User or an operator might interact with the system during its normal use.’
- Watkins 2001
This approach is also often used for Functional Systems Test -

The same Threads serve both test activities
Acceptance Testing
• Often use a big bang approach

• black box testing techniques most commonly used
• Regression testing to ensure changes have not regressed other areas of the
system
Acceptance Testing
• “If love is like an extended software Q.A. suite, then true

love is like a final Acceptance Test – one often has to be
willing to endure compromise, bug fixes and work-arounds;
otherwise, the software is never done ”
The Usenet Oracle
Acceptance Testing
Operational Acceptance Testing (OAT)
• The Acceptance of the system by those who have to administer it.
• Features covered include:

– testing of backup/restore
– disaster recovery
– user management
– maintenance tasks
– periodic checks of security vulnerabilities
• ‘The objective of OAT is to confirm that the Application Under Test (AUT)
meets its operational requirements, and to provide confidence that the
system works correctly and is usable before it is formally "handed over" to
the operation user. OAT is conducted by one or more Operations
Representatives with the assistance of the Test Team’ 1 –Watkins 2001
Acceptance Testing
Operational Acceptance Testing (OAT)
• Employs a Black Box Approach for some activities

• Also employs a Thread Testing approach – Operations representatives
performing typical tasks that they would perform during their normal usage
of the system
• Also addresses testing of System Documentation, such as Operations
manuals
Acceptance Testing
Contract/Regulation Acceptance Testing
• Contract Acceptance Testing - testing against the acceptance criteria

defined in the contract
– final payment to the developer depends on contract acceptance testing being
successfully completed
– acceptance criteria defined at contract time are often imprecise, poorly defined,
incomplete and out-of-step with subsequent changes to the application
• Regulation Acceptance testing is performed against any regulations which
must be adhered to, such as governmental, legal or safety regulations
Acceptance Testing
Alpha & Beta Testing
• early testing of stable product by customers/users

• feedback provided by alpha and beta testers
• alpha tests performed at developer’s site by customer
• beta tests conducted at the customer site by end user/customer
• published reviews of beta release test results can make or break a product
(e.g. PC games)
Acceptance Testing
Other Acceptance test terms
• Factory Acceptance Testing (FAT)
• Site Acceptance Testing (SAT)
• Both address acceptance testing for systems that are tested before and
after being moved to a customer’s site
Test Types – The Targets of Testing
Definitions
Functional Testing
Non-Functional Testing
Structural Testing
Confirmation & Regression Testing
Definitions
• Target For Testing - A group of test activities aimed at verifying the

software system (or a part of a system) based on a specific reason.
• Test type - A group of test activities aimed at testing a component or
system regarding one or more interrelated quality attributes. A test
type is focused on a specific test objective, e.g.
– reliability test
– usability test
– Structure or architecture of the system/software
– regression test
and may take place on one or more test levels or test phases.
Definitions
• A model of the software may be developed and/or used in structural

and functional testing
• For example, in functional testing
– a process flow model
– a state transition model
– a plain language specification
• and for structural testing
– a control flow model
– a menu structure model
Functional Testing
• functional testing: Testing based on an analysis of the specification of

the functionality of a component or system.
• ‘Specification’ – E.g. Requirements specification, Use Cases, Functional

specification or maybe undocumented.
• ‘Function’ – what the system does
• Functional test – based on the Functions and features – may be applied

at all Test levels (e.g. Component Test, System Test etc)
• Considers the external (not internal) behaviour of the software. Black-

Box testing. What it does rather than how it does it. More on this later!
Non-Functional Testing
• non-functional testing: Testing the attributes of a component or

system that do not relate to functionality, e.g. reliability, efficiency,
usability, interoperability, maintainability and portability
• May be performed at all Test levels (not just Non Functional Systems
Testing)
• Measuring the characteristics of the system/software that can be

quantified on a varying scale- e.g. performance test scaling
Structural Testing
• Structural testing: Testing based on an analysis of the internal

structure of the component or system
• Also known as White Box Testing or Glass Box Testing
• May be performed at all Test levels but more commonly during
Component Test and Component Integration Test
• Coverage measured as a % of items tested – i.e. how much the
structure has been tested
• May be based on the system Architecture – e.g. a calling hierarchy
• Need for use of Test Tools – e.g. for testing coverage of Statements
and Decision in the code
• More on White Box testing and Coverage later
Confirmation (Re-Testing) and Regression Testing
Re-testing: Testing that runs test cases that failed the last time
they were run, in order to verify the success of corrective
actions
Regression Testing: Testing of a previously tested program

following modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the software,
as a result of the changes made. It is performed when the
software or its environment is changed.
Confirmation Testing (Re-Testing)
• Whenever a fault is detected and fixed then the software should be

re-tested to show that the original fault has been fixed. This is
known as Re-Testing.
• It is important that the test case is repeatable.
• In order to support this the test identifier should be included on the
fault report.
• It is important that the environment and test data used are as close
as possible as those used during the original test.
Regression Testing
• If the test is re-run and passes you cannot necessarily say the fault
has been resolved because ..
• You also need to ensure that the modifications have not caused
unintended side-effects elsewhere and that the modified system still
meets its requirements – Regression Testing
• Regression testing should be carried out :-

– when the system is stable and the system or the environment changes
– when testing bug-fix releases as part of the maintenance phase
• It should be applied at all Test Levels
• It should be considered complete when agreed completion criteria
for regression testing have been met
• Regression test suites evolve over time and given that they are run
frequently are ideal candidates for automation
Regression Testing
• Selecting suitable tests involves :-
– knowledge of the bug fixes and how they affect the system
– understanding the areas that have frequent faults
– understanding which areas of the system have undergone the most
recent changes
– understanding the areas of the system which are most critical to the
user
– understanding the core features of the system which must function
correctly.
• The effectiveness of a regression test suite can diminish over time
for a number of reasons :-
– tests are added for short term goals but not removed
– tests become redundant due to functionality changes
– test suite is not updated when major functionality changes are
implemented
– execution time becomes prohibitively high
– maintenance of the test suite becomes prohibitively high.
Regression Testing
• Reduction in effectiveness can be countered by :-

– maintaining cross references between system features and their
corresponding tests
– monitoring the addition of tests to the suite
– Periodic review and removal of redundant tests
– review of the test suite when major enhancements are made to the
system
– evaluation of the effectiveness of the test suite using metrics.
Regression Testing
“ The probability of making an incorrect change is more than 50 %.

Much of this is due to overconfidence and ineffective or nonexistent
software change testing.
We change just a couple of statements and believe we have not affected
anything adversely.
We execute one case that tests the path that was changed and tell
ourselves that the change has been tested.
IS IT, THEN, ANY WONDER THAT WE EXPERIENCE SO MANY PROBLEMS?! ”

Hetzel 1998
Maintenance Testing
What is Maintenance testing?
Objectives of Maintenance testing
Problems of Maintenance testing
Concerns of Maintenance testing
How can we test changes?
Maintenance Testing
What is Maintenance Testing?

• Maintenance testing: Testing the changes to an operational system or the
impact of a changed environment to an operational system
• testing changes to a Live System
• Triggered by, for example,
– Modification
• software upgrades
• Operating system changes
• system tuning
• emergency fixes
– software Retirement (may necessitate data archiving tests)
– Migration
• System migration (including operational tests of new environment plus
changed software)
• database migration
Maintenance Testing
Objectives of Maintenance Testing

• Develop tests to detect problems prior to placing the change into
production
• Correct problems identified in the live environment
• Test the completeness of needed training material
• Involve users in the testing of software changes
Maintenance Testing
Problems of Maintenance testing
• all that is available is the source code (usually with

poor internal documentation and no record of
testing) – poor or missing specifications
• program structure, global data structures, system
interfaces and performance and/or design
constraints are difficult to determine and frequently
misinterpreted
• Baselined test plans and/or regression test packs
often not updated
Maintenance Testing
Concerns of Maintenance testing

• will the testing process be planned?
• will testing results be recorded?
• will new faults be introduced into the system?
• will system problems be detected during testing?
• how much regression testing is feasible?
• will training be considered?
Maintenance Testing
How can we test changes?

• Maintenance testing involves testing what has been changed (i.e. Re-
Testing)
• It also, importantly, utilises Impact Analysis as a method for determining
what Regression testing is required for the whole system
• Traceability of Testware to source documents essential for effective impact
analysis (we cover this more in a later topic)
• Scope of Maintenance tests based on Risk assessment – including size of
change and size of system
• Maintenance testing may involve one or more test levels and one or more
test types
Testing throughout the Software Lifecycle
Summary
• Firstly we looked at Software Development Models:
– The V-Model – identifying the stages of testing, their relationship to the
Development stages and the type of work products involved
– Iterative Development Models, as used in RAD, RUP and Agile
developments
– Also the characteristics that make for good testing in ANY life cycle model
– And that development models must be adapted to the context of project
and product characteristics
• Next we talked about the different Test Levels:

– Component Testing – testing of individual software components by the
development team
– Integration Testing at Component level – looking at different approaches
such as Top-Down, Bottom-up and Big Bang
– System Testing
• Functional System Testing – requirements and business process based
• Non-Functional System Testing – testing the non-functional attributes
of a system
– System (level) Integration Testing - testing that a system interoperates
with other systems or organisations
Summary
• And still under Test Levels ….

– Acceptance Testing, comprising:
• User Acceptance Testing
• Operational Acceptance Testing
• Contract and Regulation testing
• Alpha and Beta testing
• Next we talked about Test Types

– A group of testing activities aimed at testing one or more quality attributes
of the system, such as:
• Functional Testing
• Non-Functional Testing
• Structural testing
• Regression Testing
• Re-testing (confirmation)
– All Test Types can be performed at all test levels
Summary
• An finally we talked about Maintenance testing

– Testing the changes (software and environmental) to an operational live
system
– What the reasons (or triggers) are for Maintenance Testing
– We looked at the objectives of Maintenance Testing
– the problems that can be encountered during Maintenance testing
– and what we should consider for our Maintenance testing approach such as
Impact Analysis and Regression testing
03-23-05
Version 2.3
Static Techniques

Static Techniques
Static Testing – a Definition

Reviews and the Test Process
Review Process
Static Analysis
Summary
Static Techniques
Static Testing – a Definition
• Static testing techniques involve examination of the project’s

documentation, software and other information about the software
products without executing them
• Static Testing Includes both Reviews (e.g. of documentation) and
Static Analysis of code
• Reviews, Static Analysis and dynamic testing have the same
objective – identifying defects
• Static Testing and Dynamic Testing are complementary
• Each technique can find different types of defects effectively and
efficiently
dynamic testing: Testing that involves the execution of the

software of a component or system.
Topics
Why Review
When and What to review
What do Reviews find
Benefits of reviews
Why Review
• Find errors early in the development • Education and training of
lifecycle (more cost effective to fix) developers and other project staff
More than 60% of the errors in a • Determine the root causes of
component can be detected using defects and measures for
informal inspections (1) preventing recurrence
• Reduce the errors in delivered systems • Test and improve standards and
procedures
• Learning experience in standards and
techniques • To assess the early stages of
development for progress and
• Team building and motivation completeness
• Improve the specification, design and
code
1) Fagan, M. E., Advances in Software Inspections, IEEE Transactions on Software

Engineering, 1987
Why Review - The cost of errors
100
90
80
70
60
Relative 50
Cost
Multiples 40
30
20
10
0
Reqs Des Code Unit Accept Use
Why Review - Error/Fault Propagation
Requirement
Errors
Design
Errors
Coding
Errors
(Faults)
correct
correct
correct
Requirements Design Code
Why Review - Where errors are introduced
30%
25%
20%
15%
Errors
10%
5%
0%
Reqs Funct Logical Code Other
Des Des
Why Review
• “Quality control has as its mission the detection and

elimination of defects in the product. Reviews are the front
line in this mission ”
John W Horch 2003
When and What to Review?
• A review could be done entirely as a manual activity

• Or there is tool support available
• The main manual activity is to examine a work product and make
comments about it.
• Any software work product can be reviewed, including:
– requirements and design specifications
– source code listings
– test plans, test cases, test scripts
– user documentation
– application administration and support material
– Web pages
• A software work product can be reviewed one or more times and

using one or more review types
• Review everything as soon as possible
• Reviews start well before Dynamic Test execution
What do Reviews Find?
• In contrast to dynamic testing, reviews find defects rather than failures
• Typical defects that are easier to find in reviews than in dynamic testing
are:
– deviations from standards
– requirement defects
– design defects
– insufficient maintainability
– incorrect interface specifications.
Benefits
• Detect faults as they are introduced –i.e. early detection and correction
• Reduce the risk of error propagation
• Detect errors that Dynamic test execution unlikely to find, e.g.
requirement spec errors
• Shorten development timescales
• Reduce fault levels in delivered software
• Lower cost and shorten testing timeframes
• Lower cost over the life of the software
• Create development productivity improvements
• Reliably evaluates progress and capability (1)
• Educates and trains participants (1)
• Improve communication between project teams
(1) – The Complete Guide to Software Testing – Bill Hetzel
Review Process
Topics
Informal Reviews
Formal Review Types
Formal Review Process
Formal Review Roles and Responsibilities
Other Review Types
Key Success factors
Review Process
Informal Review
• No Formal Review Process employed
• “Desk checking”, looking for possible problems
• The author of material checking his or her own material, possibly
with one other peer (pair programming)
• Possibly a technical lead reviewing design and code
• Usually undocumented but useful, cheap and widely used
• This technique may be applied in low risk situations
• No metrics kept, review findings are not configured items
• Weaknesses – do not find as many faults as formal reviews
Review Process
Formal Review Types – Walkthroughs
• A walkthrough is a review of authored material led by

the author and attended by a group of the author's
peers (typically 2 to 6 peers). Primary purpose is
education
• The material is presented by the author to the peer
group, who focus on learning about the material,
improving it and recording defects
• Peer group should include development, operation
representatives, target audience, etc.
• Examples are Dry Runs or Scenario playing to validate
product
• Sessions can be formal or informal
• Review sessions often open-ended (not time-boxed)
• Pre-meeting preparation often involved
• Weaknesses – do not find as many defects as technical
reviews and inspections
Review Process
Formal Review Types – Technical Reviews
• May be performed as Peer Reviews without management participation
• Preferably lead by a trained moderator (not the author)
• Pre-Meeting preparation is required
• Primary purpose is to:
– Discuss
– make decisions
– evaluate alternatives
– find defects
– solve technical problems and check conformance to specifications and standards
• Degree of formality varies

• Reviewers bring a list of technical issues to the review
• Optional use of checklists and a review report
• During the meeting reviewers raise objections, ambiguities or inconsistencies
in design or technical aspects being discussed
• Problems are clarified and documented - solutions are sought after the review
has concluded
• Weaknesses – do not find as many faults as Inspections
Review Process
Formal Review Types – Inspections
• Formal, systematic reviews of material. Primary purpose is fault finding

and process improvement
• Led by an independent trained moderator (but not the author)
• Main purpose – find defects
• Attended by the author & the author's peers (usually 3 to 6) acting in
defined roles
• Pre meeting preparation essential
• Follow a strict format
– stated entry criteria and exit criteria
– seeking and recording defects
– use standardised rules, check-lists and techniques
– record metrics
• Formal follow-up process
• Optionally process improvement considerations part of review
• Weaknesses – expensive and time consuming but high defect yield!
Planning
Kick-off
Review Overview – optional
Preparation
Review Meeting
Rework
Follow-up
Repeat Review - optional
Planning
• Define Entry and Exit Criteria (for most

formal reviews)
• Ensure that the volume of material to be
reviewed is appropriate
• Identify roles, participants and establish a
time and place for the review
Kick Off
• Distribute the material to the participants

• Explain objectives, process and material to be reviewed
• Obtain copies of the relevant review and report
templates
• Create checklists of areas to cover and distribute
– checklists can make reviews more effective and efficient
– E.g. a checklist based on perspectives such as user,
maintainer, tester or operations
– or a checklist of typical requirements problems to focus on
• Make sure entry criteria has been/will be met
Review Overview (Optional)
• Required for new or difficult material

• Overviews:
– educate the participants
– allow participants to focus on technical content
– describe where the material fits in the system and in the development process
– focus on any complex functionality
– highlight any changes and explain the need for these changes
Preparation
• Each participant reviews the material to:

– learn about the material
– note suspected defects
– record questions
• In some circumstances, depending on the
expertise of the participants, the moderator may
ask certain participants to concentrate on
particular aspects of the material during
preparation
Review Meeting
• The material is read to the participants by the
reader
• Defects are raised by the participants and
recorded by the recorder
• Participants may make decisions about
categorising and even handling the defects –
though usually avoid ‘solutionising’
• Deliverables may include meeting minutes
• For Inspections - Pass or fail and repeat
review decisions are usually made by the
moderator
• The preparation time and the actual time may
be recorded
Rework
• The author must resolve all defects found during the review by reworking
the material as recommended by the review report
• Note, the cost of rework is NOT included in the cost of reviews
Follow-up
• Check the corrections to the material and account for all recorded defects
• If necessary, schedule a repeat review for the corrected material
• Inform management of the status of the corrected material
• Add the error data from the review to the project statistics database –
enables process improvement!
• Complete and sign the review report and forms (Inspections)
• Ensure exit criteria met
Repeat Review (optional)
• If the material has been passed as is or if the rework is minor, no further

reviews are required
• If a repeat review is required (e.g. if significant re-work was required) a
repeat review must be scheduled with the same participants to verify the
revised material
Review Process
Formal Review Roles and Responsibilities
Manager decides on the execution of reviews, allocates time in project

schedules and determines if the review objectives have been met
Moderator the person who leads, plans and runs the review
May mediate between the various points of view and is often the
person upon whom the success of the review rests
Author the writer or person with chief responsibility for the document(s)
to be reviewed
Reviewers Individuals with a specific technical or business background (also

called checkers or inspectors)
Identify and describe findings (e.g. defects)
Take part in any review meetings
Scribe/Recorder documents all the issues, problems and open points that were
identified during the meeting
Review Process
Other Review Types
• Some others not in the ISTQB Syllabus:
– Management reviews – reviews of plans, schedules, progress

– Audits – independent evaluation of conformance to standards, plans
and procedures
– Post Implementation – review of project approach (including test
approach)
Review Process
Key Success Factors
– Each review has a clear predefined objective
– The right people are involved
– Defects found are welcomed (Authors must leave their ego
at the door!)
– Defects are expressed objectively – no cornering the author
– make it a positive experience!
– Review techniques are applied suitable to the review
– Checklists used if appropriate – focus the review effort
– Roles pre-defined – avoid duplication and increase
effectiveness
– Training is given in review techniques- especially for
Inspections
– Management buy in – schedule in review activities and
effort
– There is an emphasis on learning and process improvement
Static Analysis
Topics
Definition
Description
Value
Types of Defects found
Use of Tools
Static Analysis
Definition
static analysis: Analysis of software artifacts, e.g.

requirements or code, carried out without execution of
these software artifacts.
Static Analysis
Description
• Objective - to find defects in software source code and software
models
• Static analysis is performed without actually executing the software

being examined by the tool whilst Dynamic testing does execute the
software code
• Static analysis can locate defects that are hard to find in testing.
• As with reviews, static analysis finds defects rather than failures
• Static analysis tools analyze program code (e.g. using control flow
graphing and data flow analysis techniques), as well as generated
output such as HTML and XML.
Static Analysis
Value
• Early detection of defects prior to test execution
• Early warning about suspicious aspects of the code or design
• Calculation of metrics such as a high complexity measure
• Identification of defects not easily found by dynamic testing
• Detecting dependencies and inconsistencies in software models, such
as links
• Improved maintainability of code and design
• Prevention of defects, if lessons are learned in development
• Cost effective – problems found earlier are cheaper to fix
• Static analysis is more effective and less expensive than dynamic
testing (1)
• Static analysis is demonstrated to find 45% of expected errors before
testing actually starts
Static Analysis
Value – Complexity Analysis
• Many complexity measures in common use, e.g.

– Cyclomatic Complexity – measures structural complexity
– Halstead Complexity – measures algorithmic complexity, measured by
counting operators and operands
– Henry and Kafura metrics – measures coupling between modules
(parameters, global variables, calls)
• Cyclomatic Complexity is the most widely used. It identifies how

difficult the code is to understand, test and maintain. It also identifies
the risk associated with testing a program
Static Analysis
Types of Defects found
• Unreachable code • Incorrect variable usage

• Undeclared variables • Syntax checking
• Parameter type mismatches • Violations of code standards
• Uncalled functions and procedures • Use of variables without first
• Possible array bound violations defining them
• Security Violations • variables that are declared but
never used
• Inconsistent interface between
modules and components • Use of variables after they have
been “killed”
Static Analysis
Use of Tools
• Static analysis tools are typically used by developers
• Used mainly before and during Component and Component Integration
testing
• The tools check against predefined rules or programming standards
• Also by designers during software modelling
• Static analysis tools may produce a large number of warning messages
• Hence the need to use the tools effectively (or can’t see the wood for
the trees!)
• Compilers may offer some support for static analysis, including the
calculation of metrics
Static Techniques
Summary
• We learned that Static Testing techniques involve examining
documentation and software products without executing them
• That Static Testing covers both Reviews and Static Analysis
• For Reviews we learned about:

– the different types of products that can be reviewed and that we review as
early as we can
– The Benefits of conducting Reviews
– And the typical defects that reviews uncover
• We learned about the Review process

– The key characteristics of the 4 Review types, Informal, Walkthrough,
Technical and Inspection
– The 6 main Stages that make up the Review process from Planning through
to Follow-up
– The Roles and Responsibilities involved in that process
– The Key Success Factors for Review – what makes them work
Static Techniques
Summary
• And we learned about Static Analysis
– Analysis of software artefacts (e.g. code/design) without executing them

– The Value of Static Analysis – what we gain by conducting it
– The types of defects found in static analysis
– And finally the use of tools such as a Compiler – who uses them and when
03-23-05
Version 2.3

Identifying Test Conditions and Designing Test Cases

Categories of Test Design Techniques
Specification based or Black Box Techniques
Structure based or White Box Techniques
Experience Based Techniques
Choosing Test Techniques
Summary
Identifying Test Conditions and Designing
Test Cases
• Developing test material can be split into two distinct stages:

– Defining “what” needs to be tested
– Defining “how” the system should be tested
• This process can vary from organisation to organisation, can be very formal or
very informal with little documentation
• The more formal, the more repeatable the tests, but it does depend on the
context of the testing being carried out
• The process of identifying test conditions and designing tests consists of the
following steps:
– Defining test conditions
– Specifying test cases
– Specifying test procedures
– Developing a Test execution schedule
Test Cases
Test Conditions
• Test Conditions are binary statements which determine a system’s fitness for purpose
• Defines what must be tested
• Sometimes referred to as a Test Item
• Grouped by Test Object or system function/process
• Test Requirement (Test basis ) documentation is analysed to determine the Test
Conditions
• Test Conditions are then cross referenced to one or more test cases for execution
• Not all Test Conditions are as important as others so each Test Condition is assigned a
risk
• Test Conditions should be linked back to their source documents from which they are
derived. This helps for two reasons:
– Impact Analysis
– Traceability
Test Cases
Test Cases
• A Test Case defines how the system should be tested
• They typically contain
– Input values
– Execution pre conditions
– Expected results (output, changes in state etc)
– Post conditions
– Cross referenced test conditions
• Remember expected results must be defined before execution
• There can be many Test Cases developed to test a single Test Condition
Expected Results
• “A necessary part of a test case is

a definition of the expected output
or result”
Myers - 1979
Test Cases
Test Procedures
• The Test Procedures Specification specifies the sequence of actions for a
test, i.e. one or more Test Cases
• It is also known as a Test Script
• The Test Script can be manual or automated
• Contents of a Test Procedure are:
– Test procedure specification identifier
– Purpose
– Special requirements
– Procedure steps
Test Cases
Test Execution Schedule
• The Test Procedure Specifications (i.e. Test Scripts) are subsequently
included in a Test Execution Schedule
• This schedule defines the order in which the test scripts are executed, when
they are to be carried out and by whom
• The Execution schedule will also need to take account of:
– Regression Tests
– Prioritisation
– And technical and logical dependencies
Test Conditions, Cases, Procedures and Schedule
What How How When
Test Test
Procedure Execution
Sourced Documentation
Specification Schedule
Manual Test
Test Script Test
Test Test Procedure
Cases
Condition Cases or Specifications
Automated
Priority Test Script
Categories of Test Design Techniques
Definition of Black Box Testing

What is black box testing?
Definition of White Box Testing
What is white box testing?
Black, White and Experience based (comparison)
Use of Techniques and tools
Black and White Box Testing
Definition of Black Box Testing
• Black-box testing: Testing, either functional or non-functional,

without reference to the internal structure of the component or
system.
What is Black Box Testing?
– testing without knowing the internal workings of the code

– WHAT a system does, rather than HOW it does it
– typically used at System Test phase, although can be useful throughout the
test lifecycle
– also known as specification based testing
– applies for Functional and Non-Functional testing
Input Output
If Output = Expected result then pass

Definition of White Box Testing
white-box testing: Testing based on an analysis of the

internal structure of the component or system.
What is White Box Testing?
– testing based upon the structure of the code

– typically undertaken at Component and Component Integration
Test phases by development teams
– also known as structural or glass box testing or structure based
testing
Input Output
Black, White and Experienced based
•Based on requirements
Black •From the requirements, tests are created
(Specification •Specification Models can be used for systematic test case design
Based) Techniques
•Equivalence Partitioning
•Boundary Value Analysis
•Decision Tables
•State Transition Testing
•Use Case Testing
•Based on code and the design of the system

White •The tests provide the ability to derive the extent of coverage of the whole
(Structure application
Based)
Techniques
•Statement coverage
•Branch Coverage
•Decision Coverage
•Based on the knowledge of the tester

Experience
•Using past experienced use & intuition to “guess” where errors may occur
(Black box)
Techniques
•Error Guessing
•Exploratory Testing
Use of Techniques and Tools
– techniques provide a systematic method of approaching

software testing
– enable tests to be repeatable
– provide a measure of software coverage
– due to the scale and complexity of systems, tools are often
used to assist with testing especially white box testing
Specification based or Black Box Techniques
Equivalence Partitioning
Boundary Value Analysis
Decision Table Testing
State Transition Testing
Use Case Testing
Other black box test techniques
Black Box Test Techniques
• aim is to treat groups of inputs as equivalent and to select one

representative input to test them all
• Best shown in the following example….
– If we wanted to test the following IF statement:

– ‘IF VALUE is between 1 and 100 (inclusive) (e.g. VALUE >=1 and
VALUE <= 100) THEN …..’
– We could put a range of numbers as shown in the next slide through
test cases
IF Value >= 1 AND Value <= 100 THEN ….
0 101
37 65 99
1 100
-1 19 53
48 87 1000
OUT OF OUT OF
RANGE IN RANGE RANGE
• the numbers fall into a partition where each would have the same, or
equivalent, result i.e. an Equivalence Partition (EP) or Equivalence Class
• EP says that by testing just one value we have tested the partition
(typically a mid-point value is used). It assumes that:
1) if one value finds a bug, the others probably will too

2) if one doesn't find a bug, the others probably won't either
• in EP we must identify Valid Equivalence partitions and Invalid
Equivalence partitions where applicable (typically in range tests)
• the Valid partition is bounded by the values 1 and 100
• plus there are 2 Invalid partitions
IF Value >= 1 AND Value <= 100 THEN ….
0 101
37 65 99
1 100
-1 19 53
48 87 1000
‘VALID’ PARTITION
‘INVALID’ ‘INVALID’
PARTITION
EDS PARTITION
ISTQB Testing Foundation Course Version 2.3 Slide 207 • EDS Internal
• Time would be wasted by specifying test cases that covered a range of
values within each of the three partitions, unless the code was designed in
an unusual way
• There are more effective techniques that can be used to find bugs in such
circumstances (such as code inspection)
• EP can help reduce the number of tests from a list of all possible inputs to a
minimum set that would still test each partition
• If the tester chooses the right partitions, the testing will be accurate and
efficient
• If the tester mistakenly thinks of two partitions as equivalent and they are
not, a test situation will be missed
• Or on the other hand, if the tester thinks two objects are different and they
are not, the tests will be redundant
• EP can be used for all Levels of Testing
• EP is used to achieve good input and output coverage, knowing exhaustive

testing is often impossible
• It can be applied to human input, input via interfaces to a system, or

interface parameters in integration testing

• Boundary Value Analysis (BVA) uses the same analysis of partitions as EP
and is usually used in conjunction with EP in test case design
• As with EP, it can be used for all Test levels
• BVA operates on the basis that experience shows us that errors are most
likely to exist at the boundaries between partitions and in doing so
incorporates a degree of negative testing into the test design
• BVA Test cases are designed to exercise the software on and at either side
of boundary values

For our previous example of ‘Value >= 1and Value <= 100’
Value = 1 Value = 100

(valid) (valid)
Value = 0 Value = 2 Value = 99 Value = 101
(invalid) (valid) (valid) (invalid)
• find the boundary and then test one value above and below it
• ALWAYS results in two test cases per boundary for valid inputs and
three tests cases per boundary for all inputs
• inputs should be in the smallest significant values for the boundary
(e.g. Boundary of ‘a > 10.0’ should result in test values of 10.0,
10.1 & 10.2)
• only applicable for numeric (and date) fields
Decision Table Testing
• Table based technique where

– Inputs to the system are recorded
– Outputs to the system are defined
• Inputs are usually defined in terms of actions which are Boolean (true or
false)
• Outputs are recorded against each unique combination of inputs
• Using the Decision Table the relationships between the inputs and the
possible outputs are mapped together
• As with State Transition testing, an excellent tool to capture certain types of
system requirements and to document internal system design. As such can
be used for a number of test levels
• Especially useful for complex business rules
Decision Table Structure
Inputs / Actions Test 1 Test 2 Test 3

Input 1 T T F
Input 2 T T F
Input 3 T DON’T CARE F
Input 4 T F T
Response 1 Y Y N
Output /
Response Response 2 Y N Y
Response 3 N Y N
Each column of the table corresponds to a business rule that defines a unique
combination of conditions that result in the execution of the actions associated
with that rule
The strength of Decision Table testing is that it creates combinations of conditions

that might not otherwise have been exercised during testing
Decision Table Example
Test 1 Test 2 Test 3
> 55 yrs old F T T
Smoker F T F
Exercises 3 times a week + T F T
History of Heart Attacks F T F
Insure Y N Y
Offer 10% Discount N N Y
Offer 30% Discount Y N N
What will be the out come of the following Scenarios?

Joe is a 22 year old non smoker who goes to the gym 4 times / week and has
no history of heart attacks in his family
Kevin is 62 year old non smoker who swims twice a week and plays tennis. He
has no history of heart attacks in his family
State Transition Testing

• State Transition Testing uses the following terms:
– state diagram: A diagram that depicts the states that a component or system can
assume, and shows the events or circumstances that cause and/or result from a
change from one state to another. [IEEE 610]
– state table: A grid showing the resulting transitions for each state combined with
each possible event, showing both valid and invalid transitions.
– state transition: A transition between two states of a component or system.
– state transition testing: A black box test design technique in which test cases are
designed to execute valid and invalid state transitions. Also known as N-switch
testing.
• An excellent tool to capture certain types of system requirements and to
document internal system design. As such can be used for a number of test
levels
• Often used in testing:
– Screen dialogues
– Web site transitions
State Transition Diagram
State A Starting State
Transition Between
Event/Action etc start and end states
Event from outside

the system
State B Action triggered by
Event
End State
State Transition Example Simplified Car Gears
Change Down/
Move Back
Neutral Reverse
Change Up/
Change Up/ Change Down/ Accelerate
Accelerate Decelerate
1st Gear
Change Up/ Change Down/
2nd Gear
Change Up/ Change Down/
3rd Gear
State Transition - Switch Coverage
• Switch Coverage is a method of determining the number tests based on the

number of “hops” between transitions. Some times known as Chow
0-Switch Coverage (1 hop) 1-Switch Coverage (2 hops)

RN RN1
NR RNR
N1 NRN
1N N12
12 N1N
21 1NR
23
32 Etc.
These hops can be used to determine the VALID tests
State Transition - State Table
• While Switch testing helps determine the valid tests, we also need to look
for the invalid tests.
Change Up Change Down
R Acc / Neutral Null

N Acc/1st Gear Dec / Reverse
1st Acc/ 2nd Gear Dec / Neutral
2nd Acc / 3rd gear Dec / 1st Gear
3rd Null Dec / 2nd Gear
Invalid tests are those identified by a null output in this case
• Changing down from reverse

• Changing up from 3rd
State Transition – Another example
for a Theatre Show reservation
Request Choose Reserve
Show Show Show
Options
Show
Show Options
Show selected Reservation
provided
Made
Pay for
Change Show
Mind/
Cancel
Return to
reservation
Options
Show
Reservation
Paid For
Cancelled Issue
Cancel reservation/
Reservation Issue Refund Ticket
Ticket
Cancel reservation
(return ticket)/Issue Received
Refund
State Transition – Another type of State Table
Current Event/ Event/ Next Event/ Next Event/ Next Event/ Next Event/ Next
State Next State State State State State State
A B C D E F
SS 1
1 2
2 1 3
3 4
4 ES
ES
Use Case Testing
• Use cases are a method of describing requirements

• Structured approach to requirements design
• Usually has one main flow (though may also have alternate flows)
• Each use case is a process flow or scenario
• Particularly useful in determining tests for (Functional) Systems Test and
for UAT
• Also good at detecting defects in the integration of interfaces
• The main parts of a Use Case are:

– Actors users of the system
– Pre conditions What are the starting requirements for the use case
– Post conditions The state the system will end up in once completed
Other Black Box Test Techniques
• Syntax testing
— test cases are prepared to exercise the rule governing the format of
data in a system (e.g. a Zip or Postal Code, a telephone number)
• Random testing
— test cases are selected, possibly using a pseudo-random generation

algorithm, to match an operational profile
Structure Based or White Box Test Techniques
Statement Testing
Decision Testing
Assessing Completeness (Coverage)
Other White Box test techniques
White Box Test Techniques
Statement Testing
– aim is to display that all executable statements have been run at

least once
– coverage measurement is:
number of statements executed

total number of statements
Statement Testing
1 Control
Example 1 Flow Graph
2
1. Read vehicle
3
2. Read colour
4
3. If vehicle = ‘Car’ Then
4. If colour = ‘Red’ Then 5
5. Print “Fast”
6
6. End If
7. End If 7
Statement Testing 1
Example 2 2
1. Read A 3
2. If A > 40 Then
4
3. A=A*2
5
4. End If
5. If A > 100 Then 6
6. A = A – 10
7
7. End If
Statement Testing 1
Example 3
1. Read bread 2
2. Read filling
3. If bread = ‘Roll’ Then 3/9
4. If filling = ‘Tuna’ Then
4/6
5. Price = 1.50
6. Else 7 5
7. Price = 1.00 10
8. End If 8
9. Else
10. Price = 0.75 11
11. End If
Decision Testing
– Aim is to demonstrate that all Decisions have been run at least once
– Coverage measurement is:
the number of Decisions executed

the total number of Decisions
– Note that Decision testing is related to Branch testing

– Decisions are lines of code where there are two or more alternative routes based on a
decision that has to be made, i.e. IF/THEN/ELSE
– Branches are where there are one of two or more alternative routes from a line of code but
also include JUMPs and GO TOs
– Decision and Branch Test coverage for a piece of code is often the same, but not always
– We are only interested in Decision coverage though.
– Note BT is always >=ST
Decision Testing 1
Example 1 2
1. Read vehicle 3
2. Read colour 4
3. If vehicle = ‘Car’ Then
5
4. If colour = ‘Red’ Then
5. Print “Fast” 6
6. End If
7
7. End If
Decision Testing 1
Example 2 2
1. Read A
3
2. If A > 40 Then
4
3. A=A*2
4. End If 5
5. If A > 100 Then
6
6. A = A – 10
7. End IfC 7
Decision Testing 1
Example 3
1. Read bread
2
2. Read filling
3. If bread = ‘Roll’ Then 3/9
4/6
5. Price = 1.50
6. Else
7 5
7. Price = 1.00
10
8. End If 8
9. Else
10. Price = 0.75 11
11. End If
Statement and Decision Testing
The pseudo-code can be expressed in Flow-chart format
1. Read A
2. If A > 40 Then
3. A=A*2
4. End If
5. If A > 100 Then
6. A = A – 10
7. End If
1. Read bread
2. Read filling We already know:
Statement Coverage = 3
3. If bread = ‘Roll’ Then
Decision Coverage = 3
Number of executable Statements = 7
5. Price = 1.50
Number of Branches = 4 (T and F of
6. Else each IF)
7. Price = 1.00
8. End If
9. Else
10. Price = 0.75
11. End If
1. Read bread
2. Read filling • Based on the following test set:
3. If bread = ‘Roll’ Then Bread Filling
Roll Tuna
Sandwich Ham
5. Price = 1.50
6. Else • What is the test Statement
Coverage and Decision Coverage?
7. Price = 1.00
8. End If
9. Else
10. Price = 0.75
11. End If
Other White Box Test Techniques
• Branch Condition testing

— Branch Condition Testing requires that the True and False of each Boolean operand
is tested (Boolean Operands in this example: If A > 30 and B >= 5)
• Branch Condition Combination testing
— Branch Condition Combination Coverage would require all combinations of Boolean
operands to be evaluated
• Modified Condition Decision testing
— Modified Condition Decision Coverage requires test cases to show that each
Boolean operand can independently affect the outcome of the decision
• Dataflow testing
— Data flow testing aims to execute sub-paths from points where each variable in a
component is defined to points where it is referenced.
• Linear Code Sequence And Jump (LCSAJ) testing
— LCSAJ testing requires a model of the source code which identifies control flow
jumps (where control flow does not pass to a sequential statement).
Definition
Error Guessing
Exploratory Testing
Experienced Based Techniques
Definition
• Uses the knowledge of people and the experience of past projects to

determine likely errors based on the knowledge
– Testers
– Users
– Stakeholders
• Experience based techniques are non structured and do not rely on
specification documents. This makes them unable to be measured in terms
of coverage
Definition
• Tests are derived from skill and intuition

• Used to AUGMENT more systematic methods
• Good to identify tests which may not be explicitly described in the
specifications
• Most beneficial when applied after more formal measures
• The success of these techniques is directly related to the knowledge and
skill of the testers
• Does not provide any form of measurement or coverage
Error Guessing
• Using experience to postulate errors
• Use Error Guessing to complement test design techniques
• Use as a “mopping up” approach to supplement systematic

techniques
• Can be useful to identify special tests not easily captured by
formal techniques, especially when applied after more formal
approaches
• So don’t use as a first choice technique!
• Structured approach to error guessing
• Create a list of all possible errors
• Then create tests to attack these errors
• Remember these defect attack lists are built on experience, previous

defects and from common knowledge as to why systems fail
Error Guessing
• Error Guessing tests may include
• ‘Enter 00000 or 99999 in to a field’
• Creating surnames with quotes in, such as O’Donnell
• Nulls in mandatory fields
• Reserved characters ($%& for web systems)
Exploratory testing
• Exploratory testing is a concurrent process where
– Test design, execution and logging happen simultaneously
– Testing is often not recorded
– Makes use of experience, heuristics and test patterns
– Testing is based on a test charter that may include
• Scope of the testing (in and out)
• Why? - what is the focus of your testing
• A brief description of how tests will be performed
• Expected problems
– Is carried out in time boxed intervals
• More structured than Error guessing
Exploratory
Risk Analysis Charter Debriefing
Sessions
Notes
Choosing test Techniques
• How do you chose the right technique?

– Type of system
– Standards
– Customer or contractual requirements
– Level of risk
– Type of risk
– Testing objectives
– Documentation available
– Knowledge / skills of the testers
– Time and budget
– Development processes
• Pick the right techniques for the right
situation
Test Design Techniques - Summary
• We learned about
– Identifying Test Conditions
– Designing Test Cases from the Test Conditions
– Creating Test Procedure Specifications to sequence our Test Cases
– Creating Test Execution schedules to define the order in which the test scripts
are executed, when they are to be carried out and by whom
– The importance of traceability to requirements and specification of expected
results
• We learned about the difference between Black and White Box testing
– White Box (Structure-based) Testing is based upon the structure of the program
code
– Black Box (Specification Based) Testing is without reference the internal working
of the program code
– The reasons why both are useful
Test Design Techniques - Summary
• We learned about the different Black box techniques, mainly:

– Equivalence Partitioning
– Boundary Value Analysis
– State Transition Testing
– Decision Tables
– Use Case Testing
– Experience based Testing
• We learned about the different White box techniques, mainly:
– Statement Testing
– Decision Testing
• For all Black and White box techniques we learned why they are of use
and for which test levels they are typically applied
03-23-05
Version 2.3
Test Management

Test Management
Test Organisation
Test Planning and Estimation
Test Progress Monitoring and Control
Configuration Management
Risk and Testing
Incident Management
Summary
Test Organisation
Test Independence
Testing Roles Within the Team
The Test Leader
The Tester
Test Independence
Levels of Independence
• More effective for someone other than the developer to test the system
– More impartial
– No preconceived ideas about what requires testing
– No bias or emotional attachment
Levels of Test Independence:

1. Independent testers within the development teams.
2. Independent test team or group within the organization, reporting
to project management or executive management.
3. Independent testers from the business organization, user
community and IT.
4. Independent test specialists for specific test targets such as
usability testers, security testers or certification testers
5. Independent testers outsourced or external to the organization.
Test Independence
• Most test projects require multiple levels of testing where some of the
testing is carried out by independent testers.
Acceptance
Testing
System Independent
Integration
Testing Testers
System Testing
Developers
Component Testing
Test Independence
Independent testing benefits & drawbacks
Benefits
• Independent testers see other and different defects, and are unbiased
• An independent tester can verify assumptions people made during
specification and implementation of the system
• Usually a Cost saving
– Better skills = more effective testing and fewer defects getting into production
– For Third Party test outsourcing, ‘better to rent than to own’
Drawbacks
• Isolation from the development team (if treated as totally independent).
• Independent testers may be the bottleneck as the last checkpoint
• Developers lose a sense of responsibility for quality
• Can be a greater cost – need to consider viability
• For Third Party test outsourcing, the project carries the risk
Testing Roles within the Team
Independent testing – who does what
• Where possible some or all of the testing should be carried out by

independent testers
• Developers may contribute at the lower levels (typically Component Testing
and maybe Component integration)
• Specialist testers mainly play a role at Functional, Non Functional and
System integration test levels
• The Business and System Administrators provide testing for Acceptance
testing (e.g. UAT, OAT)
• The Test processes and rules are often defined by the Independent testers
but must be agreed by management
The Test Leader
Role of the Test Leader
• Also known as Test Manager or Test Coordinator

• This role may also be done by
– Project manager
– QA manager
– Development manager
• In large projects this role may be split into a
– Test Manager
– Test (team) Leader
• Key tasks of a leader
– Test Planning
– Test Monitoring
– Test Control
The Test Leader
The Test Leader’s tasks

• Strategy & Management
– Write and review the test strategy
– Coordinate the test strategy
– Plan testing effort – context, risks & approach
– Proactive representation in other project activities – ensure testing has correct focus
– Ensure proper configuration management of Testware exists
– Determine what should be automated
– Select and implement the most appropriate tools for testing including necessary training
– Management and definition of the test environmental requirements
– Define the test schedule based on the delivery of code in to test
• Monitor
– Define, record and continually review the testing project metrics
– Monitor test progress against the test schedule
– Write the test summary reports
• Control
– Adapt testing effort based results and progress
The Tester
The role of the tester
• Make up the majority of the resources in the testing group.

• May be specialists in a particular area
– Automation
– Performance
– Usability
– Security
• Or alternatively may work more generally doing:
– Test Analysis
– Test Design
– Test Execution
– Test Environment management
– Test Data management
• Typically testers at the component level are developers
• At the Acceptance level testers are typically business experts or users or operators
(for operational acceptance testing)
The Tester
The tester’s tasks
• Review and contribute to test plans.

• Analyse, review and assess user requirements, specifications and models for testability.
• Create test specifications.
• Set up the test environment (often coordinating with system administration and network
management).
• Prepare and acquire test data.
• Implement tests on all test levels, execute and log the tests, evaluate the results and
document the deviations from expected results.
• Use test administration or management tools and test monitoring tools as required.
• Automate tests (may be supported by a developer or a test automation expert).
• Measure performance of components and systems (if applicable).
• Review tests developed by others.
Test Planning and Estimation
Test Planning
Test Planning Activities
Exit Criteria
Test Estimation
Test Approaches
Test Planning
Test strategies and test plans
• All projects require a set of plans and strategies which define how the
testing will be conducted.
• There are number of levels at which these are defined:
Test Policy Defines how the organisation

will conduct testing
Master Test Defines how the project will

Plan/Test Strategy conduct testing
Defines how each level of

Functional System Integ UAT
Test Plan Test Plan Test Plan testing will be conducted
Test Planning
Test planning factors
• Factors which affect test planning

– The organisation’s test policy
– Scope of the testing being performed
– Testing objectives
– Project Risks – e.g. business, technical, people
– Constraints – e.g. business imposed, financial, contractual etc
– Criticality (e.g. system/component level)
– Testability
– Availability of resources
• Test plans are continuously refined
– As more information becomes available
– As new risks arise or others are mitigated
– Not set in concrete, but changes must be carefully managed
Test Planning
Test Plan Contents (IEEE 829)
1. Test Plan Identifier
2. Introduction
3. Test Items
4. Features to be Tested
5. Features not to be Tested
6. Approach
7. Item Pass/Fail Criteria
8. Suspension Criteria and Resumption Requirements
9. Test Deliverables
10. Testing Tasks
11. Environmental Needs
12. Staffing and Training Needs
13. Responsibilities
14. Schedule
15. Planning Risks and Contingencies
16. Approvals
Test Planning Activities
Defining a test plan
• Approach - Defining the overall approach of testing (the test strategy), including the
definition of the test levels and entry and exit criteria.
• Integrating and coordinating the testing activities into the software life cycle
activities: acquisition, supply, development, operation and maintenance.
• Making decisions about:
– what to test
– who – i.e. what roles will perform the test activities
– when and how the test activities should be done and when they should be stopped (exit
criteria – see next slides)
– how the test results will be evaluated
• Assigning resources for the different tasks defined.
• Testware definition- Defining the amount, level of detail, structure and templates
for the test documentation.
• Selecting metrics for monitoring and controlling test preparation and execution,
defect resolution and risk issues.
• Process - Setting the level of detail for test procedures in order to provide enough
information to support reproducible test preparation and execution.
Exit Criteria
How do we know when to stop testing?
The business
Run out of tells you it went
time? live last night!
Boss says
Run out of stop?
budget?
All defects have

been fixed?
When our exit criteria
have been met?
Exit Criteria
What are exit criteria?
• Purpose of exit criteria is to define when we STOP testing either at the:

– End of all testing – i.e. product Go Live
– End of phase of testing (e.g. hand over from System Test to UAT)
• Exit Criteria typically measures:
– Thoroughness measures, such as coverage of requirements or of code or risk
coverage
– Estimates of defect density or reliability measures. (e.g. how many defects
open by category)
– Cost.
– Residual Risks, such as defects not fixed or lack of test coverage in certain
areas.
– Schedules - such as those based on time to market.
Test Estimation
Test Estimation - Why
• estimates provide predictions of effort and cost for conduct of testing

activities that support :
– establishment of budgets
– procurement of resources
– production of a schedule for use in tracking and control
• by definition estimates are not exact and good estimates are ones that
provide predictions within realistic tolerance limits
Test Estimation
Test Estimation - How
• Estimation based on metrics from similar or previous projects or typical

values
• Use metrics such as
– Length of time in preparation
– Length of time in execution
– Defect turn around time
– Down time and delays
• However it is important that you compare metrics from similar sized projects.
• Size of project is also required
– Function point analysis
– LoC (lines of code)
Test Estimation
Test Estimation - How
• Estimating tasks by the owner of these tasks or by experts

• Experts may use formal metrics
• Or may be more gut feel and experience based
Test Estimation
Estimation - Factors
• The testing effort can depend on:
– Characteristics of the product:
• the quality of the specification
• the size of the product
• the complexity of the problem domain
– Characteristics of the development process:
• the stability of the organization
• tools used
• test process
• skills of the people involved
• time pressure
– The outcome of testing
• the number of defects
• the amount of rework required.
• Once estimation is complete the resource requirements and projects
schedules can be finalised
Test Estimation
• “Test estimation is hard to do perfectly, but not terribly hard

to do well ”
Black - 2002
Test Approaches
• One method of classifying the way testing is done is by looking at when the
bulk of testing is carried out
Preventative Reactive
Test Design Test Design

done here done here
Requirements Delivered Code Delivered Go Live
Test Approaches
More Approaches
• Analytical approaches, such as risk-based testing where testing is directed to
areas of greatest risk.
• Model-based approaches, such as stochastic testing using statistical information
about failure rates (such as reliability growth models) or usage (such as operational
profiles).
• Methodical approaches, such as failure based (including error guessing and fault-
attacks), check-list based, and quality characteristic based.
• Process- or standard-compliant approaches, such as those specified by industry-
specific standards or the various agile methodologies.
• Dynamic and heuristic approaches, such as exploratory testing where testing is
more reactive to events than pre-planned, and where execution and evaluation are
concurrent tasks.
• Consultative approaches, such as those where test coverage is driven primarily by
the advice and guidance of technology and/or business domain experts outside the
test team.
• Regression-averse approaches, such as those that include reuse of existing test
material, extensive automation of functional regression tests and standard test suites.
Test Approaches
Approaches Combined
• A preventative, Risk & Process based approach – would start

testing early and use a standard industry approach such as the V
model using the defined risks as a basis for testing
or
• A reactive, heuristic approach - would start after the code has
been delivered and use unplanned testing techniques such as
exploratory testing
Test Approaches
Selection of Approaches - Choosing the right approach
• The selection of a test approach should consider the context, including:

– Risk of failure of the project, hazards to the product and risks of product failure
to humans, the environment and the company
– Skills and experience of the people in the proposed techniques, tools and
methods
– The objective of the testing endeavour and the mission of the testing team
– Regulatory aspects, such as external and internal regulations for the
development process
– The nature of the product and the business
Test Progress Monitoring and Control
Test Progress Monitoring

Test Reporting
Reporting and Interpreting Test metrics
Test Control
Test Monitoring - Why
• Need to know the status of the testing project at any given point in time
• Need to provide visibility on the status of testing to other stake holders
• Need to be able to measure your testing against your defined exit criteria
• Need to be able to assess progress against
– Planned schedule
– Measure how you are tracking against your defined budget
Test Monitoring - How
• Typical Metrics include

– Test preparation - Percentage of work done in test case preparation (or percentage of planned
test cases prepared).
– Test environment preparation - Percentage of work done in test environment preparation.
– Test case execution (e.g. number of test cases run/not run, and test cases passed/failed).
– Defect statistics (e.g. defect density, defects found and fixed, failure rate, and retest results).
– Test coverage -e.g. % of requirements tested, risks or code coverage.
– Dates of test milestones – monitoring the test schedule
– Testing costs, including the cost compared to the benefit of finding the next defect or to run
the next test.
– Subjective confidence of testers in the product.
Test Reporting
Test Reporting - What
• Summarizes the testing carried out

– What happened during testing, for example:
• Were the dates met?
• Was the Test Plan adhered to and if not, what
changes were made and why
• What problems were encountered and how were
these addressed
– Analysed metrics from testing, for example:

• Defects remaining unresolved along with
associated risks and resolution plans
• Number of tests run, not run, passed, failed etc
Test Reporting
Test Reporting - What
• Typical Contents are (IEEE 829)

– Test summary report identifier;
– Summary
– Variances;
– Comprehensive assessment;
– Summary of results;
– Evaluation;
– Summary of activities;
– Approvals.
Reporting and Interpreting Metrics
Test Reporting - How
• Metrics should be collected during and at the end of a test level in order to
assess:
– The adequacy of the test objectives for that test level.
– The adequacy of the test approaches taken.
– The effectiveness of the testing with respect to its objectives.
– Metrics are a valuable input into process improvement
Interpreting Metrics – testing progress
Not Tested • On the face of it things look good
• Can we go live?
Failed • What else would we need to know?
• However…
• What are the risks of the parts that
have failed
• Does this chart account for all the
testing scheduled – is there more to
come?
Pass
Interpreting Metrics – Defect reporting
160
140
Gap between = number of
120 Incidents still outstanding
100
No. Defects
found
80
fixed
60
40
20
0
d1 d2 d3 d4 d5 d6
Time
“…Testing is a measurement of software quality.”
Bill Hetzel 1988
Test Control
Test Control - Why

• Test control is the response to Test Monitoring and Test Reporting that
allows us to be IN CONTROL of the project
• Projects do go awry and issues occur. We need Monitoring and Reporting to
know whether they are likely to go, or have already gone, awry.
• The process of Control is the corrective actions required to put a testing
effort (project) back on track.
• For Example;
– Re-prioritize tests when an identified risk occurs (e.g. software delivered late).
– Change the test schedule due to availability of a test environment.
– Set an entry criterion requiring fixes to have been retested by a developer
before accepting them into a build.
Configuration Management
Symptoms of Poor CM
Configuration Items and their control
Symptoms of poor CM
• Can’t match source and object code

• Can’t identify source code changes made for a particular version of software
• Simultaneous changes to source code by more than one programmer
(changes lost)
• Old bugs previously fixed reappear
• Incorrect (e.g. out of date) versions of documents or other objects are
used
• You have no control over the composition of the test environment
(hardware and software items)
• The purpose of Configuration Management is to establish and maintain the

integrity of the products (components, data and documentation) of the
software or system through the project and product life cycle.
• Items such as:

– plans, specifications (requirements, design)
– data dictionaries and cross-references
– software modules (purchased or developed)
– user and maintenance documentation
– support software (libraries, databases, compilers, debuggers)
– test Tools (software, discs, manuals etc)
• And just as importantly all items of Testware, such as:

– Test Plans, Test Cases, Test Data, Conditions, Procedures, test environment
infrastructure components (hardware and operating software), test tools
(purchased and bespoke) etc
• These must be:
– Uniquely Identified
– Version controlled
– Tracked for changes
– Related to each other
– Related to other development items (functional specifications etc)
• Referencing to other documents allows for traceability
• The configuration management procedures and infrastructure (tools) should
be chosen, documented and implemented during the Test Planning stage.
Risk and Testing
Testing and Risk

Measuring Risk
Project Risks
Product Risks
Risk Based Testing in practice
Testing and Risk
Risk based testing
• Testing is an exercise in risk reduction

• Exhaustive testing is impossible
• As testers we are under constant time pressure
• Cannot reduce the risk of software failing to 0!
• Risk can be classified as two distinct types
– Project
– Product
Project Risk
What is Project risk?
• Project Risk: A risk related to management and control of the (test) project.
• Supplier Issues
– Contractual Issues
– Third party goes in liquidation or fails to deliver
• Organisational Issues
– Skills and Staff shortages
– Training and support issues
– Communication/Political Issues, e.g. between testers and other project teams
• Technical
– No or poor requirements
– Quality of the design or code
– Architectural solution under question
Product Risk
What is Product risk?
• Product risk: A risk directly related to the test object.

• “Buggy” software delivered
• Software does not meet its requirements
• Software delivered may harm the organisation, data or individual
• Poor software characteristics
– Functionality
– Security
– Reliability
– Usability
– Performance
• Product risks are a risk to the success of the project
Risk Based Testing in Practice
Issue 1- We only have time to do half the testing we planned

If we have defined the risks of test items which we wish to test then we can focus on
highest risk items first and deliver an application which fit for purpose
Issue 2 – we have 500 passes and 3 failures. Can we go live?

Again if we know the risk of the test items which make up 500 passes and 3 failures we
can then make a decision based on the potential risk of failure of the items left out
standing or which have failed
• Risks can help decide where we should start testing or where we may need to more
testing
• Risks also help us analyse our current state and through test monitoring we can
determine if a system is ready to be implemented
• Risks can also drive the number of test levels and determine the techniques to use
“Prioritise so that when you stop testing you

have made the best of the time you have
available.”
Weymouth
2006
• Testing can support the identification of new risks – identified during test
planning
• Testing is used to reduce the risk of an adverse effect occurring, or to
reduce its impact
• Testing provides feedback about the residual risk- through measuring the
effectiveness of critical defect removal and contingency plans
• In analysing, recording and managing the Product and Project risks the Test
Manager is following well defined project management principles
• In a risk-based approach we can not only determine our test prioritisation
but also the test techniques to use
Measuring Risk
RISK = Likelihood * Impact

High
High Impact and Likelihood
– Needs to be addressed ASAP
Likelihood
Low Impact and Likelihood

– Needs no action
Low
Low Impact High
Functional Risk Matrix - MoSCoW

Likelihood Likelihood
Could Test Must Test

EP/BVA
Statement Coverage 70% Decision Tables
Branch coverage
Pair inspection
C A
Won’t Test Should Test
Informal Test specification Formal Test Specification
Error Guessing Statement Coverage
100%
D B
Impact Impact
Incident Management
Definition
Basic principles
Benefits
Attributes of an Incident
Tracking and Analysis
Writing Good Incident Reports
Definition
Incident: Any event occurring that requires investigation
Or more simply put…
Any discrepancy between actual and expected results
=
expected actual!
Basic Principals
• All incidents should be raised

• Incident should be tracked from discovery through classification and
correction to confirmation
• A formal incident management process and workflow is required to
effectively manage incidents
• Incidents may be raised at ANY point in the lifecycle
– Requirements review
– Coding
– Testing
• Incidents may be raised AGAINST any product in the lifecycle
– Source Requirements
– System Designs
– Source Code
– Test Material
– Help or user guides
Benefits
• Incidents:
– Provide developers and other parties with feedback about the problem to enable
identification, isolation and correction as necessary
– Provide test leaders a means of tracking the quality of the system under test
and the progress of the testing
– Provide ideas for test process improvement
Incident logging – Key attributes
• Date of issue, issuing organization, author, approvals and status

• Scope, Severity* and Priority* of the incident
• References, including the identity of the test case specification that
revealed the problem
*severity: The degree of impact that a defect has on the development or

operation of a component or system.
*priority: The level of (business) importance assigned to an item, e.g. defect.
Incident logging – Other attributes

• Expected and actual results
• Date the incident was discovered
• Identification or configuration item of the software or system
• Software or system life cycle process in which the incident was observed
• Description of the anomaly to enable resolution
• Degree of impact on stakeholder(s) interests
• Status of the incident (e.g. open, deferred, duplicate, waiting to be fixed,
fixed awaiting confirmation test or closed)
• Conclusions and recommendations
• Global issues, such as other areas that may be affected by a change
resulting from the incident
• Change history, e.g. of actions taken by project team members
Tracking and analysis – Incident Status examples
• raised and logged • resolved (closed)

• prioritised and assigned • invalid incident report (closed)
• under investigation • irreproducible problem (closed)
• fixed by developers (or technical • enhancement (to be actioned next
writers)
upgrade)
• tested
• on hold (deferred pending some other
• released to live environment action)
Tracking and analysis - steps
• categorise incidents by type

• assign an appropriate priority according to incident severity
• report incidents to the appropriate management level
• determine incident causes and propose corrective measures
• ensure timely corrective action by reviewing incidents and tracking their clearance
• re- testing and reviews after changes have been made to the software (refer
regression testing)
• review corrective measures to determine their effectiveness
• analyse deficiency trends to prevent non-conforming software
• N.B. Incident Review Boards often used to to oversee the reporting, analysis and
timely resolution of incidents during the course of a project. Members from all
interested parties, Project Manager, Test Manager, Lead developer, Customer, CM
Manager etc
What makes a good incident report
• The most important part of defect report is the failure description

• It will provide the developer with all the information they need to repeat the
incident and then fix it
• Badly written reports
– Slow the project
– Provide bad metrics
– Delay fixes
• Write in short sentences which are to the point – accurate, complete and
concise
• Contents of a failure description
– Summary
– Steps to reproduce
– Isolation
Contents of a failure description
• Summary
– one or two lines which provide an overview of the incident and its severity and
impact. Be sharp and to the point.
• Steps to reproduce
– provide the exact steps that were undertaken to create the incident. Be as
concise as possible, but make sure you add EVERY step
– get these steps right makes it easier for the developers to reproduce. – it
reduces the “it works on my machine” phenomenon
• Isolation
– is the incident repeatable, what particular factors effect the ability to reproduce
the incident, For example – “I observed the error on the following platforms IE5,
NS7”
Bad Example
Summary
There were a number or errors on the add customer screen
Steps to reproduce
1. Opened the add customer screen
2. Entered a new customer
3. Pressed add
4. Got error message
Isolation
I tried on a few different branches and it worked on most of them
Good Example
Summary
Error “cannot find object” (see attached screen shot) message was displayed when
trying to add a new customer to the system using screen ADD_CUST.
Steps to reproduce
1.Opened the add customer screen using the menu
2.Entered a new customer (details are attached in spreadsheet)
3.Selected customer as “corporate”
4.Added to branch “Littleton”
5.Pressed add
6.Got error message “cannot find object”
Isolation
I tried on a few different branches and the system worked without issue when the
branch was classified as open. The Incident only occurred when the branch was
classified as closed
Test Management - Summary
We learned about the Test organization
• The importance of independent testing
• The types of Independent test organisations
• The benefits and drawbacks of independent testing within an organization
• The different team members to be considered for the creation of a test team
• The tasks of typical Test Leader and Tester
We learned about Test planning and estimation

• The different levels and objectives of test planning
• The purpose and content of the Test Plan and what we need to consider when
writing the Test Plan
• Difference between different estimation approaches: the metrics-based
approach and the expert-based approach
• Test preparation and execution tasks that need planning
• The need for, and examples of, adequate exit criteria for specific test levels
and groups of test cases
We learned about Test progress monitoring and control

• Why we need to Monitor testing and how, i.e. what metrics we
gather
• Understanding and interpreting the metrics for Test Reporting, plus
the contents of a Test Report
• Why we need Test Control
We learned about Configuration Management

• Why we need CM for testing and how it supports testing
• Typical Configuration items, including Testware
We learned about Risks and Testing

• What a Risk is
• The two main types of Risk - Project and Product
• That risks are determined by likelihood (of happening) and impact (harm
resulting if it does happen)
• How we should employ a Risk Based Testing approach – benefits Testing
and the Project
And finally we learned about Incident Management

• The definitions of Incidents and Defects
• The Benefits of Incident Management
• The Attributes of an Incident
• How we should track and Analyse Incidents
• And what makes a Good Incident Report
03-23-05
Version 2.3

Types of Test tool

Effective Use of Tools – Benefits and Risks
Introducing a Tool into an Organisation
Summary
Types of Test Tools
Test Tool Classification

Tool Support for Management of Testing
Tool Support for Static Testing
Tool Support for Test Specification
Tool Support for Execution and Logging
Tool Support for Performance and Monitoring
Tool Support for Specific Application Areas
Tool Support using other tools
Types of Test Tool
Test Tool Classification (1)

• Tools are classified according to the test activities they support
(e.g. test management, static testing, etc.)
• Some tools may support multiple activities while others just one
• Tool vendors may provide a suite of (hopefully) integrated tools
for the testing lifecycle. (Major vendors include Mercury,
Compuware, Borland, etc.)
• Tools can be any piece of purchased or locally developed software
• Tools can be tremendously helpful in improving the efficiency and
quality of testing…
• …but can also be a waste of time and effort
Types of Test Tool
Test Tool Classification (2)

• Some tools may be intrusive – i.e. they affect the test outcome itself – this is
the Probe Effect
– For example, some tools that measure the performance of a piece of code may insert
code to count or time instructions – though very detailed information can be obtained,
the timings will be affected simply because of the extra code
• Some tools offer greater support to developers rather than testers e.g., Static
Analysers are more commonly used by developers
• Some tools offer greater support for different parts of the software development
lifecycle (e.g. Unit Testing, System Testing)
• Computer Aided Software Test Tools (CAST Tools) is a common term used in
testing circles
Types of Test Tool
“Just like any carpenter, a tester needs a toolbox

full of tools to get the job done… But
...like a good carpenter, a good tester is not
defined by his tools, but by his skill at using
them ”
Scott Loveland et al 2005
Types of Test Tool
Tool Support for Management of Testing (1)

• Management tools can be applied across the entire software development
lifecycle:
– during unit testing, acceptance testing, etc.
– and by all project staff - by managers, developers, testers, etc.
Types of Test Tool

Test Management Tools:
• Help Manage tests and activities
– Provide management and control of test documentation, test cases, specifications
and results
– Support scheduling of tests, logging results, recording test environments, etc.
• Interface to other tools
– Link to test automation tools to run tests automatically
– Link to defect tracking tools to allow cross referencing to incidents/defects
– Link to requirements management tools to cross reference tests and requirements
• Support Version control
– Allow test cases, logs etc. to be baselined, “checked-out”, modified, retrieved and
controlled (often via another ‘linked’ tool)
Types of Test Tool

Test Management Tools - continued
• Allow Traceability
– Allow traceability to be recorded between test and requirements
– Allow traceability to associated defects/incidents
• Support Logging
– Allow recording of results within the tool
– Have results linked to ‘useful’ information (who, when, on what hardware, etc.)
• Provide Analysis
– Produce analysis reports on testing progress and test coverage
– Produce reports on tests passed and failed
– Produce reports on incidents raised
– Tools differ in their ease of use, accuracy of the information and portability of results for
distribution
Types of Test Tool

Requirements Management Tools
• Primary purpose is to store requirements (as simple text, as Use Cases, formal
language, etc.)
• Generally provide a means of recording the priority of each requirement
• To support testing, such tools can:
– Verify and validate requirements (consistency checking, missing links to other
requirements)
– Allow tests cases to be linked to the requirements
• Test Teams should review requirements early in the lifecycle to check for
consistency, testability and to allow test cases to be constructed
Types of Test Tool

Incident Management Tools
• Also known as defect tracking tools

• Record defects/problems/incidents/anomalies with virtually any aspect of the project
• E.g. operational problems, testing failures, documentation errors. Fortunately, it is
not common to use them for reporting problems with people
• Widely used by all members of a project team – particularly testers

• They support the prioritisation of incidents (e.g. High for incidents that must be
fixed in 4 hours through to Low for incidents that may never get fixed.)
Types of Test Tool

Incident Management Tools – continued
• Allow incidents and resulting actions to be assigned to project members.

E.g. to Investigate, to test, to cost, etc.
• Record current status and history of an incident. E.g. New, or Rejected, or
Resolved, or Waiting for test, etc.
• Some generate analysis reports on trends in incident creation, resolution,
effort recorded, etc.
• Most can be tailored to specific project needs e.g. to define project specific
states or priorities, to specify the layout of information, to construct
predefined reports, etc.
Types of Test Tool

Configuration Management Tools
• Allow software baselines to be kept and managed (See notes for
background information)
• Not strictly testing tools, but are heavily used by test teams to:
– obtain specific version of software (code, documents, etc.)
– maintain their own test material
• Traceability between testware and software can be achieved by applying
build/release labels to corresponding items
• Useful (often essential) when managing different version of a system for
different target environments or different operational releases
Types of Test Tool
Tool Support for Static Testing (1)

Review Process support tools
• During a review:
– Documents are issued, people are nominated, roles are assigned
– Templates and guidelines are distributed
– Feedback is produced, times are recorded, people are ‘nagged’ etc.
• Tools support these activities by storing the documents, providing
communication mechanisms, recording metrics, issue reminders, etc.
• Some support on-line reviews where people are geographically dispersed.
(Within EDS, eRoom is used to support online reviews)
• Many companies (including EDS) use tailored Spreadsheets to record
comments and generate review metrics
Types of Test Tool

Static analysis Tools
• Static analysis tools provide information about the quality of the software –
by examining the code but not executing the code.
• Static Testing can often be carried out early in the development lifecycle –
thus defects are detected sooner and (generally) more cheaply
• Static analysis tools usually give objective measurements of various
characteristics of the software, such as the complexity measure
• They are used by developers, testers and quality assurance personnel
• Code defects detected include unused code, unused variables, coding
standard violations, etc.
• Vendors include McCabe, Segue and many more.
Types of Test Tool

Modelling tools
• Projects may have analysis models, design models, data models, state
transition models, etc….
• These ‘models’ of the system are checked for defects
– For example, a state transition model may have states with no entry point or
exit point, etc.
• Tools will require the model to be in a format they can ‘read’ - Perhaps as a
formal language or as part of the tool itself
• Some tools actually produce test cases from the models
• These tools allow testing to take place early - defects are identified sooner
Types of Test Tool
Tool Support for Test Specification (1)

Test Design Tools
• Test design tools generate some or all of the following automatically
– Test inputs (i.e. the test data) this would be data from external system or user
– Test cases
– Expected results – i.e. may use a “test oracle”
• They use requirements, analysis models, data models, state models, code
• Tools will require the model to be in a format they can ‘read’ - Perhaps as a
formal language or as part of the tool itself
• The tools either have built in or configurable test generation criteria
• They may look for input boundary conditions, state transitions, paths
through the system, etc.
Types of Test Tool

Test Design Tools - continued
• Their value is from:
– Ensuring thoroughness of test coverage by generating independent tests and
data
– The time saved in test generation
• Some Test design tools also generate test stubs and templates from models
and code
Types of Test Tool

Test data preparation tools
• Enable data to be selected from existing databases, files or captured
transmissions
• Enable data to be created, generated, manipulated and edited for use in
tests
• The most sophisticated tools can deal with a range of file and database
formats
• The application under test is often used to ‘Drive’ the data into databases,
etc.
• They have the benefit of generating anonymous data for data protection
and security purposes.
– i.e. They can generate data in the right format, with the right bounds, but isn’t
actually live/real data
Types of Test Tool
Tool Support for Execution and logging (1)

Test execution tools
• Provide capture, replay and comparison facilities
• Tools simulate mouse movement, button clicks and keyboard inputs
• Can recognise GUI objects and states for later comparison – windows, fields,
buttons, bit maps, etc.
• Test scripts are normally captured in a programmable script language (VB, C,
Java, etc.)
• Additional code can be added to the script for data manipulations,
verifications, iterations, etc.
• When executed, tests run automatically, following the recorded/ programmed
script using the defined input data
• Compare results with the predefined expected values and record the logs in
the tool (i.e. they use a Comparator).
• A test Comparator may use a test oracle , especially if it is automated.
Types of Test Tool

Test execution tools - continued
• Generating automated scripts can be time consuming, challenging and fun
• They are best used for tests which will be repeated many times (these tools
are most often used to automate regression tests)
• Using the Capture-replay facilities of such tools, testers can record their
actions – which can be very useful when failures occur. The complete history
of the user actions can be replayed and studied.
• Can be used to replicate Users, i.e. multi-user testing.
Types of Test Tool

Test harness/unit test framework tools
• test harnesses, drivers and stubs simulate other parts of the system or other
connecting systems (or people)
• They are used to:
– Simulate other components that are not yet available – so testing can continue
– Simulate other components to allow detailed, controllable testing of a specific component
– Simulate events that may be difficult (even dangerous) to produce with the whole
system (e.g. database failures or nuclear meltdown)
• Commercially available tools exist, but custom-written programs also fall into this
category
• Simulators also come under this category
Types of Test Tool

Test harness/unit test framework tools - continued
• Test harnesses inject data into a component and capture outputs from the
component
• Comparisons with previous runs and pre-defined expected outcomes can be
performed
• Some frameworks are designed for developers for unit/component testing
-these are unit test frameworks (e.g. JUnit, HTTPUnit)
Types of Test Tool

Test comparators
• Comparison tools are used to detect differences between actual results and
expected results
• Standalone comparison tools normally deal with a range of file or database
formats
• Test execution tools usually have built-in comparators that deal with
character screens, GUI objects or bitmap images
• These tools often have filtering or masking capabilities - they can ‘ignore’
rows or columns of data or areas on screens (e.g. volatile date fields)
Types of Test Tool

Coverage measurement tools
• These tools are dynamic test tools and allow test coverage to be monitored
• Most commercial tools are designed for code coverage i.e. how much of the
code (statements, branches) has been executed
• Provide objective measures of test coverage when tests are executed
• Some tools add extra code to the source code (called instrumentation) to
allow detailed information to be obtained – but recall the Probe Effect
• After execution, the log file is analysed and coverage statistics generated
• Statistics are provided on the most common coverage measures such as
statement or branch coverage
Types of Test Tool

Security tools
• With the rise of the internet, the need for preventing malicious attacks on
systems has increased
• Tools have emerged which test a systems vulnerabilities to attacks
• Virus scanners monitor for unexpected arrival of damaging code (via email,
embedded in web pages or documents, etc.)
• Denial of service attacks
– Tools examine code and attempt to cause a failure by injecting rogue data.
– They may look for weaknesses in code (in HTML, .NET, etc.) deliberately trying
to damage the service
• Penetration testing – trying to break in. Tools will look for security
loopholes, monitor traffic, or password weaknesses, etc. and attempt to
gain access to machines or services
Types of Test Tool
Tool Support for Performance & monitoring (1)

Dynamic Analysis tools
• These are used to find defects that are apparent only
when the software is actually running (see also Static
Analysis Tools)
• These tools are most commonly used to: -
– identify unassigned pointers
– monitor the allocation, use and de-allocation of memory to
flag memory leaks
• These tools are generally used by developers
Types of Test Tool

Performance testing tools
• Performance test tools have two main facilities - load generation measurement and
test transaction measurement
• Load generation can simulate either multiple users or high volumes of input data
• Load generation is done either by driving the application using its user interface or
by test drivers
• Records of the numbers of transactions executed are logged
• Response times are taken and logged for selected transactions
• Performance testing tools normally provide reports based on test logs and graphs
of load against response times
• Tools in this category are also called load test tools or stress test tools
Types of Test Tool

Monitoring tools
• These tools are not strictly test tools, but enable failures or potential
failures to be detected
• Monitoring tools monitor the systems memory, CPU usage, file spaces,
network traffic, disk I/O, current processes running, database optimisation
(e.g. monitoring embedded into products such as Oracle) etc.
• They can be ‘tuned’ to report potential failures – e.g. file space 80% full,
process XYZ should always be running, etc.
Types of Test Tool
Tool Support for Specific Application areas

• Tools exist which are specialised for a specific task, environment or
application
• For example:
– Harnesses, monitoring tools, etc for embedded systems – where their
specialised hardware and application requires specialised tools
– hyperlink testing tools (Spiders) are used to check that no broken hyperlinks are
present on a web site
– Tools for a specific operating system (Windows, UNIX, etc.)
– Tools for a specific language (JAVA, C++, .Net, etc)
– Performance tools specifically for WEB based user interfaces
Types of Test Tool
Tool Support using other tools

• Many other tools are used to support the test effort.
• For example:
– Word processing tools – widely used for producing test plans, test reports, to
record test cases, etc.
– Spreadsheets – also widely used for test cases, cross-referencing, analysis of
data, reporting, etc.
– Database interrogation tools (e.g. SQL, Toad, Raptor) – database interrogation is
performed to validate the contents of tables. Simple SQL statements can be
used or more sophisticated tools (Toad, Raptor, etc. which allow the ‘novice’ to
interrogate a database)
– Operating system utilities and scripts to monitor resources, generate data, run
tests
– Code debugging tools – to step through code, monitor variables, etc.
Effective Use of Tools – benefits and risks
Potential Benefits and risks – all tools (1)

• Some tools bring quick and easy benefit to the test effort, but…
• Some tools require considerable investment in time and thought
to realise their benefits
• Don’t just look at the adverts or listen to the salesman

Potential Benefits of using tools
• Repetitive work reduced – “Relieve the boredom”
– re-running the same test many times can be extremely boring and
error prone
– Tools can reduce the time we spend on repetitive tasks
• Consistency and repeatability – “Errare Humanum Est” (to Err
is Human)
– Test identified by a tool or run by a tool can be more repeatable and
consistent. Tools do what they are told
• Objective assessment – tools have no “axe to grind”
– they are objective with their assessments and results
• Ease of access to information
– tools can give easy access to test information stored and present it in
multiple formats for distribution
• Automated tests can be run out of hours

Risks of using tools - every silver lining has a cloud
• Unrealistic expectation - Don’t just look at the adverts or listen to
the salesman. Tools may be rich in features – but are they useful
to your test effort?
• Underestimating how much effort/money is required to purchase,
install, get trained and simply get started with the tool
• Underestimating how much effort is required to gain experience
so that benefits can be realised
• Underestimating the effort to incorporate into the test process
• Underestimating the effort required to maintain the assets
(automated scripts may need to change for every change in the
data or user interface)
• Over-reliance – not considering other tests that need to be found
or test that need to be run
“Selection of the tool affects the effectiveness and

efficiency of testing. .
The technique for hammering a nail into a piece of
wood is well understood…
… if the wrong hammer is selected the entire
process can be inefficient….”
William E. Perry 2000
Special Considerations for some tools (1)

Test Execution tools
• Scripts for such tools are generated in a scripting language
(usually VB, C, Java)
• Scripts can be very time-consuming to produce and difficult for
non programmers to maintain - which could delay significant
testing while creating automated tests
• Scripts do exactly what they are told – and nothing more. A
simple pop up from another application on the desktop could halt
the script
• Two approaches are used to assist with script generation:
– Data-driven approach – script is produced to read test data from
(typically) a spreadsheet. Testers run tests with different data by
manipulating the spreadsheet and not the script
– Key-word driven approach – scripts are produced using building
blocks – each given a key-word. E.g. “Logon”, “Add_User”,
“Delete_User”. Testers create tests by picking the ‘blocks’ they need

Performance Testing Tools
• Requires specialists to design, execute and analyse the results
• Often requires detailed understanding of how the system is
constructed (databases, networks, etc.)
• Often requires detailed understanding of expected system usage
profiles
• Need to be re-run when system ‘tuning’ takes place

Static Analysis Tools
• Code Static Analysers examine code and reports on defects and
‘bad practice’
• Are best used at the start of coding to ensure problems are
removed quickly
• But…
• Many systems contain automatically generated code, which often
causes static analysers to generate many reports
• Applying these tools to legacy code can produce a flood of
unhelpful messages
• Get the tool to filter out unwanted messages and so target the
required code

Test Management Tools
• Need to interface with other tools. E.g to import and export
information from word processors and spreadsheets
• Must help your test process – not hinder progress
• Must ensure the information they hold is readily available for
viewing and distribution
• Must produce reports easily
Principles to follow (1)

• Organisational Maturity –
– Tools should help the projects test effort and should fit into the culture
– Spending time introducing automated execution tools for rapid reaction
or prototype projects may be of no value
• Evaluation of product –
– Produce weighted Acceptance Criteria
– Determine test tool options, prices and conditions
– Produce a Return on Investment (ROI)
– Investigate the market to see what products are available
– Produce a Short List of suppliers
– Conduct evaluation tests against application(s) with at least two
suppliers and score results

• Proof-of-concept – before rolling out the tool across the
organisation, “test it out” on a manageable, resourced, pilot
project.
– If it fails – the impact will be minimised
– If it succeeds – greater experience will be available and organisational
roll out can be better supported
• Evaluate the vendor
– Discuss contractual issues – company stability, licensing and
maintenance costs, usage restrictions
– Evaluate training courses, technical support and consultancy
– Review the tools development roadmap – is it going in the right
direction for you
• Internal Coaching
– Identify staff to be trained
– Who will provide internal support for the tools installation and usage

• Other questions to ask:
– What protocols, languages, networks, hardware are being used now
and in future?
– What are the budget and timescale for delivery?
– What is the problem to be solved?
– Are there alternative solutions – other than tool purchase?
– Are there any specific tool features and characteristics?
– Do you need a project specific tool, or is this for the organisation?
– But avoid too many tools of the same type
- tools are not just for Christmas
“Identifying, investigating, and evaluating tools

are important responsibilities of a test leader…
… But it’s important to recognize how much time
and effort is spent doing so.”
Scott Loveland et al 2005
Objectives of the Pilot project

• Learn about the tool
– No training course or brochure will give enough information about how
the tool actually works in your environment.
– It is vital to learn about the tool before rolling it out for wider use
• Is the tool a good fit?
– Will the organisations processes accommodate this tool?
– What organisational processes need to be updated?
– Can it be integrated with other tools sets for improved effectiveness?
• Generate a best practice guide
– create guides and work instructions in the use of the tools
• Benefits?
– understand and measure as objectively as possible the benefits to be
gained
– will it be worth the effort?
Success factors for tool roll-out (1)

• Incremental roll out
– Avoids a sudden increase in demands for extra support and training
– Allows experience to be gained and passed on ensuring a more successful roll out
– If problems are encountered, there is less impact if the roll out is delayed or aborted
• Adapt processes
– Processes may benefit from changes because of the tool
– For example, when and how regression testing is done (with automation), how test
cases are developed and recorded, etc.
• Training
– To avoid tools becoming shelfware new users will need local training and support
– Some users will be easily deterred by new tools

• Produce Guidelines
– Tools may come with extensive user guides but it is useful to
supplement them with local conventions, FAQs, standards, etc.
– make the guidelines easily accessible
• Learn lessons and monitor usage
– As tool usage increases capture users experience for sharing and
process improvement. Consider:
• Web based forums
• Email distribution groups
• Discussion groups
• Nominated tool authorities to approve changes
• Regular assessment of benefits vs costs

• Other success factors:
– Maintain close relationship with tool vendor – to understand their
future plans, to get quick access to information
– Monitor test tool market for new or improved products
– Ongoing research into how to better utilise test tools
– Use consultants to resolve short term or difficult issues where there is
no gain from wasting resource time on complex one-off problems
EDS Application Engineering Toolkit (AET)
• The Applications Engineering Tools team evaluates, recommends and pilots

standard Enterprise Application Development Life Cycle tools, utilities and
frameworks to enable the delivery of service offerings within the global
applications engineering organization
• AET Objectives are to: -
– Improve applications development sustainable productivity and reduce rework
through the deployment of integrated toolkit and framework recommendations
– Decrease operating cost and improve functionality for dollar spent through
implementation of application software and database product replacement
recommendations
• http://www.gsms-ea.eds.com/Tools/index.html
Collateral on Tools
External Sites (non supplier based)

• QA Forums - http://qaforums.com/
• StickyMinds - http://www.stickyminds.com/
• Automation Junkies - http://www.automationjunkies.com/
• Software Test & Performance - http://www.stpmag.com/
Summary (1)
Summary (2)
Test management tools Test management Requirements management
Incident management Configuration management
Static testing tools Static analysis Review process support
Modelling
Test specification tools Test design Test data preparation
Test execution and logging tools Test execution Test harnesses

Test comparators Coverage measurement
Security
Performance, monitoring tools Dynamic analysis Performance/load/stress
Monitoring
Application specific tools Embedded systems Web testing
Language specific
Other tools Word Processors Operating system utilities
Spreadsheets SQL, Code debugging
Summary (3)
• Then we looked at the effective use of tools:
– Benefits of tools:
• Save time, thorough testing, reduce repetitive tasks, etc.
– And the risks:
• Underestimating time and effort, Over-reliance, etc.
– Special considerations for:
• Test execution tools, performance tools, static analysis tools and
test management tools
• And finally, we looked at how to introduce a tool:
– Main principles (evaluation, assessments, etc.)
– Pilot project objectives
– Success factors
Index
The following slides provides a quick reference guide to some of the key terms used in the course, to aid revision. Each term is hyperlinked and has slide numbers plus associated ISQTB Syllabus section numbers. Remember to use the Glossary of terms for exact definitions.
Term Slide(s) Topic Syllabus

reference
Acceptance Testing 116 Testing Throughout the S/W Lifecycle 2.2.4
Black Box Testing 197 Test Design Techniques 4.2
Boundary Value Analysis 211 Test Design Techniques 4.3.2
Component Integration Testing 85 Testing Throughout the S/W Lifecycle 2.2.2
Component Testing 80 Testing Throughout the S/W Lifecycle 2.2.1
Configuration Management 282 Test Management 5.4
Decision Tables 214 Test Design Techniques 4.3.3
Decision Testing 229 Test Design Techniques 4.4.2
Equivalence Partitioning 204 Test Design Techniques 4.3.1
Error 10 Fundamentals of Testing 1.1.2
Error Guessing 239 Test Design Techniques 4.5
Exit criteria 261 Test Management 5.2.3
Index
Term Slide(s) Topic Syllabus reference
Exploratory Testing 241 Test Design Techniques 4.5
Failure 12 Fundamentals of Testing 1.1.2
Fault 10 Fundamentals of Testing 1.1.2
Formal Review Process 166 Static Techniques 3.2.1
Functional Testing 131 Testing Throughout the S/W Lifecycle 2.3.1
Fundamental Test process 44 Fundamentals of Testing 1.4
General Principles of Testing 34 Fundamentals of Testing 1.3
Incident Management 295 Test Management 5.6
Independent Testing 248 Test Management 5.1.1
Informal Review 162 Static Techniques 3.2.3
Inspection 165 Static Techniques 3.2.3
Maintenance Testing 140 Testing Throughout the S/W Lifecycle 2.4
Non- Functional Testing 132 Testing Throughout the S/W Lifecycle 2.3.2
Product Risk 289 Test Management 5.5.2
Project Risk 288 Testing Throughout the S/W Lifecycle 5.5.1
Regression Testing 136 Testing Throughout the S/W Lifecycle 2.3.4
Re-testing (Confirmation) 135 Testing Throughout the S/W Lifecycle 2.3.4
Index
Risk Based Testing 287 Static Techniques 5.5

State Transition Testing 217 Test Design Techniques 4.3.4
Statement Testing 225 Test Design Techniques 4.4.1
Static analysis 178 Static Testing 3.3
Structural Testing 133 Testing Throughout the S/W Lifecycle 2.3.3
System Integration Testing 111 Testing Throughout the S/W Lifecycle 2.2.2
System Testing 95 Testing Throughout the S/W Lifecycle 2.2.3
Technical Review 164 Static Techniques 3.2.3
Test Case 191 Test Design Techniques 4.1
Test Condition 190 Test Design Techniques 4.1
Test Estimation 263 Test Management 5.2.4
Test Execution Schedule 194 Test Design Techniques 4.1
Test Monitoring/Control 272 Test Management 5.3.1
Test Objectives 27 Test Management 1.2
Test Planning 256 Test Management 5.2.1
Test Procedure 193 Test Design Techniques 4.1
Index
Test Tool types 312 Test Tools 6.1
Use Case Testing 222 Test Design Techniques 4.3.5
V-model 72 Testing Throughout the S/W Lifecycle 2.1.1

Walkthrough 163 Static Techniques 3.2.3
White Box Testing 199 Test Design Techniques 4.2
Statement of Confidentiality
The information contained in this document is proprietary to

EDS. EDS submits this document with the understanding that it
will be held in strict confidence and will not be disclosed,
duplicated or used, in whole or in part, for any purpose other
than the evaluation of EDS’ qualifications, without the prior
written consent of EDS.
EDS is a registered mark and the EDS logo is a trademark of Electronic Data Systems
Corporation.
EDS is an equal opportunity employer and values the diversity of its people.
Copyright 2006 Electronic Data Systems Corporation. All rights reserved.
30
Version
Sept 2005
2.3
Presentation and Course owner Paul Weymouth, UKIA Testing ADU Paul.Weymouth@eds.com
Presentation and Course contributors:
•Paul Weymouth, Testing Architect UKIA Testing ADU
•Mark Otter, Senior Test Consultant Australia South ADU
•Dave Broughton, Testing Architect UKIA Testing ADU
EDS and the EDS logo are registered trademarks of Electronic Data Systems Corporation. EDS is an equal opportunity employer
and values the diversity of its people. © 2005 Electronic Data Systems Corporation. All rights reserved.

Istqb TFC v2.3

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Istqb TFC v2.3

Enviado por

Direitos autorais:

Formatos disponíveis

03-23-05

EDS ISTQB Testing Foundation Course

Presentation by Paul Weymouth, UK Testing ADU

Slide 1 • EDS Internal

• Click on the symbol to go to the next slide.

• Click on the symbol to go to the previous slide.

• Click on the symbol to go to the main Areas Covered slide.

• Click on the symbol to move on to associated slides.

• Click on the symbol to go back to the previous menu.

• You may also see the following symbols displayed :-

Refers to the ISTQB Glossary Of Terms.

This is a ‘Pearl of Wisdom’. Typically a quote

• The primary objective of this course is to provide testing foundation

Index of key terms

Slide 6 • EDS Internal

Why Testing is Necessary

Software Systems – Some context

Software Systems – When things go wrong

Causes of Software Failure

* ISTQB Standard Glossary of Terms

Errors, Defects and Failures

Errors, Defects and Failures

• A Failure is a ‘Deviation of the component or system from its

* ISTQB Standard Glossary of Terms

Errors, Defects and Failures

Deviation of the component or

The Role of Testing

Testing and Quality

Testing and Quality

Testing Pearl of Wisdom

• “Quality is not intangible.

Bill Hetzel 1988

Why exhaustive testing is impossible

• Exhaustive testing of complex software applications:

Testing Pearl of Wisdom

• “In any form of testing it is impossible to achieve

Bill Hetzel 1988

Why don’t we test everything ?

Approximate total for exhaustive testing

Test length = 1 sec then test duration = 17.7 days

So, How Much Testing is Enough?

So, How Much Testing is Enough?

So, How Much Testing is Enough?

• Exit criteria may be defined in terms of :-

A Definition (and a Misconception)

A Definition (and a Misconception)

Testing Pearl of Wisdom

• “Any activity that is undertaken with the

– Development testing (e.g. component, integration and system testing) -

• In some cases the main Objective of testing may be to assess the

• More on these test stages later in the course

Testing Pearl of Wisdom

• “Testing is the process of executing a program or

The Seven Key Principles

1. Testing shows presence of Defects

The Seven Key Principles

• We test to find Faults (a.k.a Defects)

Testing Pearls of Wisdom

• “The probability of the existence of more errors in a section of

The Seven Key Principles

• We have learned that we cannot test everything (i.e. all combinations

• That is we must Prioritise our testing effort using a Risk Based