Testing

Defect testing
• Testing programs to establish the presence of system defects

Objectives
• To understand testing techniques that are geared to discover program faults
• To introduce guidelines for interface testing
• To understand specific approaches to object-oriented testing
• To understand the principles of CASE tool support for testing

Topics covered
• Defect testing
• Integration testing
• Object-oriented testing
• Testing workbenches

The testing process

● Component testing
 Testing of individual program components
 Usually the responsibility of the component developer (except sometimes for
critical systems)
 Tests are derived from the developer’s experience
● Integration testing
 Testing of groups of components integrated to create a system or sub-system
 The responsibility of an independent testing team
 Tests are based on a system specification
Testing phases

Component Integration
testing testi ng

Software  developer Independent  testing team

Defect testing
●The goal of defect testing is to discover defects in programs
●A successful defect test is a test which causes a program to behave in an anomalous
way
●Tests show the presence not the absence of defects

Testing priorities
●Only exhaustive testing can show a program is free from defects. However, exhaustive
testing is impossible
●Tests should exercise a system's capabilities rather than its components
●Testing old capabilities is more important than testing new capabilities
●Testing typical situations is more important than boundary value cases

Test data and test cases

●Test data Inputs which have been devised to test the system

1
 Testing

●Testcases Inputs to test the system and the predicted outputs from these inputs if the
system operates according to its specification
The defect testing process

Test Test Test Test

cases data results reports

Design test Prepare test Run program Compare r esults

cases data with test data to test cases

Black-box testing
●An approach to testing where the program is considered as a ‘black-box’
●The program test cases are based on the system specification
●Test planning can begin early in the software process

Inputs causing
anomalous
Input test data I behaviour
e

System

Outputs which reveal
the presence of
Output test results Oe defects

Equivalence partitioning
●Input data and output results often fall into different classes where all members of a
class are related
●Each of these classes is an equivalence partition where the program behaves in an
equivalent way for each class member
●Test cases should be chosen from each partition

2
 Testing

Invali d  in pu ts Vali d  in pu ts

Sy stem

Ou tput s

●Partitionsystem inputs and outputs into ‘equivalence sets’

 If input is a 5-digit integer between 10,000 and 99,999, equivalence partitions are
<10,000, 10,000-99, 999 and > 10, 000
●Choose test cases at the boundary of these sets
 00000, 09999, 10000, 99999, 10001
3 11
4 7 10

Less than 4 Between 4 and 10 More than 10

Number of  input values

9999 100000
10000 50000 99999

Less than 10000 Between 10000 and 99999 More than 99999

Input values
Search routine specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key)
or
-- the element is not in the array
( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

3
 Testing

Search routine - input partitions

●Inputs which conform to the pre-conditions
●Inputs where a pre-condition does not hold
●Inputs where the key element is a member of the array
●Inputs where the key element is not a member of the array

Testing guidelines (sequences)

●Test software with sequences which have only a single value
●Use sequences of different sizes in different tests
●Derive tests so that the first, middle and last elements of the sequence are accessed
●Test with sequences of zero length

Search routine - input partitions

Search routine ­ input partitions
Array Element
Single value In sequence
Single value Not in sequence
More than 1 value First element in sequence
More than 1 value Last element in sequence
More than 1 value Middle element in sequence
More than 1 value Not in sequence

Input sequence (T) Key (Key) Output (Found, L)

17 17 true, 1
17 0 false, ??
17, 29, 21, 23 17 true, 1
41, 18, 9, 31, 30, 16, 45 45 true, 7
17, 18, 21, 23, 29, 41, 38 23 true, 4
21, 23, 29, 33, 38 25 false, ??
©Ian Sommerville 2000  Software Engineering, 6th edition. Chapter 20 Slide  19

Structural testing

●Sometime called white-box testing

●Derivation of test cases according to program structure. Knowledge of the program is
used to identify additional test cases
●Objective is to exercise all program statements (not all path combinations)

White-box testing

4
 Testing

Test data

Tests Derives

Component Test
code outputs

Binary search (Java)

class BinSearch {

// This is an encapsulation of a binary s earch function that takes an array of
// ordered  objects and a k ey a nd returns an o bject with 2 attributes namely
// index ­ the  value of the array  index
// found ­ a boolean indicating whether or not the key  is in the array
// An object is returned because it is not possible in J ava to pass basic types by
// reference to a function and so return two  values
// the key  is ­1 if the element is not found

public static v oid search ( int key, int [] elemArray, Result r )
{
int bottom = 0 ;
int top = elemArray.length ­ 1 ;
int mid ;
r.found = false ; r.index = ­1 ;
while (  bottom <= top )
{
mid = (top + bottom) / 2  ;
if (elemArray  [mid] == key)
{
r.index  = mid ;
r.found = true ;
return ;
} // if part
else
{
if (elemArray  [mid] < key)
bottom = mid + 1 ;
else
top = mid ­ 1 ;
}
} //while loop
} // search
Binary search (Java)
} //BinSearch

Binary search - equiv. partitions

●Pre-conditions satisfied, key element in array
●Pre-conditions satisfied, key element not in array
●Pre-conditions unsatisfied, key element in array
●Pre-conditions unsatisfied, key element not in array
●Input array has a single value
●Input array has an even number of values
●Input array has an odd number of values
Binary search equiv. partitions

5
 Testing

Equivalence class boundaries

Elements < Mid Elements  > Mid

Mid­point
Binary search - test cases

Binary search ­ test cases

Input array (T) Key (Key) Output  (Found, L)

17 17 true, 1
17 0 false, ??
17, 21, 23, 29 17 true, 1
9, 16, 18, 30, 31, 41, 45 45 true, 7
17, 18, 21, 23, 29, 38, 41 23 true, 4
17, 18, 21, 23, 29, 33, 38 21 true, 3
12, 18, 21, 23, 32 23 true, 4
21, 23, 29, 33, 38 25 false, ??

©Ian Sommerville 2000  Software Engineering, 6th edition. Chapter 20 Slide  25

Path testing
●The objective of path testing is to ensure that the set of test cases is such that each
path through the program is executed at least once
●The starting point for path testing is a program flow graph that shows nodes
representing program decisions and arcs representing the flow of control
●Statements with conditions are therefore nodes in the flow graph
Program flow graphs
●Describes the program control flow. Each branch is shown as a separate path and
loops are shown by arrows looping back to the loop condition node
●Used as a basis for computing the cyclomatic complexity
●Cyclomatic complexity = Number of edges - Number of nodes +2

Cyclomatic complexity
●The number of tests to test all control statements equals the cyclomatic complexity
●Cyclomatic complexity equals number of conditions in a program
●Useful if used with care. Does not imply adequacy of testing.
●Although all paths are executed, all combinations of paths are not executed
Binary search flow graph

6
 Testing

while bottom < = top
bottom > top
2

3 if (elemArray [mid] == key

8 4
(if (elemArray [mid]<  key

5 6
9

Binary search flow graph

Independent paths
●1, 2, 3, 8, 9
●1, 2, 3, 4, 6, 7, 2
●1, 2, 3, 4, 5, 7, 2
●1, 2, 3, 4, 6, 7, 2, 8, 9
●Test cases should be derived so that all of these paths are executed
●A dynamic program analyser may be used to check that paths have been executed
Integration testing
●Testscomplete systems or subsystems composed of integrated components
●Integration testing should be black-box testing with tests derived from the specification
●Main difficulty is localising errors
●Incremental integration testing reduces this problem
Incremental integration testing
A T1
T1
A
T1 T2
A B
T2
T2 B T3
T3
B C
T3 T4
C
T4
D T5

Test sequence Test sequence Test sequence

1 2 3
Approaches to integration testing
●Top-down testing
•Start with high-level system and integrate from the top-down replacing individual

7
 Testing

components by stubs where appropriate

●Bottom-up testing
•Integrate individual components in levels until the complete system is created
●In practice, most integration involves a combination of these strategies

Top-down testing
Testing
Level 1 Level 1 . . .
sequence

Level 2 Level 2 Le vel 2 Level 2

Le vel 2
stubs

Le vel 3
stubs

Bottom-up testing

Test
drivers
Testing
Level N Level N Le vel N Level N Level N
sequence

Test
drivers
Level N–1 Level N–1 Level N–1

Tetsing approaches
●Architectural validation
•Top-down integration testing is better at discovering errors in the system
architecture

●System demonstration
•Top-down integration testing allows a limited demonstration at an early stage in the
development
●Test implementation
•Often easier with bottom-up integration testing
●Test observation

8
 Testing

•Problems with both approaches. Extra code may be required to observe tests
Interface testing
●Takes place when modules or sub-systems are integrated to create larger systems
●Objectives are to detect faults due to interface errors or invalid assumptions about
interfaces
●Particularly important for object-oriented development as objects are defined by their
interfaces

Test
cases

A B

Interfaces types
●Parameter interfaces
•Data passed from one procedure to another
●Shared memory interfaces
•Block of memory is shared between procedures
●Procedural interfaces
•Sub-system encapsulates a set of procedures to be called by other sub-systems
●Message passing interfaces
•Sub-systems request services from other sub-systems

Interface errors

●Interface misuse
•A calling component calls another component and makes an error in its use of its
interface e.g. parameters in the wrong order
●Interface misunderstanding
•A calling component embeds assumptions about the behaviour of the called
component which are incorrect
●Timing errors
•The called and the calling component operate at different speeds and out-of-date
information is accessed
Interface testing guidelines
●Design tests so that parameters to a called procedure are at the extreme ends of their

9
 Testing

ranges
●Always test pointer parameters with null pointers
●Design tests which cause the component to fail
●Use stress testing in message passing systems
●In shared memory systems, vary the order in which components are activated

Stress testing
●Exercises the system beyond its maximum design load. Stressing the system often
causes defects to come to light
●Stressing the system test failure behaviour.. Systems should not fail catastrophically.
Stress testing checks for unacceptable loss of service or data
●Particularly relevant to distributed systems which can exhibit severe degradation as a
network becomes overloaded
Object-oriented testing
●The components to be tested are object classes that are instantiated as objects
●Larger grain than individual functions so approaches to white-box testing have to be
extended
●No obvious ‘top’ to the system for top-down integration and testing

Testing levels
●Testing operations associated with objects
●Testing object classes
●Testing clusters of cooperating objects
●Testing the complete OO system

Object class testing

●Complete test coverage of a class involves
•Testing all operations associated with an object
•Setting and interrogating all object attributes
•Exercising the object in all possible states
●Inheritance makes it more difficult to design object class tests as the information to be
tested is not localised
Weather station object interface
●Testcases are needed for all operations
●Use a state model to identify state transitions for testing
●Examples of testing sequences

•Shutdown Waiting Shutdown

•Waiting Calibrating Testing Transmitting Waiting
•Waiting Collecting Waiting Summarising Transmitting Waiting

Object integration
●Levels of integration are less distinct in object-oriented systems
●Cluster testing is concerned with integrating and testing clusters of cooperating objects
●Identify clusters using knowledge of the operation of objects and the system features
that are implemented by these clusters
Approaches to cluster testing

10
 Testing

●Use-case or scenario testing

•Testing is based on a user interactions with the system
•Has the advantage that it tests system features as experienced by users
●Thread testing
•Tests the systems response to events as processing threads through the system
●Object interaction testing
•Tests sequences of object interactions that stop when an object operation does not
call on services from another object
Scenario-based testing
●Identify
scenarios from use-cases and supplement these with interaction diagrams that
show the objects involved in the scenario
●Consider the scenario in the weather station system where a report is generated

Collect weather data

:CommsController :WeatherStation :WeatherData

request (report)

acknowledge ()
report ()
summarise ()

send (report)
reply (report)

acknowledge ()

Weather station testing

●Thread of methods executed
•CommsController:request WeatherStation:report WeatherData:summarise
●Inputs and outputs
•Input of report request with associated acknowledge and a final output of a report
•Can be tested by creating raw data and ensuring that it is summarised properly
•Use the same raw data to test the WeatherData object
Testing workbenches
●Testing is an expensive process phase. Testing workbenches provide a range of tools
to reduce the time required and total testing costs
●Most testing workbenches are open systems because testing needs are organisation-

11
 Testing

specific
●Difficult to integrate with closed design and analysis workbenches

A testing workbench
Test data
Specification
generator

Source Test Oracle

code manager Test data

Dynamic Program Test Test

analyser being tested results predictions

Execution File
Simulator
report comparator

Report Test results
generator report

Tetsing workbench adaptation

●Scripts may be developed for user interface simulators and patterns for test data
generators
●Test outputs may have to be prepared manually for comparison
●Special-purpose file comparators may be developed

Key points
●Test parts of a system which are commonly used rather than those which are rarely
executed
●Equivalence partitions are sets of test cases where the program should behave in an
equivalent way
●Black-box testing is based on the system specification
●Structural testing identifies test cases which cause all paths through the program to be
executed
●Test coverage measures ensure that all statements have been executed at least once.
●Interface defects arise because of specification misreading, misunderstanding, errors or
invalid timing assumptions
●To test object classes, test all operations, attributes and states
●Integrate object-oriented systems around clusters of objects

1. What is testing?
The act of checking if a part or a product performs as expected.
2. Why test?
• Gain confidence in the correctness of a part or a product.

12
 Testing

• Check if there are any errors in a part or a product.

3. What to test?
• During software lifecycle several products are generated.
• Examples:
 Requirements document
 Design document
 Software subsystems
 Software system
4. Test all!
• Each of these products needs testing.
• Methods for testing various products are different.
• Examples:
 Test a requirements document using scenario construction and simulation
 Test a design document using simulation.
 Test a subsystem using functional testing.
5. What is our focus?
• We focus on testing programs.
• Programs may be subsystems or complete systems.
• These are written in a formal programming language.
• There is a large collection of techniques and tools to test programs.
6. Few basic terms
Program:
A collection of functions, as in C, or a collection of classes as in java.
Specification
Description of requirements for a program. This might be formal or informal.
Test case or test input
A set of values of input variables of a program. Values of environment
variables are also included.
Test set
Set of test inputs
Program execution
Execution of a program on a test input.
Oracle
A function that determines whether or not the results of executing a program
under test is as per the program’s specifications.
Example:
Correctness
• Let P be a program (say, an integer sort program).
• Let S denote the specification for P.
• For sort let S be:
Sample Specification
• P takes as input an integer N>0 and a sequence of N integers called
elements of the sequence.
0< K< (e-1) for some e
• Let K denote any element of this sequence,

13
 Testing

• P sorts the input sequence in descending order and prints the sorted
sequence.
Correctness again
P is considered correct with respect to a specification S if and only if:
For each valid input the output of P is in accordance with the specification
S.
7. Errors, defects, faults
Error: A mistake made by a programmer
Example: Misunderstood the requirements.
Defect/fault: Manifestation of an error in a program.
Example:
Incorrect code: if (a<b) {foo(a,b);}
Correct code: if (a>b) {foo(a,b);}
Failure
• Incorrect program behavior due to a fault in the program.
• Failure can be determined only with respect to a set of requirement
specifications.
• A necessary condition for a failure to occur is that execution of the
program force the erroneous portion of the program to be executed. What
is the sufficiency condition?
Errors and failure

Inputs Error-revealing
inputs cause
failure

Program
Erroneous
Outputs outputs indicate
failure
8. Debugging
• Suppose that a failure is detected during the testing of P.
• The process of finding and removing the cause of this failure is known as
debugging.
• The word bug is slang for fault.
• Testing usually leads to debugging
• Testing and debugging usually happen in a cycle.
Test-debug cycle

14
 Testing

Test

Yes
Failure? No

Debug Yes
Testing No
complete?
Done!
9. Types of testing

Source of clues for

Object under test
test input construction

Testing: based on source of test inputs

• Functional testing/specification testing/black-box testing/conformance testing:
Clues for test input generation come from requirements.
• White-box testing/coverage testing/code-based testing
Clues come from program text.
Stress testing
Clues come from “load” requirements. For example, a telephone system must be
able to handle 1000 calls over any 1-minute interval. What happens when the
system is loaded or overloaded?
Performance testing
Clues come from performance requirements. For example, each call must be
processed in less than 5 seconds. Does the system process each call in less
than 5 seconds?

15
 Testing

Fault- or error- based testing

Clues come from the faults that are injected into the program text or are
hypothesized to be in the program.
Random testing
Clues come from requirements. Test are generated randomly using these clues.
Robustness testing
Clues come from requirements. The goal is to test a program under scenarios
not stipulated in the requirements.
OO testing
Clues come from the requirements and the design of an OO-program.
Protocol testing
Clues come from the specification of a protocol. As, for example, when testing for
a communication protocol.
Unit testing
Testing of a program unit. A unit is the smallest testable piece of a program. One
or more units form a subsystem.
Subsystem testing
Testing of a subsystem. A subsystem is a collection of units that cooperate to
provide a part of system functionality
Integration testing
Testing of subsystems that are being integrated to form a larger subsystem or a
complete system.
System testing
Testing of a complete system.
Regression testing
Test a subsystem or a system on a subset of the set of existing test inputs to
check if it continues to function correctly after changes have been made to an
older version.
10. Software Testing Fundamentals
Software testing is a critical element of software quality assurance and represents
the ultimate review of specification, design, and coding.
Software testing demonstrates that software function appear to be working
according to specifications and performance requirements.
Testing Objectives:
Myers [MYE79] states a number of rules that can serve well as testing objectives:
• Testing is a process of executing a program with the intent of finding an error.
• A good test case is one that has high probability of finding an undiscovered
error.
• A successful test is one that uncovers an as-yet undiscovered error.
The major testing objective is to design tests that systematically uncover types of
errors with minimum time and effort.

Software Testing Principles

Davids [DAV95] suggests a set of testing principles:
• All tests should be traceable to customer requirements.
• Tests should be planned long before testing begins.
• The Pareto principle applies to software testing.
• 80% of all errors uncovered during testing will likely be traceable to 20% of all

16
 Testing

program modules.
• Testing should begin “in the small” and progress toward testing “in the large”.
• Exhaustive testing is not possible.
• To be most effective, testing should be conducted by an independent third
party.
Software Testability
According to James Bach:
Software testability is simply how easily a computer program can be tested.
A set of program characteristics that lead to testable software:
• Operability: “the better it works, the more efficiently it can be tested.”
• Observability: “What you see is what you test.”
• Controllability: “The better we can control the software, the more the testing
can be automated and optimized.”
• Decomposability: “By controlling the scope of testing, we can more quickly
isolate problems and perform smarter retesting.”
• Simplicity: “The less there is to test, the more quickly we can test it.”
• Stability: “The fewer the changes, the fewer the disruptions to testing.”
• Understandability:”The more information we have, the smarter we will test.”
Test Case Design
Two general software testing approaches:
Black-Box Testing and White-Box Testing
Black-box testing:
knowing the specific functions of a software, design tests to demonstrate
each function and check its errors.
Major focus:
functions, operations, external interfaces, external data and
information
White-box testing:
knowing the internals of a software, design tests to exercise all internals of a
software to make sure they operates according to specifications and designs
Major focus: internal structures, logic paths, control flows, data flows internal data
structures, conditions, loop, etc.
White-Box Testing and Basis Path Testing
White-box testing, also known as glass-box testing.
It is a test case design method that uses the control structure of the procedural design
to derive test cases.
Using white-box testing methods, we derive test cases that
• Guarantee that all independent paths within a module have been exercised at
least once.
• Exercise all logical decisions on their true and false sides.
• Execute all loops at their boundaries and within their operational bounds.
• Exercise internal data structures to assure their validity.
Basic path testing (a white-box testing technique):
• First proposed by TomMcCabe [MCC76].
• Can be used to derive a logical complexity measure for a procedure design.
• Used as a guide for defining a basis set of execution path.

17
 Testing

• Guarantee to execute every statement in the program at least one time.

Cyclomatic Complexity
Cyclomatic complexity is a software metric
-> provides a quantitative measure of the global complexity of a program.
When this metric is used in the context of the basis path testing, the value computed for
cyclomatic complexity defines the number of independent paths in the basis set of a
program.
Three ways to compute cyclomatic complexity:
• The number of regions of the flow graph correspond to the cyclomatic complexity.
• Cyclomatic complexity, V(G), for a flow graph G is defined as
V(G) = E - N +2
where E is the number of flow graph edges and N is the number of flow
graph nodes.
• Cyclomatic complexity, V(G) = P + 1
where P is the number of predicate nodes contained in the flow graph G.
Deriving Test Cases
Step 1 : Using the design or code as a foundation, draw a corresponding flow graph.
Step 2: Determine the cyclomatic complexity of the resultant flow graph.
Step 3: Determine a basis set of linearly independent paths.
For example,
path 1: 1-2-10-11-13
path 2: 1-2-10-12-13
path 3: 1-2-3-10-11-13
path 4: 1-2-3-4-5-8-9-2-…
path 5: 1-2-3-4-5-6-8-9-2-..
Path 6: 1-2-3-4-5-6-7-8-9-2-..
Step 4: Prepare test cases that will force execution of each path in the basis set.
Path 1: test case:
value (k) = valid input, where k < i defined below.
value (i) = -999, where 2 <= I <= 100
expected results: correct average based on k values and proper totals.
Equivalence Partitioning
Equivalence partitioning is a black-box testing method
• divide the input domain of a program into classes of data
• derive test cases based on these partitions.
Test case design for equivalence partitioning is based on an evaluation of equivalence
classes for an input domain.
An equivalence class represents a set of valid or invalid states for input condition.
An input condition is:
• a specific numeric value, a range of values
• a set of related values, or a Boolean condition
Equivalence classes can be defined using the following guidelines:
• If an input condition specifies a range, one valid and two invalid equivalence class
are defined.
• If an input condition requires a specific value, one valid and two invalid
equivalence classes are defined.

18
 Testing

• If an input condition specifies a member of a set, one valid and one invalid
equivalence classes are defined.
• If an input condition is Boolean, one valid and one invalid classes are
defined.
Examples:
area code: input condition, Boolean - the area code may or may not be present.
input condition, range - value defined between 200 and 900
password: input condition, Boolean - a password nay or may not be present. input
condition, value - six character string.
command: input condition, set - containing commands noted before.

Boundary Value Analysis

• a test case design technique
• complements to equivalence partition
Objective:
Boundary value analysis leads to a selection of test cases that exercise bounding
values.
Guidelines:
• If an input condition specifies a range bounded by values a and b, test cases
should be designed with value a and b, just above and below a and b.

Example: Integer D with input condition [-3, 10],

test values: -3, 10, 11, -2, 0
• If an input condition specifies a number values, test cases should be developed to
exercise the minimum and maximum numbers. Values just above and below
minimum and maximum are also tested.

Example: Enumerate data E with input condition: {3, 5, 100, 102}

test values: 3, 102, -1, 200, 5
• Guidelines 1 and 2 are applied to output condition.
• If internal program data structures have prescribed boundaries, be certain to
design a test case to exercise the data structure at its boundary
Such as data structures:
–array input condition:
empty, single element, full element, out-of-boundary
search for element:
- element is inside array or the element is not inside
array
You can think about other data structures:
–list, set, stack, queue, and tree

Reviews = non-execution based testing

Testing = execution based testing
Dijkstra’s law of testing
Program testing can be used to show the presence of bugs, but never their
absence.
Testing is hence the process of executing a program with the intent to produce
failures.

19
 Testing

Validation:
– Does a s/w product satisfy its requirement?
alternatively
– Are we building the right-product?
Verification:
– Does a s/w product satisfy its specification?
alternatively
– Are we building the product right?

Levels of testing

Systematic testing
Ad hoc approach to testing is:
– ineffective: too late to influence design decisions
– inefficient: testing cannot be reused during maintenance (e.g.)
– hit and miss
Systematic testing:
– systematic: consists of a test plan and test strategy
– planned: actually design for testability
– documented: for repeatability and understandability
– maintained: repeat tests after every change
Test-case selection
Black-box or functional selection
– Test cases are based on the requirements or specification.
– Note: may miss important aspects of the implementation.
White-box or structural selection
– Test cases are based on the implementation.
– Note: may miss important aspects of the requirements (won’t test missing
functionality).

20
 Testing

Functional testing (black-box)

Choose test values for the state and parameters
– correct input
– input to induce exceptions
Test all combinations (combinatorial explosion)
Example of functional testing

Function f(p,q)
p: [0..100] and q: {red, green, blue}
calculate test values for p (interval rule)
– correct values: {0, 50, 100}
– exception values: {-100,-1,101,1000}
values for q
– correct values: {red, green, blue}
– exception values: none
number of combinations is 7 * 3 = 21

Structural testing (white-box)

 Execute as many statements as possible
 Coverage can be:
 statement: each statement executed at least once
 branch: each decision at least once
 path: each path in flowchart at least once
Example of structural testing
 statement coverage: {2}
 branch coverage: {-1, 2}
 Consider the code:
{ if (x>0) pos=pos+1;
if (x%2) even=even+1;
return;
}
 path coverage: {-2, -1, 1, 2}
 The function `triangle’ takes 3 integer parameters that are interpreted as the lengths
of the sides of a triangle. It returns whether the triangle is equilateral, isosceles,
scalene or invalid.
Define a set of test cases for `triangle’
 typedef enum {EQ, IS, SC, IN} type;

 type triangle (a, b, c)

 int a, b, c;
 {
 if (a>=b+c || b>=a+c || c>=a+b) return(IN);
 if (a==b && b==c) return(EQ);
 if (a==b || b==c || a==c) return(IS);

21
 Testing

 return (SC);
 }
Test cases for `triangle’
 1) valid scalene
 2) valid equilateral
 3) valid isosceles
 4) 3 isosceles: a==b, b==c, a==c
 5) a=0
 6) a=b=c=0
 7) a=-1
 8) 3 invalid: a=b+c, b=a+c, c=a+b
 9) 3 invalid: a>b+c, b>a+c, c>a+b
Assume the compiler checks for correct types
Myer scoring: 1 point for each test case, and 1 point if you have specified the outcome of
each test case. Maximum score is 10.
In 1978, when the book was published, experienced professional programmers scored
on average about 6.

 Use functional testing to select test cases.

 Use structural testing to measure degree of coverage.

Module, integration, system testing

 Module testing:test module in isolation using stubs and drivers
 Integration testing: test combinations of modules
 System: test the entire system
Module testing

22
 Testing

 Poor testability, like poor performance, is a design weakness.

 Good testability means:
controllability
 ease by which arbitrary input can be applied
 able to initialize a module
 have test cases that will generate all test cases
observability
 ease by which output can be observed
 must have proper error handling (errors signalled to an error handler)
Integration testing
 Incremental composition of module testing
 Two purposes:
 test subsystems as units
 test interfaces between units
 Two basic strategies:
 top-down
 bottom-up
The system

Test scaffolding

23
 Testing

Top-down testing
 Use stubs to test top-level modules
 Stepwise replace stubs by lower-level modules
 depth-first or breadth first
Bottom-up testing
 Use drivers to test bottom-level modules
 Stepwise replace drivers by higher-level modules
Top-down vs bottom-up testing
 Top-down testing
+ early availability of executable
+ early detection of flaws in interfaces
- poor controllability and observability
- cost of developing and maintaining stubs
 Bottom-up testing
+ better controllability and observability
- executable available very late
- cost of developing and maintaining drivers
System testing
 Test the entire system in production environment
 Various forms:
 volume/stress testing (load limits)
 performance testing (efficiency)
 reliability testing (e.g. mean-to-failure)
 recovery testing (e.g. h./w failure)
 acceptance testing (by the client)
 regression testing (maintenance)
Regression Testing
As a consequence of the introduction of new bugs, program maintenance requires
far more system testing per statement than any other programming. Theoretically,

24
 Testing

after each fix, one must run the entire bank of test cases previously run against the
system, to ensure that it has not been damaged in an obscure way ... and this is very
costly.
Three dimensions of quality
 operational
 correctness (functional testing)
 reliability (reliability testing)
 efficiency (performance testing)
 integrity (recovery testing)
 usability (acceptance testing)
 transitional
 portability, reusability, interoperability
 revisional
 maintainability, flexibility, testability

Test documents
 Test plan:
 the specification of the test system
 test selection strategy
 e.g. IEEE Standard 829 for S/W Test Document
 Test report
 report of the results of the test implementation
 Planning
 build the test `scaffolding’, which consists of:
drivers: modules that call code under test
stubs: modules called by code under test
 generate the test cases
 determine the expected output for each test case
 write the test plan
 Execution
 execute the test cases and generate a report that includes the actual outputs
 Evaluation
 compare the actual and expected outputs
Test plan
 an informal specification of the tests
 consists of:
 assumptions
assumptions on which the testing depends
 environment
environment in which testing is performed
 test-case selection strategy
choice of test cases
 implementation strategy
key aspects of the implementation

25
 Testing

Test-case selection strategy

 Functional testing
tests are based on the specification
 Structural testing
test are based on the implementation
 Both
Test implementation strategy
 Implement the test plan (test cases)
need drivers and stubs
need test-case data files
need a procedure for automating the tests
Test drivers
 Calls code under test (CUT)
 initialises the CUT
 provides the test cases
 captures the results (outputs and exceptions)
 may conpare actual and expected results
 Benefits
 isolates the CUT
 allows test cases to be re-run
Test stubs
 Called by the CUT
 simulates and replaces modules
 may accept or return data
 Benefits
 isolates the CUT
 allows test cases to be re-run
Kinds of test drivers
 Interactive drivers
 prompts the user for name of routine and parameters
 prints return values and exceptions
 good for debugging
 tedious when there are many cases
 tedious when cases are repeated often
 batch drivers
 test script language
 automates execution and checking of results
 e.g. Roast (SVRC, Hoffman and Strooper)
Stack module
name inputs outputs exceptions
s_init
s_push integer empty

26
 Testing

s_pop empty
g_top integer empty
g_depth integer

 set and get routines

 s_init and s_depth are for test purposes
Stack test plan for roast
assumptions
MAXSIZ > 2
test environment
roast driver
no stubs
test implementation strategy
load(n) - loads stack with 10, 20, . . . 10*n
test case selection strategy
special values
module state
interval rule on stack size: [0,MAXSIZ]
access routine parameters
none
test cases
for each of the special module state values
call s_push, s_pop, g_top, g_depth
check exception behavior
Roast test case
 <trace, expexc, actval, expval, type>
 trace: a sequence of calls
 expexc: the exception trace is expected to generate
 actval: an expression evaluated after trace
 expval: the value that actval is expected to have
 type: the data type of actval and expval
 e.g.
 <s_init().s_push(10), noexc,g_top(0, 10, int>

27
 Testing

Formal Proof vs. Testing

An argument that testing has limits in certain aspects to be complete to replace proof

Proof and Testing both serve their own purposes

There are some areas where testing is infeasible
“The problem with testing is not that they cannot show the absence of bugs, but that, in
practice, it fails to show their presence”
Proof appears to be substantially more efficient at finding faults than the testing phase
–Relatively litter effort
–Early in development phase
Proof can find bugs different from those found with testing techniques

28
 Testing

Limits of Testing
Concurrent / Distributed / Mobile systems
–Concurrency
–Non-determinism
–Mobility
Non-functional properties
–Performance
–Reliability, Fault Tolerance
(Statistical metrics)
Non-testable condition
–Environment
Testing based solely on analysis of program implementation is not sufficient
Few formal methods are available explicitly for testing
Formal Proof

Good mathematical models for the behavior of sequential programs

Models for concurrent / distributed systems / mobile computing are also available
Now in addition to required functionality (mapping from input to output), we can also
model non-functional properties like fault-tolerance and timing
Limits of Proof

Not scalable to large programs

Models cover only some aspects of a program’s behavior
–Models not available
–Correspondence between formal models and software behavior not well
understood
We can make mistakes in the proofs!
Use of Formal Proof
Inappropriate use:
–Complete, formal verifications of large algorithms
Appropriate use:
–Verify small but crucial segment of an algorithm – achieve cost-effectiveness by
developing partial proof for the “weakest links”
–Maintain acceptable levels of rigor without complete formalization
Relationships between proof and testing?
Formal proof And Testing are complementary approaches To software verification and
validation

Exploratory Testing

29
 Testing

In scripted testing, tests are first designed and recorded. Then they may be executed at
some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they often
are not recorded.

Exploratory testing is useful when…

 it is not obvious what the next test should be.
OR
 we want to go beyond the obvious tests.
What is ET?
concurrent, interacting test tasks
• “Burst of Testing” could mean any of these:
 Study the product.
 Model the test space.
 Select what to cover.
 Determine test oracles.

30
 Testing

 Configure the test system.

 Operate the test system.
 Observe the test system.
 Evaluate the test results.
 Notice issues.
 Organize notes.
the core practiceof a skilled tester
• Excellent exploratory testers…
– challenge constraints and negotiate their mission.
– alert their clients to project issues that prevent good testing.
– spontaneously coordinate and collaborate, as needed.
– train their minds to be cautious, curious, and critical.
– know how to design questions and experiments.
– know the difference between observation and inference.
– have developed resources and tools to enhance performance.
– adapt their test strategy to fit the situation.
– tolerate substantial ambiguity, indecision, and time pressure.
– take notes and report results in a useful and compelling way.
– have earned the trust placed in them.
Getting the Most Out of ET

Augment ET with scripted tests and automation.

–Usually it’s best to use a diversified strategy.
Exploit inconsistency.
–Let yourself be distracted by anomalies and new ideas.
–Avoid repeating the exact same test twice.
Exploit the human factor.
–Encourage variability among testers.
–Exploit subject matter expertise.
–Use your confusion as a resource.
–Work in pairs or groups, whenever possible.
–Get to know your developers.
–Test in short bursts.
Learn the logic of testing
–conjecture and refutation
–abductive inference
–correlation and causality
–design of experiments
–forward, backward, and lateral thinking
–biases, heuristics, and human error
–risk, benefit, and the meaning of “good enough”
Practice critical reading and interviewing
–Analyzing natural language specifications
–Analyzing and cross-examining a developer’s explanation.
Exploratory questions

31
 Testing

How well does your car work?

How do you know how well your car works?
What evidence do you have about how your car works?
Is that evidence reliable and up to date?
What does it mean for your car to “work”?
What facts would cause you to believe that your car doesn’t work?
In what ways could your car not work, yet seemto you that it does?
In what ways could your car work, yet seem to you that it doesn’t?
What might cause your car not to work well (or at all)?
What would cause you to suspect that your car will soon stop working?
Do other drivers operate your car? How does it work for them?
How important is it for your car to work?
Are you qualified to answer these questions? Is anyone else qualified?
Adopt a clear and consistent testing vocabulary
–bug, specification, risk, test strategy, test case, test plan
Learn to model a product rapidly
–flowcharting; data flows; state model
–matrices and outlines
–function/data square
–study the technology
Use a “grid search” strategy to control coverage
–Model the product in some way, then specify broad test areas in terms of that
model (not specific test cases). Keep track of what areas you have and have not
tested in.
Learn to take reviewable notes
–Take concise notes so that they don’t interrupt your work.
–Record at least what your strategy was, what you tested, what problems you
found, and what issues you have.
Practice responding to scrutiny
–Why did you test that?
–What was your strategy?
–How do you know your strategy was worthwhile?
Learn to spot obstacles to good testing
–Not enough information, yet.
–The product isn’t testable enough.
– Don’t have enough of the right test data.
Develop and use testing heuristics
–Try the heuristic test strategy model, on my web site.
–Practice critiquing test techniques and rules of thumb.
–Notice the test ideas you use that lead to interesting results.
If
you’re in a highly structured environment, consider using session-based test
management to measure and control exploratory testing.
–Packages ET into chunks that can be tracked and measured.
–Protects the intuitive process; gives bosses what they want.

Verification Validation Dynamic and Testing Techniques

32
 Testing

Black-box testing/Functional testing

 What is black-box testing
 Generation of test data
 Examples of generation of test data
 Examples of application of black-box testing
Field testing
 What is field testing
 Examples of application of field testing
Fault/Failure insertion testing
 What is fault/failure insertion testing
 Examples of fault/failure insertion testing
Reference

Black-Box Testing
• Synonyms for black-box testing include: behavioral, functional, opaque-box, and
closed-box testing.
• Black-box testing treats the system as “black-box”, so it doesn’t explicitly use
knowledge of the internal structure.
• Black-box testing is usually described as focusing on testing functional
requirements.
Black-box testing is used to access the accuracy of model input-output transformation.

Input Output

Black-box testing is applied by feeding test data to model and evaluating the
corresponding outputs. The concern is how accurately the model transform a given set
of input data into a set of output data. If we can test all input-output transformation paths,
then we can get 100% confidence.
Black-box testing is based on the view that any model can be considered to be a
function that maps values from its input domain to values in its output range. The content
( implementation ) of a black-box is not know, and the function of the black-box is
understood completely in terms of its inputs and outputs. Many times, we operate very
effectively with black-box knowledge; in fact this is central to object orientation. As an
example, most people successfully operate automobiles with only black-box knowledge.

Generation of test data is a crucially important but a very difficult task.

The more the system input domains is covered in testing, the more confidence we gain
in the accuracy of the system input-output transformation.
Examples of generation of test data
 Exhaustive Testing
 Random Testing
 Systematic Way

33
 Testing

Exhaustive Testing

Void printBoolean (bool error)

//Print the Boolean Value on the screen
{ if (error)
cout<<“True”;
else
cout<<“False”;
cout<<end1:
}
Unfortunately, we’ll never want such a simple printer. For a reasonably large and
complex simulation system, the number of input-output transformation paths could be
very large. So it is virtually impossible to test all input-output transformation paths.
Therefore, the object of functional testing is to increase our confidence in model input-
output transformation accuracy as much as possible rather than trying to claim absolute
correctness.
Random Testing
 Attempt testing in a haphazard way, entering data randomly until we cause the
system to fail.
 Test data may be chosen randomly or by a sampling procedure reflecting “ the
actual probability distribution on the input sequences.’’
 It is likely to uncover some errors in the system, but it is very unlikely to find
them all.
Random testing is essentially a black-box testing strategy in which a system is tested by
randomly selecting some subsets of all possible input values. This allows one to
estimate the “ operational reliability”.
Void printBoolean (int intValue)
//Print the integer Value on the screen
{ if (inValue>10)
cout<<inValue”
else
cout<<end1”;
}

Integer 9 Nothing on the screen

The simple model is designed to print every input integer on the screen.

34
 Testing

It is not practical to test this model by running it with every possible data input; the
number of elements in the set of intValue is clearly too large. In such cases, we don’t
attempt exhaustive testing.
To test this simple printer model, we may try over hundreds of time until finally we feed
an integer 9 to it, and find there is a error in the model, for it can not print the input
integer less than 10 on the screen.
Fortunately, however, there are strategies for testing in a systematic way. One goal-
oriented approach is to cover general classed of data. We should test at least one
example of each category of inputs, as well as boundaries and other special cases.
A Systematic Way of Testing
 There are strategies for testing in a systematic way. One goal-oriented approach
is to cover general classed of data.
 We should test at least one example of each category of inputs, as well as
boundaries and other special cases.

Negativ
e Zero Positive Values
Values

Three cases

Field Testing
 Field testing known as live environment testing places the system in an
operational (real) environment (i.e. using real data as the input source).
 The purpose is collecting as much information as possible.
 It is necessary to demonstrate system capabilities for acceptance.

Although it is usually difficult expensive and sometimes impossible to devise

meaningful field tests for complex systems, their use however possible helps both
the project team and decision makers to develop confidence in the model.
Since the advent of high level language, the practice of developing software in a
different environment to the environment in which it will eventually be used has
become common.

 Host ------ The development environment

 Target ------ The environment in which the system will be used
 Which system should be tested in the target environment?
Theoretically, all testing should be conducted in the target environment. After all, it is
the target environment in which the system will eventually be used. However,
constraining all testing to the target environment can result can result in a number of
problems.
 The target environments may not yet be available.
 Target environments are usually less convenient to work with than host
environments.

35
 Testing

 Target environments and associated development tools are usually much more
expensive to provide to system developer than host environments.
It is rare for a host environment to be identical to a target environment. There may
just minor differences in configuration, or there may be major differences such as
between a workstation and an embedded control processor, where the two
environments may even have a different instruction set.
Test Implementation
 Direct access to target hardware, such as input-output devices.
 Calls to target environment system, such as an operating system or real-time
kernel.
Acceptance testing has to be conducted in the target environment!
The final stage of system testing, acceptance testing, has to be conducted in the target
environment. Acceptance of a system must be based on tests of the actual system, and
not a simulation in the host environment.

Examples of application of field testing

 It is especially useful for validating models of military combat systems.

 Testing the expert system with actual data is a method for the validation of expert
systems. Put the system in real life situations and see the direct effectiveness of
the system in those situation.
In the development of an expert system, there is typically substantial asymmetry of
expertise: the expert knows more about the domain than the developers of the system.

Fault/Failure Insertion Testing

 Fault ------- Incorrect system component
 Failure------ Incorrect behavior of a system component
 This technique is used to insert a kind of fault or a kind of failure into the system
and observe whether the system produces the invalid behavior as expected.
 Unexplained behavior may reveal errors in system representation.
Fault/failure insertion testing is an error-based testing technique. Error based testing is
presented in nearly all heuristic approaches to testing. For example, informal debugging
sessions frequently include checks on extreme values of variables. In addition, the folk
of many applications areas consist of heuristic rules(e.g. in testing compliers, one of the
first test cases tried is usually “null” program.
In error-based testing, the goal is to construct test cases that reveal the presence or
absence of specific errors.
 Test data adequacy
Test data set is adequate if the model run successfully on the data set and if all
incorrect models
run incorrectly.
 Mutants
The mutants are a set of models which are “close” to the model being tested.
“Close” refers the potential errors which could have occurred in the model
being tested.
Basic Methodology

36
 Testing

 D ------ Test data used to test the model

 M (model) ------ A set of mutants of the model, differ from the model in containing
a single error chosen form a given list of error types
 m ------ Number of elements in M (model)
 E (model) ------ Set of equivalent mutants of the model
 e ------ Number of elements in E (model)
 DM (model, D) ------ Set of the mutants that will return results differ from the
results which
 dm ------ Number of elements in DM (model, D)
 ms (model, D) = dm / (m-e)
 ms (model, D) ------ Mutation score, defined to the fraction of the nonequivalent
mutants of the model, which are distinguished by the test data D.

Some of the mutant models will turn out to be functionally equivalent to the model.
That is, they will be indistinguishable form the model under the test data.
A mutation score is a number in the interval [0,1]. A high score indicate that D is very
close to being adequate for the model relative to the set of mutants of the model. A
low score indicates a weakness in the test data. The test data does not distinguish
the test model form the mutant model which contain an error.

 If all mutants of the being tested model give incorrect results on execution by
the test data D, we say they die on the execution. If all mutants die, it is highly
likely that the tested model is very likely to be correct.
 If some mutants are live and the test data D is adequate, then either the live
mutants are functionally equivalent to the model or there still might be complex
errors in the model.

Examples of fault/failure insertion testing

 Budd and Miller have studied fault/failure insertion testing as tool to uncover
typographical errors in Matrix Calculation Programs. It can be used to uncover
simple statement errors, dead code errors, domain errors, dead branch errors, data
flow errors, special values errors, and coincidental correctness errors.
 It is used by Lipton to uncover resistant error in production programs.
 Its application to regression testing is carried out by Demillo.

37
 Testing

The Top 10 Testing Problems

10. Not enough training
9. “Us vs. Them” mentality
8. Lack of test tools
7. Lack of management understanding/support of testing
6. Lack of customer and user involvement
5. Not enough time for testing
4. Over-reliance on independent testers
3. Rapid change
2. Testers are in a “lose/lose” situation
1. Having to say “no”
Solutions for Training
Obtain formal training in testing techniques.
Seek Certification.
–CSTE (Certified Software Test Engineer)
Attend conferences.
Read books and articles.
Solutions to the Teamwork Challenge
The goal is to get to “Us and them.”
Each person on the team can have a role in testing:
–Developers: unit and structural testing
–Testers: independent testing
–Users: business-oriented testing
–Management: to support testing activities
Solutions for Acquiring and Using Test Tools
Identify a “champion” for obtaining test tools.
Base the case for test tools in costs vs. benefits.
Have a basic testing process in place.
Train people in tool usage.
Measure the benefits.
Solutions to Educating Management in Testing Issues
Cultural change is needed.

38
 Testing

Focus your message to management on:

–reducing the cost of rework
–meeting the project schedule
The benefits of testing must relate to these two things to be persuasive.
Solutions to Identifying and Involving the Customer in Testing
Involve the customer and users throughout the project by performing reviews and
inspections.
Include users on the system test team.
Perform user acceptance testing.
Understand the difference between the customer and users. 
Solutions to the Time Crunch
Base schedules and estimates on measurable testing activities.
–Scripts to be executed
–Cases to be tested
–Requirements to be tested
Have contingency plans for schedule slippage.
Integrate automated testing tools to the project.
Solutions to Overcoming Throwing Stuff Over the Wall
Developers must take ownership and responsibility for the quality of their work.
Quality control is most effective when performed at the point of creation.
Train developers to become excellent testers.
Get management support for developer responsibility for quality.
Solutions for Hitting a Moving Target
The testing process must accommodate change.
Focus on testable requirements.
Use automated testing tools.
Manage the rate and degree of change.
Solutions for Fighting a Lose-Lose Situation
The perception of testing must change.
–Testers are paid to find defects.
–Each defect found is one more the customer or user will not find.
Testers are not to blame for bottlenecks. It is management’s responsibility to have an
efficient process.
Solutions for Having to Say “No”

39
 Testing

Most responsibility is on management to:

–have a quality software development process in place.
–have contingency plan in place in case of problems.
–understand that testing is only an evaluation activity.
–accept the honest facts.
Keep the test results objective.

QAI Workbench Model

z

Test Terminology (Cont’d.)

Verification
–All QC activities throughout the life cycle that ensure interim deliverables meet
specific specifications.
Validation
–The “test phase” of the life cycle which ensures that the end product (e.g...,
software or system) meets specifications or user needs.

When Testing Occurs

40
 Testing

Test Terminology (Cont’d.)

When Testing Occurs
Business Acceptance
Need Veri fy Test
Business Validate
Need Business Need

Define System
Requirements Test
Veri fy Validate
Requirements Requirements

Design Integration
System Veri fy Test Validate
Verificat ion System System Validat ion

Code Unit
System Veri fy Test Validate
Code Code

Unit Testing
–Testing performed on a single, standalone module or unit of code.
Integration Testing
–Testing performed on groups of related modules to ensure data and control are
passed properly between modules.
System Testing
–A predetermined combination of tests that, when executed successfully, satisfy
management that the system meets specifications
–Validates that the system was built right.
User Acceptance Testing
–Testing to ensure that the system meets the need of the organization and the end
user/customer
–Validates that the right system was built.
Regression Testing
–Testing after changes have been made to ensure that no unwanted changes were
introduced to the software or system.

Functional Tests
–Tests that validate business requirements
–Tests what the system is supposed to do
Black Box Tests
–Functional testing
–Based on external specifications without knowledge of how the system is
constructed

41
 Testing

–Usually process and/or data driven

Structural Tests
–Tests that validate the system architecture
–Tests how the system was implemented
White Box or Glass Box Tests
–Structural testing
–Testing based on knowledge of internal structure and logic
–Usually logic driven
If x=curr-date then
set next-val to 03
else
set next-val to 05.
To effectively test systems, both functional and structural testing need to be performed.
The Economics of Testing - Making the Message to Management

Where Defects Originate

Where Testing Resources are Used

42
 Testing

The Relative Cost of Fixing Defects

The Bottom Line

Most defects are created in the early stages of a project
Most defects are found in the later stages of a project
It costs 10 to 100 times as much to fix a defect in the later phases of a project.
If you want to reduce the cost of testing, spend time early in the system development
(or purchase) process to make sure the requirements and design are correct.

Basic Testing Principles

Test early and often
Involve everyone on the project
Management support is critical
The greater the risk, the more intense the test should be
The higher the test coverage, the more confidence you’ll have in the test

43
 Testing

Test Strategy

Planning Step 1 - Determine Test

Strategy
What Type of
How Crit ical is the Projects? What Type of
system to the Software?
Organization?

What are the Determine What Type of

Tradeoffs? Testing Technical
Strategy Environment?

Who Will Conduct What is the

Testing? Project’s Scope?

What are the

Critical Success When Will Testing
Factors? Occur?

What Type of Project?

–Traditional
–Prototyping/CASE
–Maintenance
What Type of Software?
–On-line
–Real Time
–Batch
What Type of Technical Environment?
–Mainframe
–Client/Server
What is the Project’s Scope?
–New Development
–System Maintenance
When Will Testing Occur?
–Requirements
–Design
–Testing
What Are the Critical Success Factors?
–Correctness

44
 Testing

–Reliability
Who Will Conduct Testing?
–Users
–Developers
What Are the Tradeoffs?
–Schedule
–Cost/Resources
–Quality
How Critical is the System to the Organization?
–Risk Assessment
Risk Assessment

A Tool For Performing Risk Assessment

45
 Testing

Effective Testing Methods and Techniques

The QAI Testing Process

Step 1 - Set Test Objectives

Step 2 - Develop Test Plan
Step 3 - Execute Tests
Step 4 - Evaluate/Report Test Results
Step 1 - Set Test Objectives
Select test team
Perform risk assessment
Define test objectives
–A test objective is what the test is to validate.
–There should be a one-to-one correspondence between system objectives and test
objectives.

Step 2 - Develop Test Plan

The better the test plan, the easier the test.
Test plans should be specific, yet flexible for change.
Test planning should be a team activity.
Test plans should be reviewed just as any other project deliverable.
The test plan should be easily read by management.
Major Elements of a Test Plan
Introduction

46
 Testing

Approach (Strategy)
Test Objectives
Description of the system or software to be tested
Test environment
Description of the test team
Milestones/Schedule
Functions and attributes to be tested
Evaluation criteria
Data recording
References
Tests to be performed
How Much Time Should be Spent on Test Planning?

Many organizations report spending onethird to one-half of a project’s time in

testrelated activities.
Planning Time Guidelines

Of the total test time, roughly one-third of the time can be allocated each to:
–Test planning
–Test execution
–Test evaluation
Tips for Test Planning

Start early.
Keep the test plan flexible to deal with change.
Frequently have the test team review the test plan.
Keep the test plan concise and readable.
Step 3 - Execute Tests
Select test tools
Develop test cases
Execute tests
Select Test Tools
A test tool is any vehicle that assists in testing.
May be manual or automated
Automated Tools

47
 Testing

Not the complete solution, but an important part

Requires:
–a process
–an understanding of testing in general
•Knowing what to test
•Defining test cases
•Knowing how to evaluate test results
–cultural acceptance
More than just capture/playback
Categories of Automated Tools

Capture/playback or script execution

Defect trackers
Test management
Test case generators
Coverage analyzers
Path and complexity analyzers
Manual Testing

Automated Testing

48
 Testing

Critical Success Factors

Get senior management support for buying and integrating test tools
Know your requirements
Be reasonable in your expectations - Start small and grow
Have a strong testing process that includes tools
Don’t cut the training corner
Regression Testing
Why perform regression testing?
The process
The issues
The role of automated testing tools
How much is enough?
No Regression Testing: Hidden Defects

49
 Testing

Regression Testing: No Hidden Defects

Regression Testing - The Process

Regression Testing Issues

Test data must be maintained.
There must be a way to conduct two identical tests.

50
 Testing

There must be a way to compare two identical tests.

Some tests cannot use previous versions of test data.
–Data conversion may be required
–Date dependencies
The greater the difference between versions, the less effective the regression test.
There must be a stable baseline version for comparisons.
Regression Testing - How Much is Enough?
The easy answer: “It depends.”
What does it depend on?
–Risk
–Scope of the change
–System dependencies
Proving the Value of Regression Testing
You need a benchmark of non-regression testing.
Consider manual vs. automated.
Consider initial investment in creating test environment and test scripts/procedures
Measure time and defects.
Return on Investment (ROI) can include:
–Shorter test times
–More accurate testing
–More consistent testing
–Improved communication of defects
–More effective testing (e.g. Fewer test cases needed to find more defects.)
Tips for Performing Regression Testing
Control the scope of testing.
Build a reusable test bed of data.
Use automated tools.
Base the amount of regression testing on risk.
Build a repeatable and defined process for regression testing.
Step 3 - Execute Tests
Develop Test Cases
Functional Techniques
–Requirements-based
–Process-based

51
 Testing

–Data-oriented
–Boundary value analysis
–Decision tables
–Equivalence partitioning
Structural Techniques
–Complexity analysis
–Coverage
•Statement
•Branch
•Condition
•Multi-condition
•Path
Step 4 - Evaluate/Report Test Results
Occurs throughout the testing life cycle
Tracks testing progress
Keeps management informed of testing progress
Valuable Test Metrics
Two key areas to measure
–Time
•For future estimating
–Defects
•For determining effectiveness of testing
•For improving the development and testing processes
Time
–Time per test case
–Time per test script
–Time per unit test
–Time per system test
Sizing
–Function points
–Lines of code
Defects
–Numbers of defects
–Defects per sizing measure
–Defects per phase of testing
–Defect origin
–Defect removal efficiency

52
 Testing

Defect Removal Efficiency = Number of defects found in producer testing

Number of defects during the life of the product

The Critical Path of Testing

What Must be in Place for Effective Testing?
Management support
A defined and repeatable process for testing
Adequate tools
Trained testers
Cooperation between testers, developers and end users
Maximum coverage with minimal test cases
The Five Levels of Software Process Maturity

Best Practices for Control/Test Management Processes

53
 Testing

Best Practices for Control/Test

Management Processes
••Risk
Riskanalysis
analysis Level 5 –
••Project
Projectcustomization
customization Preventive
••Defect
Defectprofiles
profiles
Management
••Dashboards
Dashboards
••Root
Rootcause
causeanalysis
analysis Level 4 - Statistical
••Statistical
Statisticalanalysis
analysis Process Control
••Defect
Defectdatabase
database
••Defect
Defectreporting
reporting
••Defect
Defectanalysis
analysis Level 3 - Defect Management

••Code
Codeanalyzers
analyzers
••Walkthroughs
Walkthroughs
••Inspections
Inspections Level 2 - Verification
••Acceptance
Acceptancetest
test

••Unit
Unittest
test
••Integration
Integrationtest
test
••System
Systemtest
test
Level 1 - Validation

Software Testing

54
 Testing

Table of Contents

Introduction to Software Testing

Basic Methods
Testing Levels
Unit testing
Integration testing
External Function Testing
System Testing
Regression Testing
Acceptance Testing
Installation Testing
Completion Criteria
Metrics
Organization
Testing and SQA, Inspections
A Closer Look: Fault Based Methods
Conclusion

Introduction to Software Testing

Software testing is an vital part of the software lifecycle. To understand its role, it is
instructive to review the definition of software testing in the literature.
Among alternative definitions of testing are the following:
"... the process of exercising or evaluating a system or system component by manual or
automated means to verify that it satisfies specified requirements or to identify
differences between expected and actual results ..."
"... any activity aimed at evaluating an attribute or capability of a program or system and
determining that it meets its required results.
Testing is the measurement of software quality ..."
"... the process of executing a program with the intent of finding errors..."
Of course, none of these definitions claims that testing shows that software is free from
defects. Testing can show the presence, but not the absence of problems.
According to Humphrey [1], software testing is defined as 'the execution of a program to
find its faults'. Thus, a successful test is one that finds a defect. This sounds simple
enough, but there is much to consider when we want to do software testing. Besides
finding faults, we may also be interested in testing performance, safety, fault-tolerance or
security.

Testing often becomes a question of economics. For projects of a large size, more
testing will usually reveal more bugs. The question then becomes when to stop testing,
and what is an acceptable level of bugs. This is the question of 'good enough software'.
It is important to remember that testing assumes that requirements are already
validated.

55
 Testing

Basic Methods
White Box Testing
White box testing is performed to reveal problems with the internal structure of a
program. This requires the tester to have detailed knowledge of the internal structure. A
common goal of white-box testing is to ensure a test case exercises every path through
a program. A fundamental strength that all white box testing strategies share is that
the entire software implementation is taken into account during testing, which facilitates
error detection even when the software specification is vague or incomplete. The
effectiveness or thoroughness of white-box testing is commonly expressed in terms of
test or code coverage metrics, which measure the fraction of code exercised by test
cases.
Black Box Testing
Black box tests are performed to assess how well a program meets its requirements,
looking for missing or incorrect functionality. Functional tests typically exercise code with
valid or nearly valid input for which the expected output is known. This includes
concepts such as 'boundary values'.
Performance tests evaluate response time, memory usage, throughput, device
utilization, and execution time. Stress tests push the system to or beyond its specified
limits to evaluate its robustness and error handling capabilities. Reliability tests monitor
system response to representative user input, counting failures over time to measure or
certify reliability.
Testing Levels
Different Levels of TestTesting occurs at every stage of system construction. The larger a
piece of code is when defects are detected, the harder and more expensive it is to find
and correct the defects.
The different levels of testing reflect that testing, in the general sense, is not a single
phase of the software lifecycle. It is a set of activities performed throughout the entire
software lifecycle.
In considering testing, most people think of the activities described in figure. The
activities after Implementation are normally the only ones associated with testing.
Software testing must be considered before implementation, as is suggested by the
input arrows into the testing activities.

56
 Testing

The following paragraphs describe the testing activities from the 'second half' of the
software lifecycle.
Unit Testing
Unit testing exercises a unit in isolation from the rest of the system. A unit is typically a
function or small collection of functions (libraries, classes), implemented by a single
developer. The main characteristic that distinguishes a unit is that it is small enough to
test thoroughly, if not exhaustively. Developers are normally responsible for the testing of
their own units and these are normally white box tests. The small size of units allows a
high level of code coverage. It is also easier to locate and remove bugs at this level of
testing.
Integration Testing
One of the most difficult aspects of software development is the integration and testing
of large, untested sub-systems. The integrated system frequently fails in significant and
mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module,
subsystem, or system. Integration testing focuses on the interfaces between units, to
make sure the units work together. The nature of this phase is certainly 'white box', as
we must have a certain knowledge of the units to recognize if we have been successful
in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big
bang'. Top-down combines, tests, and debugs top-level routines that become the test
'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level
units into progressively larger modules and subsystems. 'Big bang' testing is,
unfortunately, the prevalent integration test 'method'. This is waiting for all the module
units to be complete before trying them out together.
Bottom-up
Allows early testing aimed t proving feasibility and practicality of particular
modules.
Modules can be integrated in various clusters as desired.
Major emphasis is on module functionality and performance.
Advantages
No test stubs are needed
It is easier to adjust manpower needs
Errors in critical modules are found early
Disadvantages
Test drivers are needed
Many modules must be integrated before a working program is available
Interface errors are discovered late
Comments
At any given point, more code has been written and tested that with top down
testing. Some people feel that bottom-up is a more intuitive test philosophy.
Top-Down
The control program is tested first
Modules are integrated one at a time
Major emphasis is on interface testing

57
 Testing

Advantages
No test drivers are needed
The control program plus a few modules forms a basic early prototype
Interface errors are discovered early
Modular features aid debugging
Disadvantages
Test stubs are needed
The extended early phases dictate a slow manpower buildup
Errors in critical modules at low levels are found late
Comments
An early working program raises morale and helps convince management
progress is being made. It is hard to maintain a pure top-down strategy in practice.

Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished
subroutines or sub-systems. A stub might consist of a function header with no body, or it
may read and return test data from a file, return hard-coded values, or obtain data from
the tester. Stub creation can be a time consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what
drives the use of 'big bang' testing. This approach waits for all the modules to be
constructed and tested independently, and when they are finished, they are integrated all
at once. While this approach is very quick, it frequently reveals more defects than the
other methods. These errors have to be fixed and as we have seen, errors that are
found 'later' take longer to fix. In addition, like bottom up, there is really nothing that can
be demonstrated until later in the process.
External Function Testing
The 'external function test' is a black box test to verify the system correctly implements
specified functions. This phase is sometimes known as an alpha test. Testers will run
tests that they believe reflect the end use of the system.
System Testing
The 'system test' is a more robust version of the external test, and can be known as an
alpha test. The essential difference between 'system' and 'external function' testing is the
test platform. In system testing, the platform must be as close to production use in the
customers&#8217; environment, including factors such as hardware setup and database
size and complexity. By replicating the target environment, we can more accurately test
'softer' system features (performance, security and fault-tolerance).
Because of the similarities between the test suites in the external function and system
test phases, a project may leave one of them out. It may be too expensive to replicate
the user environment for the system test, or we may not have enough time to run both.
Acceptance Testing
An acceptance (or beta) test is an exercise of a completed system by a group of end
users to determine whether the system is ready for deployment. Here the system will
receive more realistic testing that in the 'system test' phase, as the users have a better
idea how the system will be used than the system testers.
Regression Testing
Regression testing is an expensive but necessary activity performed on modified
software to provide confidence that changes are correct and do not adversely affect

58
 Testing

other system components. Four things can happen when a developer attempts to fix a
bug. Three of these things are bad, and one is good:
New Bug No New Bug
Successful Change Bad Good
Unsuccessful Change Bad Bad

Because of the high probability that one of the bad outcomes will result from a change to
the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of
the development cycle. Most industrial testing is done via test suites; automated sets of
procedures designed to exercise all parts of a program and to show defects. While the
original suite could be used to test the modified software, this might be very time-
consuming. A regression test selection technique chooses, from an existing test set,
the tests that are deemed necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a
minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require
minimization of the test set. Instead, they seek to select all tests that exercise changed
or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce
different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing
to the "vicinity" of the change. (Ex. If I put a new radio in my car, do I have to do a
complete road test to make sure the change was successful?) A new breed of regression
test theory tries to identify, through program flows or reverse engineering, where
boundaries can be placed around modules and subsystems. These graphs can
determine which tests from the existing suite may exhibit changed behavior on the new
version.
Regression testing has been receiving more attention as corporations focus on fixing the
'Year 2000 Bug'. The goal of most Y2K is to correct the date handling portions of their
system without changing any other behavior. A new 'Y2K' version of the system is
compared against a baseline original system. With the obvious exception of date
formats, the performance of the two versions should be identical. This means not only do
they do the same things correctly, they also do the same things incorrectly. A non-Y2K
bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem
is fixed. Why do I need to re-test?&#8217; to which the answer is 'The same person
probably told you it worked in the first place'.
Installation Testing The testing of full, partial, or upgrade install/uninstall processes.
Completion Criteria
There are a number of different ways to determine the test phase of the software life
cycle is complete. Some common examples are:
 All black-box test cases are run
 White-box test coverage targets are met

59
 Testing

 Rate of fault discovery goes below a target value

 Target percentage of all faults in the system are found
 Measured reliability of the system achieves its target value (mean time to failure)
 Test phase time or resources are exhausted
When we begin to talk about completion criteria, we move naturally into a discussion of
software testing metrics.
Metrics
Goals As stated above, the major goal of testing is to discover errors in the software. A
secondary goal is to build confidence that the system will work without error when testing
does not reveal any errors. Then what does it mean when testing does not detect any
errors? We can say that either the software is high quality or the testing process is low
quality. We need metrics on our testing process if we are to tell which is the right
answer.
As with all domains of the software process, there are hosts of metrics that can be used
in testing. Rather than discuss the merits of specific measurements, it is more important
to know what they are trying to achieve.
Three themes prevail:
 Quality Assessment (What percentage of defects are captured by our testing
process, how many remain?)
 Risk Management (What is the risk related to remaining defects?)
 Test Process Improvement (How long does our testing process take?)
Quality Assessment An important question in the testing process is "when should we
stop?" The answer is when system reliability is acceptable or when the gain in reliability
cannot compensate for the testing cost. To answer either of these concerns we need a
measurement of the quality of the system.
The most commonly used means of measuring system quality is defect density. Defect
density is represented by:
# of Defects / System Size
where system size is usually expressed in thousands of lines of code or KLOC. Although
it is a useful indicator of quality when used consistently within an organization, there are
a number of well documented problems with this metric. The most popular relate to
inconsistent definitions of defects and system sizes.
Defect density accounts only for defects that are found in-house or over a given amount
of operational field use. Other metrics attempt to estimate of how many defects remain
undetected. A simplistic case of error estimation is based on "error seeding". We assume
the system has X errors. It is artificially seeded with S additional errors. After a testing,
we have discovered Tr 'real' errors and Ts seeded errors. If we assume (questionable
assumption) that the testers find the same percentage of seeded errors as real errors,
we can calculate X:
 S / (X + S) = Ts / (Tr + Ts)
 X = S * ((Tr + Ts) / Ts -1)
For example, if we find half the seeded errors, then the number of 'real' defects found
represents half of the total defects in the system.

60
 Testing

Estimating the number and severity of undetected defects allows informed decisions on
whether the quality is acceptable or additional testing is cost-effective. It is very
important to consider maintenance costs and redevelopment efforts when deciding on
value of additional testing.
Risk Management
Metrics involved in risk management measure how important a particular defect is (or
could be). These measurements allow us to prioritize our testing and repair cycles. A
truism is that there is never enough time or resources for complete testing, making
prioritization a necessity.
One approach is known as Risk Driven Testing, where Risk has specific meaning. The
failure of each component is rated by Impact and Likelihood. Impact is a severity rating,
based on what would happen if the component malfunctioned. Likelihood is an estimate
of how probable it is that the component would fail. Together, Impact and Likelihood
determine the Risk for the piece.
Obviously, the higher rating on each scale corresponds to the overall risk involved with
defects in the component. With a rating scale, this might be represented visually:
The relative importance of likelihood and impact will vary from project to project and
company to company.
A system level measurement for risk management is the Mean Time To Failure
(MTTF). Test data sampled from realistic beta testing is used find the average time until
system failure. This data is extrapolated to predict overall uptime and the expected time
the system will be operational. Sometimes measured with MTTF is Mean Time To Repair
(MTTR). This represents the expected time until the system will be repaired and back in
use after a failure is observed. Availability, obtained by calculating MTTF / (MTTF +
MTTR), is the probability that a system is available when needed. While these are
reasonable measures for assessing quality, they are more often used to assess the risk
(financial or otherwise) that a failure poses to a customer or in turn to the system
supplier.
Process Improvement
It is generally accepted that achieve improvement you need a measure against which to
gauge performance. To improve our testing processes we the ability to compare the
results from one process to another.
Popular measures of the testing process report:
Effectiveness: Number of defects found and successfully removed / Number of Defect
Presented
Efficiency: Number of defects found in a given time
It is also important to consider reported system failures in the field by the customer. If a
high percentage of customer reported defects were not revealed in-house, it is a
significant indicator that the testing process in incomplete.
A good defect reporting structure will allow defect types and origins to be identified. We
can use this information to improve the testing process by altering and adding test
activities to improve our changes of finding the defects that are currently escaping
detection. By tracking our test efficiency and effectiveness, we can evaluate the changes
made to the testing process.

61
 Testing

Testing metrics give us an idea how reliable our testing process has been at finding
defects, and can is a reasonable indicator if its performance in the future. It must be
remembered that measurement is not the goal, improvement through measurement,
analysis and feedback is what is needed.

Software Testing Organization

Test GroupsThe following summarizes the Pros and Cons of maintaining separate test
groups ...

Pros
 Testers are usually the only people to use a system heavily as experts;
 Independent testing is typically more efficient at detecting defects related to
special cases, interaction between modules, and system level usability and
performance problems
 Programmers are neither trained, nor motivated to test
 Overall, more of the defects in the product will likely be detected.
 Test groups can provide insight into the reliability of the software before it is
actually shipped
Cons
 Having separate test groups can result in duplication of effort (e.g., the test group
 expends resources executing tests developers have already run.
 The detection of the defects happens at a later stage, designers may have to
wait for responses from the test group to proceed. This problem can be
exacerbated in situations where the test group is not physically collocated with
the design group.
 The cost of maintaining separate test groups
The key to optimizing the use of separate test groups is understanding that developers
are able to find certain types of bugs very efficiently, and testers have greater abilities in
detecting other bugs. An important consideration would be the size of the organization,
and the criticality of the product.
Testing Problems
When trying to effectively implement software testing, there are several mistakes that
organizations typically make. The errors fall into (at least) 4 broad classes:
Misunderstanding the role of testing.
The purpose of testing is to discover defects in the product. Furthermore, it is important
to have an understanding of the relative criticality of defects when planning tests,
reporting status, and recommending actions.
Poor planning of the testing effort.
Test plans often over emphasize testing functionality at the expense of potential
interactions. This mentality also can lead to incomplete configuration testing and
inadequate load and stress testing. Neglecting to test documentation and/or installation
procedures is also a risky decision.

62
 Testing

Using the wrong personnel as testers.

The role of testing should not be relegated to junior programmers, nor should it be a
place to employ failed programmers. A test group should include domain experts, and
need not be limited to people who can program. A test team that lacks diversity will not
be as effective.
Poor testing methodology.
Just as programmers often prefer coding to design, testers can be too focussed on
running tests at the expense of designing them. The tests must verify that product does
what it is supposed to, while not doing what it should not. As well, using code coverage
as a performance goal for testers, or ignoring coverage entirely are poor strategies.

Testing and SQA, Inspections

Inspections are undoubtedly a critical tool to detect and prevent defects. Inspections are
strict and close examinations conducted on specifications, design, code, test, and other
artifacts. An important point about inspections is that they can be performed much earlier
in the design cycle, well before testing begins. Having said that, testing is something that
can be started much earlier than is normally the case. Testers can review their test plans
with developers as they are creating their designs. Thus the developer may be more
aware of the potential defects and act accordingly. In any case, the detection of defects
early is critical, the closer to the time of its creation that we detect and remove a defect,
the lower the cost, both in terms of time and money. This is illustrated in figure :

Figure : Defect Detection and cost to correct (Source: McConnell)

63
 Testing

Evidence of the benefits of inspections abounds. The literature (Humphrey 1989) reports
cases where:
 inspections are up to 20 times more efficient than testing;
 code reading detects twice as many defects/hour as testing;
 80% of development errors were found by inspections;
 inspections resulted in a 10x reduction in cost of finding errors;
In the face of all this evidence, it has been suggested that "software inspections can
replace testing". While the benefits of inspections are real, they are not enough to
replace testing. Inspections could replace testing if and only if all information gleaned
through testing could be obtained through inspection. This is not true for several
reasons.
Firstly, testing can identify defects due to complex interactions in large systems (e.g.
timing/synchronization). While inspections can detect this event, as systems become
more complex the chances of one person understanding all the interfaces and being
present at all the reviews is quite small.
Second, testing can provide a measure of software reliability (i.e. failures/execution time)
that is unobtainable from inspections. This measure can often be used as a vital input to
the release decision.
Thirdly, testing identifies system level performance and usability issues that inspections
cannot. Therefore, since inspections and testing provide different, equally important
information, one cannot replace the other. However, depending on the product, the
optimal mix of inspections and testing may be different!

A Closer Look: Fault Based Methods

The following paragraphs will describe some newer techniques in the software testing
field. Fault based methods include Error Based Testing, Fault seeding, mutation testing,
and fault injection, among others.
After briefly describing each of the 4 techniques, fault injection will be discussed in more
detail.
Error based testing defines classes of errors as well as inputs that will reveal any error
of a particular class, if it exists.
Fault seeding implies the injection of faults into software prior to test. Based on the
number of these artificial faults discovered during testing, inferences are made on the
number of remaining ‘real’ faults. For this to be valid the seeded faults must be assumed
similar to the real faults.
Mutation testing injects faults into code to determine optimal test inputs.
Fault Injection evaluates the impact of changing the code or state of an executing
program on behavior of the software.
These methods attempt to address the belief that current techniques for assessing
oftware quality are not adequate, particularly in the case of mission critical systems.
Voas et. al. suggests that the traditional belief that improving and documenting the
software development process will increase software quality is lacking. Yet, they
recognize that the amount of testing (which is product focussed) required in order to
demonstrate high reliability is impractical. In short, quality processes cannot demonstrate
reliability and the testing necessary to do so is impossible to perform.

64
 Testing

Fault injection is not a new concept. Hardware design techniques have long used
inserted fault conditions to test system behavior. It is as simple as pulling the modem out
of your PC during use and observing the results to determine if they are safe and/or
desired. The injection of faults into software is not so widespread, though it would
appear that companies such as Hughes Information Systems, Microsoft, and Hughes
Electronics have applied the techniques or are considering them. Properly used, fault
insertion can give insight as to where testing should be concentrated, how much testing
should be done, whether or not systems are fail-safe, etc.
As a simple example consider the following code:

Original Fault Injected

X = (r1 – 2) + (s2 – s1) X = (r1 – 2) + (s2 – s1)

Y=z–1 X = perturb(x)
… Y=z–1
T = x/y …
T = x/y
If T > 100 then print (‘WARNING’)

In this case it is catastrophic if T > 100. By using perturb(x) to generate changed values
of X (i.e. a random number generator) you can quickly determine how often corrupted
values of X lead to undesired values of T.
The technique can be applied to internal source code, as well as to 3rd party software,
which may be a "black box"
Conclusion
Software testing is an important part of the software development process. It is not a
single activity that takes place after code implementation, but is part of each stage of the
lifecycle. A successful test strategy will begin with consideration during requirements
specification. Testing details will be fleshed through high and low level system designs,
and testing will be carried out by developers and separate test groups after code
implementation. As with the other activities in the software lifecycle, testing has its
own unique challenges. As software systems become more and more complex, the
importance of effective, well planned testing efforts will only increase.

65
 Testing

Software Testing Techniques

 Software Testing Fundamentals

Testing Objectives, Principles, Testability
 Software Test Case Design
 White-Box Testing
Cyclomatic Complexity
Graph Matrices
Control Structuring Testing (not included)
Condition Testing (not included)
Data Flow Testing (not included)
Loop Testing (not included)
 Black-Box Testing
Graph-based Testing Methods (not included)
Equivalence Partitioning
Boundary Value Analysis
Comparison Testing (not included)

Software Testing Fundamentals

Software testing is a critical element of software quality assurance and

represents the ultimate review of specification, design, and coding.

Software testing demonstrates that software function appear to be working

according to specifications and performance requirements.

Testing Objectives:

66
 Testing

Myers [MYE79] states a number of rules that can serve well as testing objectives:

 Testing is a process of executing a program with the intent of finding an

error.
 A good test case is one that has high probability of finding an
undiscovered error.
 A successful test is one that uncovers an as-yet undiscovered error.

The major testing objective is to design tests that systematically uncover types of
errors with minimum time and effort.

Software Testing Principles

Davids [DAV95] suggests a set of testing principles:

 All tests should be traceable to customer requirements.

 Tests should be planned long before testing begins.
 The Pareto principle applies to software testing.
 80% of all errors uncovered during testing will likely be traceable to 20% of
all program modules.
 Testing should begin “in the small” and progress toward testing “in the
large”.
 Exhaustive testing is not possible.
 To be most effective, testing should be conducted by an independent third
party.

Software Testability

Software testability is simply how easily a computer program can be tested.

A set of program characteristics that lead to testable software:

 Operability: “the better it works, the more efficiently it can be tested.”

 Observability: “What you see is what you test.”
 Controllability: “The better we can control the software, the more the
testing can be automated and optimized.”
 Decomposability: “By controlling the scope of testing, we can more
quickly isolate problems and perform smarter retesting.”
 Simplicity: “The less there is to test, the more quickly we can test it.”
 Stability: “The fewer the changes, the fewer the disruptions to testing.”
 Understandability:”The more information we have, the smarter we will
test.”

Test Case Design

67
 Testing

Two general software testing approaches:

Black-Box Testing and White-Box Testing

Black-box testing:

knowing the specific functions of a software, design tests to demonstrate

each function and check its errors.

Major focus:

functions, operations, external interfaces, external data and

information

White-box testing:

knowing the internals of a software, design tests to exercise all internals of

a software to make sure they operates according to specifications and
designs
Major focus:

internal structures, logic paths, control flows, data flows

internal data structures, conditions, loops, etc.

White-Box Testing and Basis Path Testing

White-box testing, also known as glass-box testing.

It is a test case design method that uses the control structure of the procedural
design to derive test cases.

Using white-box testing methods, we derive test cases that

 Guarantee that all independent paths within a module have been exercised at
least once.
 Exercise all logical decisions one their true and false sides.
 Execute all loops at their boundaries and within their operational bounds.
 Exercise internal data structures to assure their validity.

Basic path testing (a white-box testing technique):

 First proposed by TomMcCabe [MCC76].

 Can be used to derive a logical complexity measure for a procedure design.
 Used as a guide for defining a basis set of execution path.
 Guarantee to execute every statement in the program at least one time.

68
 Testing

Cyclomatic Complexity

Cyclomatic complexity is a software metric

 provides a quantitative measure of the global complexity of a program.

When this metric is used in the context of the basis path testing, the value
computed for cyclomatic complexity defines the number of independent paths in
the basis set of a program.

Three ways to compute cyclomatic complexity:

 The number of regions of the flow graph correspond to the cyclomatic

complexity.
 Cyclomatic complexity, V(G), for a flow graph G is defined as V(G) = E - N
+2
where E is the number of flow graph edges and N is the number of flow graph
nodes.
 Cyclomatic complexity, V(G) = P + 1
where P is the number of predicate nodes contained in the flow graph G.
Deriving Test Cases

Step 1 : Using the design or code as a foundation, draw a corresponding flow

graph.

Step 2: Determine the cyclomatic complexity of the resultant flow graph.

Step 3: Determine a basis set of linearly independent paths.

For example,

path 1: 1-2-10-11-13
path 2: 1-2-10-12-13
path 3: 1-2-3-10-11-13
path 4: 1-2-3-4-5-8-9-2-…
path 5: 1-2-3-4-5-6-8-9-2-..
Path 6: 1-2-3-4-5-6-7-8-9-2-..

Step 4: Prepare test cases that will force execution of each path in the basis set.

Path 1: test case:

value (k) = valid input, where k < i defined below.

value (i) = -999, where 2 <= I <= 100
expected results: correct average based on k values and proper totals.

69
 Testing

Equivalence Partitioning

Equivalence partitioning is a black-box testing method

 divide the input domain of a program into classes of data

 derive test cases based on these partitions.

Test case design for equivalence partitioning is based on an evaluation of

equivalence classes for an input domain.

An equivalence class represents a set of valid or invalid states for input condition.

An input condition is:

 a specific numeric value, a range of values

 a set of related values, or a Boolean condition

Equivalence Classes

Equivalence classes can be defined using the following guidelines:

 If an input condition specifies a range, one valid and two invalid

equivalence class are defined.
 If an input condition requires a specific value, one valid and two invalid
equivalence classes are defined.
 If an input condition specifies a member of a set, one valid and one invalid
equivalence classes are defined.
 If an input condition is Boolean, one valid and one invalid classes are
defined.

Examples:

area code: input condition, Boolean - the area code may or may not be present.
input condition, range - value defined between 200 and 900

70
 Testing

password: input condition, Boolean - a password nay or may not be present.

input condition, value - six character string.

command: input condition, set - containing commands noted before.

Boundary Value Analysis

 a test case design technique

 complements to equivalence partition

Objective:

Boundary value analysis leads to a selection of test cases that exercise bounding
values.

Guidelines:

 If an input condition specifies a range bounded by values a and b, test cases

should be designed with value a and b, just above and below a and b.

Example: Integer D with input condition [-3, 10],

test values: -3, 10, 11, -2, 0

 If an input condition specifies a number values, test cases should be

developed to exercise the minimum and maximum numbers. Values just
above and below minimum and maximum are also tested.

Example: Enumerate data E with input condition: {3, 5, 100, 102}

test values: 3, 102, -1, 200, 5

 Guidelines 1 and 2 are applied to output condition.

 If internal program data structures have prescribed boundaries, be certain to
design a test case to exercise the data structure at its boundary

Such as data structures:

array input condition:

empty, single element, full element, out-of-boundary
search for element:
element is inside array or the element is not inside array

You can think about other data structures:

71
 Testing

list, set, stack, queue, and tree

72
Testing

73