Whitebox Testing

CS 420 - Spring 2007

A Testing Life Cycle

Error Requirement Specs Fault Fix Error Fault Resolution Fault Isolation Fault Classification incident Testing

Design
Fault Coding Fault

Error

Terminology
Error
Represents mistakes made by people

Fault
Is result of error. May be categorized as
Fault of Commission we enter something into representation that is incorrect Fault of Omission Designer can make error of omission, the resulting fault is that something is missing that should have been present in the representation

Cont
Failure
Occurs when fault executes.

Incident
Behavior of fault. An incident is the symptom(s) associated with a failure that alerts user to the occurrence of a failure

Test case
Associated with program behavior. It carries set of input and list of expected output

Cont
Verification
Process of determining whether output of one phase of development conforms to its previous phase.

Validation
Process of determining whether a fully developed system conforms to its SRS document

Verification versus Validation

Verification is concerned with phase containment of errors Validation is concerned about the final product to be error free

Relationship program behaviors

Program Behaviors

Fault Of Omission

Specified (expected) Behavior

Programmed (observed) Behavior

Fault Of Commission

Correct portion

Classification of Test
There are two levels of classification
One distinguishes at granularity level
Unit level System level Integration level

Other classification (mostly for unit level) is based on methodologies

Black box (Functional) Testing White box (Structural) Testing

Relationship Testing wrt Behavior

Program Behaviors 5 Specified (expected) Behavior 4 6 Programmed 2 (observed) 1 Behavior 3

Test Cases (Verified behavior) 8

Test methodologies
Functional (Black box) inspects specified behavior Structural (White box) inspects programmed behavior

Functional Test cases

Specified

Programmed

Test Cases

Structural Test cases

Specified

Programmed

Test Cases

When to use what

Few set of guidelines available A logical approach could be
Prepare functional test cases as part of specification. However they could be used only after unit and/or system is available. Preparation of Structural test cases could be part of implementation/code phase. Unit, Integration and System testing are performed in order.

Unit testing essence

Applicable to modular design
Unit testing inspects individual modules

Locate error in smaller region

In an integrated system, it may not be easier to determine which module has caused fault Reduces debugging efforts

Test cases and Test suites

Test case is a triplet [I, S, O] where
I is input data S is state of system at which data will be input O is the expected output

Test suite is set of all test cases Test cases are not randomly selected. Instead even they need to be designed.

Need for designing test cases

Almost every non-trivial system has an extremely large input data domain thereby making exhaustive testing impractical If randomly selected then test case may loose significance since it may expose an already detected error by some other test case

Design of test cases

Number of test cases do not determine the effectiveness To detect error in following code
if(x>y) max = x; else max = x;

{(x=3, y=2); (x=2, y=3)} will suffice {(x=3, y=2); (x=4, y=3); (x=5, y = 1)} Each test case should detect different errors

White-Box Testing
Statement coverage Branch coverage Path coverage Condition coverage Data flow-based testing

Statement Coverage
Statement coverage methodology:
design test cases so that every statement in a program is executed at least once.

The principal idea:

unless a statement is executed, we have no way of knowing if an error exists in that statement

Statement coverage criterion

Observing that a statement behaves properly for one input value:
no guarantee that it will behave correctly for all input values.

Example
1. 2. 3. 4. 5. 6. int f1(int x, int y){ while (x != y){ if (x>y) then x=x-y; else y=y-x; } return x; }

Euclid's GCD computation algorithm

By choosing the test set
{(x=3,y=3),(x=4,y=3), (x=3,y=4)} all statements are executed at least once.

Branch Coverage
Test cases are designed such that:
different branch conditions is given true and false values in turn.

Branch testing guarantees statement coverage:

a stronger testing compared to the statement coverage-based testing.

Example
Test cases for branch coverage can be:
{(x=3,y=3), (x=4,y=3), (x=3,y=4)}

Condition Coverage
Test cases are designed such that:
each component of a composite conditional expression given both true and false values.

Example
Consider the conditional expression ((c1.and.c2).or.c3): Each of c1, c2, and c3 are exercised at least once i.e. given true and false values.

Branch testing
Branch testing is the simplest condition testing strategy compound conditions appearing in different branch statements are given true and false values. (note: the entire condition is given true and false values, not ALL possible sub expressions)

Branch testing
Condition testing
stronger testing than branch testing:

Branch testing
stronger than statement coverage testing.

Condition coverage
Consider a Boolean expression having n components:
for condition coverage we require 2n test cases.

practical only if n (the number of component conditions) is small.

Path Coverage
Design test cases such that:
all linearly independent paths in the program are executed at least once.

Defined in terms of
control flow graph (CFG) of a program.

Control flow graph (CFG)

A control flow graph (CFG) describes:
the sequence in which different instructions of a program get executed. the way control flows through the program.

How to draw Control flow graph?

Number all the statements of a program. Numbered statements:
represent nodes of the control flow graph.

An edge from one node to another node exists:

if execution of the statement representing the first node can result in transfer of control to the other node.

Example
int f1(int x,int y){ 1. while (x != y){ 2. if (x>y) then 3. x=x-y; 4. else y=y-x; 5. } 6. return x; }

Example Control Flow Graph

1 2 3 5 6 4

Path
A path through a program:
A node and edge sequence from the starting node to a terminal node of the control flow graph. There may be several terminal nodes for program.

Independent path
Any path through the program:
introducing at least one new node or one new edge that is not included in any other independent paths.

It may be straight forward to identify linearly independent paths of simple programs. However For complicated programs it is not so easy to determine the number of independent paths.

McCabe's cyclomatic metric

An upper bound:
for the number of linearly independent paths of a program

Provides a practical way of determining:

the maximum number of linearly independent paths in a program.

McCabe's cyclomatic metric

Given a control flow graph G, cyclomatic complexity V(G):
V(G)= E-N+2
N is the number of nodes in G E is the number of edges in G

Example
Cyclomatic complexity = 7 6 + 2 = 3.

Cyclomatic complexity
The cyclomatic complexity of a program provides:
a lower bound on the number of test cases to be designed to get coverage of all linearly independent paths. only gives an indication of the minimum number of test cases required.

Path Testing - Test Cases

Draw control flow graph. (Loops can cause an explosion in the number of independent paths, so use looping values of 0 and 1 to test loops). Determine V(G). Determine the set of linearly independent paths. Prepare test cases:
to force execution along each path

Example Control Flow Graph

1 2 3 5 6 4

Derivation of Test Cases

Number of independent paths: 4
1, 6 test case (x=1, y=1) 1, 2, 3, 5, 1, 6 test case(x=1, y=2) 1, 2, 4, 5, 1, 6 test case(x=2, y=1)

An interesting application of cyclomatic complexity

Relationship exists between:
McCabe's metric the number of errors existing in the code, the time required to find and correct the errors.

Cyclomatic complexity
Cyclomatic complexity of a program:
also indicates the psychological complexity of a program. difficulty level of understanding the program.

Cyclomatic complexity
From maintenance perspective,
limit cyclomatic complexity
of modules to some reasonable value.

Some software development organizations:

restrict cyclomatic complexity of functions to some max number.

Data-flow-based Testing
Basic idea: test the connections between variable definitions (write) and variable uses (read) Starting point: variation of the control flow graph
Each node represents a single statement, not a chain of statements

Set DEF(n) contains variables that are defined at node n (i.e., they are written) Set USE(n): variables that are read

Example
Assume y is already initialized 1 s:= 0; 2 x:= 0; 3 while (x<y) { 4 x:=x+3; 5 y:=y+2; 6 if (x+y<10) 7 s:=s+x+y; else 8 s:=s+x-y; 1 2 3 4 5 6 DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) := DEF(3) := , USE(3) := {x,y} DEF(4) := {x}, USE(4) := {x} DEF(5) := {y}, USE(5) := {y} DEF(6) := , USE(6) := {x,y} DEF(7) := {s}, USE(7) := {s,x,y} DEF(8) := {s}, USE(8) := {s,x,y} DEF(9) := , USE(9) := DEF(10) := , USE(10) :=

7
9
10

Reaching Definitions
1 A definition of variable x at node n1 reaches node n2 if and only if there is a path between n1 and n2 that does not contain a definition of x 2 3 4

Reaches nodes 2,3,4,5,6,7,8, but not 9 and 10.

DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) := DEF(3) := , USE(3) := {x,y} DEF(4) := {x}, USE(4) := {x} DEF(5) := {y}, USE(5) := {y} DEF(6) := , USE(6) := {x,y} DEF(7) := {s}, USE(7) := {s,x,y} DEF(8) := {s}, USE(8) := {s,x,y}

5
6

7
9
10

Def-use Pairs
A def-use pair (DU) for variable x is a pair of nodes (n1,n2) such that
x is in DEF(n1) The definition of x at n1 reaches n2 x is in USE(n2)

In other words, the value that is assigned to x at n1 is used at n2

Since the definition reaches n2, the value is not killed along some path n1...n2.

Examples of Def-Use Pairs

Reaches nodes 2, 3, 4, 5, 6, 7, 8, but not 9,10 For this definition, two DU pairs: 1-7, 1-8 1 2 3 4

DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) := DEF(3) := , USE(3) := {x,y} DEF(4) := {x}, USE(4) := {x} DEF(5) := {y}, USE(5) := {y} DEF(6) := , USE(6) := {x,y} DEF(7) := {s}, USE(7) := {s,x,y} DEF(8) := {s}, USE(8) := {s,x,y}

5
6

7
9
10

Data-flow-based Testing
Identify all DU pairs and construct test cases that cover these pairs
Several variations with different relative strength

All-DU-paths: For each DU pair (n1,n2) for x, exercise all possible paths n1, n2 that are clear of a definition of x All-uses: for each DU pair (n1,n2) for x, exercise at least one path n1 n2 that is clear of definitions of x

Data-flow-based Testing
All-definitions: for each definition, cover at least one DU pair for that definition
i.e., if x is defined at n1, execute at least one path n1..n2 such that x is in USE(n2) and the path is clear of definitions of x

Clearly, all-definitions is subsumed by all-uses which is subsumed by all-DU-paths Motivation: see the effects of using the values produced by computations
Focuses on the data, while control-flow-based testing focuses on the control