Você está na página 1de 57

TESTING

Plan Tests Design Tests


White Box Black Box Regression

Implement Tests Perform Tests


Unit Integration System Acceptance

Evaluate Tests Debugging

testing is the process of finding differences between the specified (expected) and the observed (existing) system behavior
Goal: design tests that will systematically find defects aim is to break the system

usually done by developers that were not involved in system implementation to test a system effectively, must have a detailed understanding of the whole system not a job for novices it is impossible to completely test a nontrivial system

systems often deployed without being completely tested

Software configuration Test results

Expected results

Error rate data

Reliability & quality model

Testing

Evaluation Errors

No errors Done

Test configuration

Debug Corrections

the time uncertainty in testing is the debug part!

Core Workflows
Requirements

Phases
Inception
Elaboration

Construction

Transition

Analysis
Design Implementation Testing
iter. #1 iter. #2

iter. #n-1

iter. #n

Increments

Iteration

Test Engineer

Component Engineer

Integration System Tester Tester

responsible for

responsible for

responsible for

X
Test Model Test Case

X
Test Test Test Procedure Plan Evaluation Test Component

Defect

Test Engineer

Plan Tests

Design Tests

Evaluate Tests

Integration Tester

Perform Integration Tests

System Tester

Perform System Tests

Component Engineer

Implement Tests

Perform Unit Test

testing cannot show the absence of software errors;


it can only show that software errors are present!

unlike software construction,


testing is a destructive process > try to make the software fail

a test is successful only if it uncovers a defect (bug) hence, good testing has the opposite nature
of good software engineering (i.e., for good software engineering, success > no defect found for good testing, success > defect found)

It is impossible to completely test the entire system!


Overall goal: design a set of tests that has the highest likelihood of uncovering defects with the minimum amount of time and effort since resources are usually limited
(up to 40% of project effort often devoted to testing)

Testing strategy: specifies the criteria and goals of testing


what kinds of tests to perform and how to perform them the required level of test and code coverage the percentage of tests that should execute with a specific result

Estimate of resources required: human/system Schedule for the testing: when to run the tests

test case: one way of testing the system (what, conditions, how)

knowing the internal workings of a component, design test cases


to ensure the component performs according to specification White Box Testing: testing-in-the-small Derive test cases to verify component logic based on data/control structures. > Availability of source code is required.

knowing the specified functionality of a component, design


test cases to demonstrate that the functionality is fully operational Black Box Testing: testing-in-the-large Derive test cases to verify component functionality based on the inputs and outputs. > Availability of source code is not required! Regression Testing: selective White Box and Black Box re-testing to ensure that no new defects are introduced after a change

Goal: ensure that we have executed/exercised:

all independent paths in the code at least once all logical decisions on their true and false sides all loops at their boundaries and within their bounds internal data structures to ensure their validity

Why is this important?

logic errors and incorrect assumptions are inversely proportional to a paths execution probability

errors more likely in less used parts of the code

we often believe that a path is not likely to be executed

reality is often counter intuitive

typos are random

its likely that untested paths will contain some

How long would it take to exhaustively test this program?

loop < 20 times

There are 1014 possible paths!


It takes 3,170 years to exhaustively test the program if each test requires 1 ms!!

i=j+1 if (A && B { x = y + z; } n = p + 1;
suppose we modify the program to:
i=0 if (A && B { i = 1; } x = y / i;

i=j+1
False

A && B

True

i=j+1 i=j+1

Even though the False


branch executes no additional statements, it needs to be tested!

Goal: exercise each independent path in the code at least once

1. From the code, draw a corresponding flow graph.

. . .

Sequence If-then-else Case

Do-while

Do-until

procedure: process records 1. do while records remain 2. read record; 3. if record field 1 = 0 4. then store in buffer; 5. increment counter; 6. elseif record field 2 = 0 7. then reset counter; 8. else store in file; 9. endif 10. endif 11. enddo end

2.Determine the cyclomatic complexity of the flow graph. cyclomatic complexity: a quantitative measure of the logical complexity of code

provides an upper bound on the number of paths


that need to be tested in the code Ways to compute cyclomatic complexity V(G):

V(G) = number of regions (areas bounded by nodes and edges


area outside graph is also a region)

V(G) = number of edges - number of nodes + 2


V(G) = number of (simple) predicate nodes + 1

3.Determine a basis set of linearly independent paths.

independent path any path that introduces at least one new


set of processing statements or a new condition

basis set set of independent paths through the code

test cases derived from a basis set are guaranteed to execute


every statement at least one time during testing

basis set is not unique

1 2 3 6 7 11 8 4 5 10

1-2-3-6-7-9-10-1-11 1-2-3-6-8-9-10-1-11 1-2-3-4-5-10-1-11 1-11

4. Prepare test cases that force the execution of each path in the basis set.
Notes:

you do not need an activity diagram, but the picture will help when you trace component paths
count each logical testcompound tests count as the number of Boolean operators + 1 basis path testing should be applied to all components, if possible, and to critical components always

Flag1=FALSE int Q; ... i = j + 1; False if ( A() && B() ) { x = y + z; } n = p + 1; ...


Boolean A() { if (Flag1) { Q = 0; return TRUE; } else { return FALSE; } }

i=j+1
A()&&B() True

Flag1=TRUE Flag 2=TRUE This is sufficient for basis path testing

x=y+z n=p+1

Boolean B() { if (Flag2) { return TRUE; } else { int x = 10/Q; return FALSE; } }

What happens when Flag1=TRUE and Flag2=FALSE?

Goal: further exercises the true and false value of each simple logical condition in the component

condition testing
simple condition: (a rel-op b) where rel-op={<, , =, , , >} (may be negated with NOT), e.g., ab; NOT(ab) compound condition: two or more simple conditions connected with AND, OR, e.g., (a>b) AND (c<d) relational expression: (E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g., ((a*b+c)>(a+b+c))

errors to test for include (incorrect/missing/extra):


Boolean operator Boolean variable Boolean parenthesis relational operator arithmetic expression

branch testing

for a compound condition C, test true and false branches of C and every simple condition of C
e.g., for C = (a>b) AND (c<d) we test for: C a>b c<d TRUE and FALSE TRUE and FALSE TRUE and FALSE

domain testing

for an expression E1 rel-op E2, test for E1 greater than, equal to, or less than E2
guarantees detection of rel-op error if E1 and E2 are correct to detect errors in E1 and E2, the difference between E1 and E2 for the tests E1 greater or less than E2 should be as small as possible for an expression with n variables, 2n tests are required

Goal: execute loops at their boundaries and within their bounds

Simple Loops (n iterations)

1 skip the loop entirely 2 only one pass through the loop 3 two passes through the loop 4 m passes through the loop where m < n 5 n-1, n, n+1 passes through the loop

Nested Loops
1 conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration 2 work outward, conducting test for the next innermost loop 3 continue until all the loops have been tested

tests grow geometrically as the level of nesting increases

Concatenated Loops independent loops simple loop testing dependent loops nested loop testing

Unstructured Loops

redesign!!!

Goal: ensure that the value of a variable is correct at certain points of execution in the code

select test paths according to the locations of


definitions and uses of a variable DEF(S) = {X | S contains a definition of X} locations of definition of X USE(S) = {X | S contains a use of X} locations of use of X

Data Use (DU) Chain (X) is the set of [X, S, S] where X is live (i.e., not redefined between S and S)
one testing strategy: every data use chain must be covered once

for every variable, do a test along the path from where the
variable is defined to the statement(s) where the variable is used

input

Code

actual ? predicted output = output

black box techniques attempt to find:


incorrect or missing functions interface errors data structure or external database access errors performance errors initialization and termination errors

Test cases should:

reduce by more than 1 the number of additional test cases required tell us something about presence or absence of a class of errors
(e.g., all character data is correctly/incorrectly processed)

partition inputs and outputs to create thorough test coverage of a class of errors

user queries
mouse picks on menu

numerical data

output format requests responses to prompts

all possible input values


command key input

design test cases which are based on


valid and invalid inputs

1. Input is a range > one valid and two invalid equivalence classes:

less than range

in range

greater than range

2. Input is a specific value > one valid and two invalid equivalence classes:

less than value

value

greater than value

3. Input is a set of related values > one valid and one invalid class:
not member of set member of set

4. Input is Boolean > one valid and one invalid class:


non Boolean Boolean

more errors occur at the boundaries of an input domain than in the center

user queries

numerical data

output format requests

mouse picks on menu

responses to prompts
command key input

as well as selecting test data inside an

equivalence class, we also select data at the edges of the equivalence class

1. If the input range is bounded by a and b, then use a, b, and values just above and just below a and b, respectively.

2. If the input is a number of values, use the minimum and maximum of the values and values just above and just below them, respectively.
min max

3. Apply guidelines 1 and 2 to create output values at the minimum and maximum expected values.

4. If data structures have boundaries, test these boundary values and values just above and just below them, respectively.
e.g., for an array of bounds 1 to 10 test array index equal to 0, 1,2, 9, 10, 11

Error Guessing
use application domain experience or knowledge to select test values Cause-Effect Graphing
1. causes (input conditions) and effects (actions) are identified for a subsystem 2. a cause-effect graph is developed 3. the graph is converted to a decision table 4. decision table rules are converted into test cases

Comparison Testing
when reliability is absolutely critical, write several versions of the software perhaps to be used as backups run same test data on all versions; cross-check results for consistency Caution: if the error is from a bad specification and all versions are built from the same specification, then the same bug will be present in all copies!

event-based approach where tests are based on events which trigger system actions particularly appropriate for object oriented
systems

used after classes have been unit tested and integrated into subsystems need to identify and execute each possible processing thread from use cases

may not be possible to do complete thread testing

I1(obj2) I1(obj1)

obj1
I2(obj1) I3(obj1) I1(obj3)

obj2

obj5

O1(obj5)

obj3 obj4

O1(obj4) O2(obj4)

multiple thread: obj1obj2{obj5, obj4} multi-input thread: into obj1

single thread: obj3obj2obj4

focuses on comparing the resulting state of a class with the expected state derive test cases using statechart diagram for a class derive a representative set of stimuli for each transition check the attributes of the class after applying a stimuli to determine of the specified state has been reached first need to put the class in the desired state before applying the stimuli

Test case name name of the test case


Description of the test what the test is designed to do Target class/component/subsystem name name of the thing to be tested Target class/component/subsystem operation name of the operation to be tested Test type black box/white box; valid/invalid input

Goal: automate test procedures as much as possible

running test cases can be very tedious and time consuming


many possible input values and system states to test

test component: a program that automates one or several test procedures or parts of them

there are tools available to help write test components:


perform actions for a test case and the tool records the actions parameterize the recorded script to accept a variety of input values

spreadsheets and/or database applications can be used to


store the required input data and the results of each test

we need to integrate test cases into a wellplanned series of steps that are taken to test a component/subsystem > test procedure
what are the steps that need to be conducted to test a component? the testing steps start at the component /subsystem level and work when are the steps planned and system undertaken? outward toward the integration of the entire how much effort, time and resources will be a testing strategy indicates which testing techniques (white box, required to do the steps?
black box, thread) are appropriate at which point in time

need to balance:
flexibility & creativity with planning & management

testing often occurs when deadline pressures are most severe


(progress must be measurable and problems identified as early as possible)

Requirements capture Analysis Design

R A D C U

We test the system from the inside out

Implementation (coding)

Unit testing
Integration testing System testing

We develop the system from the outside in

I
ST AT

Acceptance testing

Component

interface local data structures boundary conditions independent paths error-handling paths

Test cases

emphasis is on White Box techniques

Driver

interface local data structures boundary conditions independent paths error-handling paths

Component to be tested

Results Stub Stub

Test cases

driver: program that calls component to be tested stub: program called by component to be tested

driver and/or stubs must be developed


for each unit test

If components all work individually, why


more be testing? Interface errors cannot uncovered by unit testing
integration approaches:

Incremental construction strategy

Big Bang!!

versus

Incremental builds Regression testing

S1 S2 S3 S4 S5 S6

top subsystem is tested with stubs S7 stubs are replaced one at a time depth-first or breadth-first

as new subsystems are integrated, some subset of tests is re-run (regression testing)

pro > early testing and error detection of user interface components; can demonstrate a complete function of the system early con > cannot do significant low-level processing until late in the testing;

S1 D2 S2 S6 S7

S3

drivers are replaced one at a time depth-first

S4

S5

subsystems are grouped into builds and integrated

pro> interface faults are more easily found; easier test case design and no need for stubs con > user interface components are tested last

S1 S2 S2 S6

top-level subsystems are tested with stubs S7

D2
S3 S4

lower-level subsystems are grouped into builds and tested with drivers

pro> parallel testing activities


S5

con > may need additional drivers and stubs

Critical subsystems should be tested as


early as possible!
Critical subsystems are those that: 1. have high risk 2. address several software requirements (i.e., several use cases) 3. have a high level of control 4. are complex or error prone > high cyclomatic complexity 5. have specific performance requirements

Regression testing is a must for these subsystems!

We need to test the entire system to be sure the system functions properly when some specific types of system tests: integrated
Functional Performance > developers verify that the nonfunctional requirements are met Pilot Acceptance

> developers verify that all user functions work as specified in the system requirements specification

> a selected group of end users verifies common functionality in the target environment > customer verifies usability, functional and nonfunctional requirements against system requirements specification > customer verifies usability, functional and nonfunctional requirements in real use

Installation

stress tests > verify that the system can continue functioning when confronted with many simultaneous requests volume tests > verify that thego? system large amounts How high can we Do can we handle fail-soft or
of collapse? data, high complexity algorithms, or high disk fragmentation security tests > verify that access protection mechanisms work

make penetration cost more than value of entry


timing tests > verify that the system meets timing constraints

usually for real-time and embedded systems


recovery tests > verify that the system can recover when forced to fail in various ways

database recovery is particularly important

Alpha test: a test in a controlled environment so that developers can observe users software
Developer site

customer tests
Customer site

Beta test: a test in a real environment so that bugs are uncovered from regular usage patterns

developer reviews
Developer site

customer tests

software
Customer site

the process of ensuring that the software meets the users reasonable expectations requirements specification from the start is important!

Can be very subjective good

The method of deficiency resolution should


be negotiated before the transition phase!

acceptance tests (black box) cover classes of tests such as:


functionality documentation

We need to
evaluate the results of our testing compare the results with the goals outlined in the test plan prepare metrics that let the test engineers determine the current quality of the software How do we know when to stop testing?

We can consider:
testing completeness: % of test cases run and % of code tested

reliability: based on testing error rate of previous projects

Failures per test hour

Possible outcomes: perform additional tests to find more defects relax the test criteria, if set too high deliver acceptable parts of the system; continue testing unacceptable parts
This project

Expected failure rate

Testing time

expected failure rate. We compare this with actual failure rate for

Past history can be used to plot

What to unit test?

unit test has to be at least a class, but object state makes testing difficult a class must be tested in every state it can ever enter (i.e., state-based testing)

Encapsulation
encapsulation hides what is inside an object > hard to know its state

need to provide a method, for testing only,


that reports all of an objects state

Inheritance and polymorphism


If a subclass overrides methods of an already tested superclass, what needs to be tested only the overridden methods?

No! All of a subclasss methods need to be tested again


due to dynamic binding and substitutability

All test are important, but if you need to restrict testing

testing a systems capabilities is more important than testing its components


identify things that will stop users doing their jobs

testing old capabilities is more important than testing new capabilities


users expect existing features to keep working

testing typical situations is more important than testing boundary cases


normal usage patterns are more important than atypical ones

a consequence of testing, but a separate activity > not testing can be one of the most exasperating parts of development > easy to lose focus or follow the wrong path!

Debugging is like trying to cure a sick person.

time required to correct the error


We see the symptoms and try to find the cause.

time required to determine the nature and location of the error

symptom and cause may be geographically separated symptom may disappear when another problem is fixed cause may be due to a system or compiler error

cause may be due to human error that is hard to trace


cause may be due to assumptions that everyone believes symptom may be intermittent

symptom cause

So, what can we do?

pressure to fix bugs often causes new bugs to be introduced

1.brute force: try to let the computer find the error


memory dumps run-time traces everywhere

The hope is that somewhere in all this information


well find the bug!

2. backtracking: start where the problem occurs and read the code backward until the problem is found

OK for small programs.


3. cause elimination: try to create input test data sets that would prove or disprove a particular theory

Difficult, but sometimes your best bet!

debugging tools can help gain more insight into the problem

talk to your colleagues They can see things youve grown too tired to see! isolate and focus > Think through the problemsome of the best debugging happens away from the computer! Be absolutely sure to conduct regression tests when you fix the bug!