Module 3: Software Testing

3.1 Fundamentals of Software Testing

3.2 Testing Strategy and Planning
3.3 Testing Methods
3.4 Economics of Testing
3.5 Testing Metrics
3.6 Defect Tracking
3.7 Templates and Checklists
3.8 Case Study
SQAM Course, Proficience, IISc 1
 Testing
 Goal is to find faults
 What kind of faults?
– Algorithmic
– Computation and Precision
– Documentation
– Overload
– Capacity
– Timing
– Performance
– Recovery
– System
– Standards

SQAM Course, Proficience, IISc 2

 Examples
 Code’s logic does not produce the proper output for a
given input
– Some examples
 Branch under wrong conditions

 Fail to set loop invariant

 Fail to initialize variables

 Comparing inappropriate data types

 Formula is implemented incorrectly

 Computation not performed to the required accuracy
 Data structures filled past their capacity
– Buffer overflow
– Dimension of tables larger than code can handle
– Length of queue too great
SQAM Course, Proficience, IISc 3
 Some more..
 Performance becomes unacceptable as activity reaches system limit
– E.g., system capable of handling 1,000,000 records becomes slower and
slower as more records are added
– May be examined in relation to disk access, number of processes, etc.
 System does not perform at the speed prescribed by the requirements
 Problem with underlying hardware or operating system
– Disk failure
– Out of memory
– Process table full
 Software doesn’t adequately handle the system fault
 Interface
– Fault in interacting with another component or driver
 Checking
– Fault in logic that fails to validate data properly before it is used
 Build/Package/Merge
– Fault that occurs because of problems in repositories and version control

SQAM Course, Proficience, IISc 4

 Testing Types

 Goal is to find the faults

 Different types of testing
– Unit testing
– Integration testing
– Functional testing
– Performance testing
– Acceptance testing
– Installation testing

SQAM Course, Proficience, IISc 5

 Testing Phases

Requirements System System Detailed

Specification Specification Design Design

System Sub-system Unit code and

Acceptance
Integration Integration Test
Test Plan
Test Plan Test Plan

Acceptance System Sub-system

Service
Test Integration test Integration test

SQAM Course, Proficience, IISc 6

 Unit Testing

 The first opportunity to test whether a particular module or

component is complete
 Often performed by the developer
 Can choose test cases in many ways. Some sample to get
broad test coverage:
– Statement testing : Every statement is executed at least once
– Branch testing : Every decision point is seen at least once.
– Path testing : Variation of above, each path through the
code tested at least once (visualize code as a flow chart)
– Definition-use path testing : Every path from every
definition of every variable to every use of that definition is
exercised
SQAM Course, Proficience, IISc 7
 Integration Testing

 Integration  Bottom-up
– Piecing together – Low-level first, with test
individual components harness
 Top-Down
to make a working
system – Stubs
– Individual components  Big-Bang

should already have – Put them together and see if

passed their unit tests it works
 Sandwich
 Integration Testing
– System viewed as three
– Tests to ensure the
layers
system functions when – Top-down in top layer,
components pieced bottom-up in lower,
together converge in the middle
SQAM Course, Proficience, IISc 8
 Top-Down Integration Testing

 Modules are integrated by

moving downward M1
through control hierarchy
 Modules subordinate to
M2 M3 M4
main control module are
incorporated
– Depth-first
M5 M6 M7
– Breadth-first

M1

SQAM Course, Proficience, IISc 9

 Top-Down Integration Testing
Strategy:

 Verifies major control or decision early in the test process.

 Allows early recognition of problems
 Depth first strategy allows a complete function of the software to be
implemented
 Stubs replace lower-level modules so no significant data can flow upward
– Delay some tests until have actual modules
– Add code to simulate module
Steps:

 The main control module is used as a test driver, and stubs are substituted for
all components directly subordinate to the main control module.
 Depending on the integration approach selected, subordinate stubs are
replaced one at a time with actual components.
 Tests are conducted as each component is integrated.
 On completion of each set of tests, another stub is replaced with a real
component .
 Regression testing may be conducted to ensure that new errors have not been
introduced.
SQAM Course, Proficience, IISc 10
 Bottom-Up Integration
 Begins construction and testing with atomic modules
 Need for stubs is eliminated
 Operational modules tested thoroughly
 Steps
– Low-level components are combined into clusters (builds) to perform specific sub-function
– A driver is written to coordinate test case input and output
– Cluster is tested Testing
– Drivers are removed and clusters are combined moving upward in program structure.
sequence

Test drivers

Level N Level N Level N Level N Level N

Test
drivers

Level N-1 Level N-1 Level N-1

SQAM Course, Proficience, IISc 11
 Sandwich integration Big-bang integration

 Combination of bottom-up and top-  Test all components in isolation, then

down integrations mix them all together and see how it
 The system is viewed as several works
layers with a targeted layer, chosen  Not recommended as it requires to
on the basis of system code both drivers and stubs
characteristics and the structure of  As all components are merged at once,
the component hierarchy it is difficult to identify the cause of a
 Top-down approach is used for the failure
top layer
 A bottom-up approach is used for
the bottom layer
 Allows integration to begin early in
the testing phase
 Does not test individual
components thoroughly before
integration

SQAM Course, Proficience, IISc 12

 Comparison of Integration Strategies

Bottom-Up Top-Down Big-Bang Sandwich

Integration Early Early Late Early

Time to basic working Late Early Late Early

program
Component drivers Yes No Yes Yes
needed
Stubs needed No Yes No Yes

Work parallelism at Medium Low High Medium

beginning
Ability to test Easy Hard Easy Medium
particular paths

SQAM Course, Proficience, IISc 13

 System Testing

 Software incorporated with hardware, people, information to form

system
 System testing is a series of different tests whose primary purpose is to
fully exercise the computer-based system
 Types
– Recovery testing
– Security testing
– Stress testing
– Performance testing

SQAM Course, Proficience, IISc 14

 Validation (Product) and
Acceptance Testing
 Validation test
– Series of tests that demonstrate conformity with requirements
– Test plan and procedure ensure that
 all functional requirements are satisfied

 all behavioral characteristics achieved

 all performance requirements attained

 documentation is correct

 usability and other requirements are met

 Acceptance test
– Alpha test: test at developer’s site by end-users
– Beta test: test conducted at end-user sites

SQAM Course, Proficience, IISc 15

 Results of Acceptance Tests

 Ideally: acceptance testing uncovers discrepancies in requirements

 Reality: by working with the system users discover aspects of the
problem which they were not aware
– New requirements
– New development and features
– Other direction:
 Some requirements may not be needed

SQAM Course, Proficience, IISc 16

 Installation Testing

 Installing
the system at the user site
 System-specific configuration
 Co-existence test with other software
– E.g. does it work with Office software?
– Does anything break if the software is
removed?

SQAM Course, Proficience, IISc 17

 Regression Testing

 Adding new or changing Approaches:

module impacts the system
– New data flow paths
established
– New I/O may occur
– New control logic invoked
 Regression testing is re-
execution of subset of tests that
have already been conducted
– Ensures changes have not
propagated unintended side
effects
Manual testing
Capture/Playback tools: capture test
cases and results for subsequent
playback and comparison
Test suite contains following classes of
test cases:
Representative sample of tests that
exercises all software functions
Focus on functions likely affected
by change
Focus on components that have been
changed

SQAM Course, Proficience, IISc 18

 Test Cases

 Want a broad set of test cases to cover the range of possible values and
code paths
 Closed Box
– Apply all possible inputs, compare with expected output according
to requirements
– Includes “out of range” inputs
 Open Box
– View code’s internal structure
– Generate tests based on this structure, e.g. give values on both
sides of an if-then else test

SQAM Course, Proficience, IISc 19

 Who Should Test?

 In your case, you (the developer) will test

– But the developer is often “too close” to the code to be objective
and recognize some subtle faults
– Difficult for developer to separate implementation structure and
required function
 Independent tester or test team is preferable
– Familiar with how to “break” software
– Familiar with testing methods and tools
 Users should be involved throughout the process
– Might catch missing requirements at early stages

SQAM Course, Proficience, IISc 20

 Automated Testing Tools
 Programs that will help test your code
 Static Analysis, e.g. lint
– Code Analysis
 Check syntax, of a construct is fault-prone, etc.

– Structure Checker
 Tool to depict logic flow, check for structural flaws

– Data analyzer
 Review data structures, illegal data usage, improper linkage

– Sequence checker
 Highlight events in the wrong sequence

 Dynamic Analysis, e.g. VTune

– Monitor and collect statistics as program is running
– How many times a statement was executed, branches covered, memory use,
etc.
– Useful for performance analysis to find the hotspots of the code

SQAM Course, Proficience, IISc 21

 How Much Testing?

 How do we know when to stop?

– When all tests pass?
– When ‘enough’ tests pass meeting some threshold
 Myers (1979)
– As the number of detected faults increases, the probability of the
existence of more undetected faults increases
– Suggests that if the number of detected faults is decreasing, we
may not have too many left and can stop soon
 Fault Seeding
– Deliberately add faults to the process
– Use their detection as a measure of the remaining faults

SQAM Course, Proficience, IISc 22

Test Metrics

Effective Test
Metrics

SQAM Course, Proficience, IISc 23

Defect Tracking

Bug defect life cycle

Defect Removal
Effectiveness

SQAM Course, Proficience, IISc 24

Case studies, Checklists, Templates

Testing Health Care

Creating Checklists
Road map for for testing
testing

Web Testing
Testing ERP V&V Template
Testing CRM

Load Plan template

Test Plan Outline

SQAM Course, Proficience, IISc 25

 References

 Effective Methods of Software Testing,

W.Perry
 Managing Testing Process, Rex Black

SQAM Course, Proficience, IISc 26