Test Coverage

Wei-Tek Tsai
Department of Computer Science and
Engineering
Arizona State University
Tempe, AZ 85287

What is Test Coverage

• measures the amount of testing done of a
certain type
• a measure of effort to detect a certain class
of potential errors
• a structural testing technique
• provides a measure of how well your test
suite actually tests the product

1
 Structural testing Vs. functional testing

• Structural testing
– compares test program behavior against the
apparent intention of the source code
– examines how the program works, taking into
account possible pitfalls in the structure and
logic

Structural testing Vs. functional testing

(Cont.)
• functional testing
– compares test program behavior against a
requirements specification
– examines what the program accomplishes,
without regard to how it works internally

2
 Complete Coverage
• The phrase, complete coverage, is
misleading
• "completeness" is measured only relative to a
specific population of possible test cases
• Even if you achieve complete coverage for a given
population of tests (such as, all lines of code
tested), you have not done complete, or even
adequate, testing

Two Broad Classes of Coverage Measures

• Path-based coverage
– requires the execution of particular components
of the program

• Fault-based coverage
– requires that the test suite exercise the program
in a way that would reveal likely faults.

3
Process of Test coverage analysis
• Finding areas of a program not exercised by
a set of test cases
• Creating additional test cases to increase
coverage, and
• Determining a quantitative measure of code
coverage, which is an indirect measure of
quality

Fundamental Measures
• Statement Coverage
– reports whether each executable statement is
encountered
– Also known as: line coverage, segment
coverage, C1, and basic block coverage

4
 Fundamental Measures (Cont.)
• Decision Coverage
– reports whether boolean expressions tested in
control structures evaluated to both true and
false
– Also known as: branch coverage, all-edges
coverage, basis path coverage, C2, decision-
decision-path testing

Fundamental Measures (Cont.)

• Condition Coverage
– reports the true or false outcome of each
boolean sub-expression, separated by logical-
and and logical-or if they occur
– similar to Decision Coverage but has better
sensitivity to the control flow

10

5
 Fundamental Measures (Cont.)
• Multiple Condition Coverage
– reports whether every possible combination of
boolean sub-expressions occurs

• Condition/Decision Coverage
– a hybrid measure composed by the union of
condition coverage and decision coverage

11

Fundamental Measures (Cont.)

• Modified Condition/Decision Coverage
– requires enough test cases to verify every
condition can affect the result of its
encompassing decision
• Path Coverage
– reports whether each of the possible paths in
each function have been followed

12

6
 Other Measures
• Function Coverage
– reports whether you invoked each function or procedure
• Call Coverage
– reports whether you executed each function call
• Linear Code Sequence and Jump (LCSAJ)
– variation of path coverage
– considers only sub-paths that can easily be represented
in the program source code, without requiring a flow
graph
13

Other Measures (Cont.)

• Data Flow Coverage
– variation of path coverage
– considers only the sub-paths from variable assignments
to subsequent references of the variables
• Object Code Branch Coverage
– reports whether each machine language conditional
branch instruction both took the branch and fell through
• Loop Coverage
– reports whether you executed each loop body zero
times, exactly once, and more than once
14

7
 Other Measures (Cont.)
• Race Coverage
– reports whether multiple threads execute the same code at the same
time
• Relational Operator Coverage
– reports whether boundary situations occur with relational operators
(<, <=, >, >=)
• Weak Mutation Coverage
– whether test cases occur which would expose the use of wrong
operators and also wrong operands
• Table Coverage
– indicates whether each entry in a particular array has
been referenced
15

Functional Coverage Criteria

• Requirements coverage

• Input domain coverage

• Output domain coverage

16

8
 Advanced Coverage Metrics for
Object-Oriented Software

• Traditional structural coverage metrics are inadequate

measures of test thoroughness for object-oriented software
systems

• New object-oriented context coverage metrics are required

to ensure thorough testing

17

Object-oriented context
coverage metrics
• Inheritance context coverage

• State-based context coverage

• Thread-based context coverage

18

9
 RTCA/DO-178B Coverage
• The international avionics standard for
safety critical software
• Imposes very strict Dynamic Coverage
Analysis requirements on projects

19

Intent of DO- 178B

• Describe objectives for Life- Cycle
Processes
• Describe process activities
• Describe evidence required at different
assurance levels

20

10
 Links for DO- 178B
• RTCA/DO-178B Guidelines
• Testing Embedded Software to
FAA Regulated Requirements
• DO-178B Description
• DO-178B, "Software Considerations in
Airborne Systems and Equipment
Certification"

21

Links for DO- 178B (Cont.)

• DO-178B and Safety-Critical Software
Technical Overview
• Aonix DO-178B Desktop
• Wind River Unveils Products for DO-
178B Airborne Systems Software
Certification

22

11
 Links for DO- 178B (Cont.)
• Guidelines for Applying the RTCA/DO-
178B Level D Criteria to Previously
Developed Software (PDS)
• Guidelines for the Qualification of Software
Tools Using RTCA/DO-178B
• Guidelines for the Approval of Software
Changes in Legacy Systems Using RTCA
DO-178B

23

Links for DO- 178B (Cont.)

• Tornado for DO-178B
• DO-178B & ED-12B Software
Verification Using CodeTEST® ACT
• DO-178B Certification with LDRA
Testbed®
• Birds Introduction to DO-178B

24

12
 References
• http://www.testing.com/writings/coverage-
terminology.html
• http://www.testing.com/writings/coverage.pdf
• http://www.testing.com/writings/experience.pdf
• http://www.kaner.com/coverage.htm#f1
• http://www.bullseye.com/webCoverage.html
• http://www.kaner.com/coverage.htm

25

References (Cont.)
• http://www.testing.com/writings/iview1.htm
• http://www.iplbath.com/pdf/p0833.pdf
• http://www.iplbath.com/pdf/p0823.pdf
• http://www.iplbath.com/pdf/p0833.pdf
• http://www.csc.calpoly.edu/~dbutler/tutorials/wint
er96/coverage/
• http://www.rtca.org
• http://www.stsc.hill.af.mil/crosstalk/1996/sep/num
ber.asp
26

13
 References (Cont.)
• http://www.ldra.co.uk/pages/core-info/sw_test.htm
• http://www.construx.com/seminars/flyers/CxReal
WorldTesting.htm
• http://www.ida.liu.se/~TDDB62/lectures/testing.p
df
• http://sern.ucalgary.ca/~denzinger/lecture17.ppt
• http://www.elet.polimi.it/Users/DEI/Sections/Com
peng/Mauro.Pezze/Tutorials/ICSE98/icse98/

27

References (Cont.)
• http://www.acm.org/sigada/conf/sigada2000/privat
e/SIGAda2000-CDROM/SIGAda2000-
Proceedings/Romanski-Presentation.pdf
• http://www2.informatik.uni-erlangen.de/IMMD-
II/Lehre/WS00_01/testing/TestSem.ppt
• http://www.cs.purdue.edu/homes/apm/talks/covera
ge-principle.ppt
• http://www.bullseye.com/coverage.html

28

14