Você está na página 1de 156

SW Testing for Microcontroller Applications

Software Testing
for
Microcontroller Applications
by Eberhard De Wille

Copyright 2010 Eberhard De Wille

Page 1

SW Testing for Microcontroller Applications


Content of the Course
1.
2.
3.
4.
5.
6.
7.

Motivation Some famous SW bugs


Some definitions related to SW testing
SW testing overview
Test documentation according to IEEE 829
Defect data collection
Test organization
State of the art testing techniques (white-box testing techniques, black-box testing
techniques, integration testing techniques
8. Static white-box tests
9. Functional tests (black-box tests)
10. Dynamic testing with a Perl testing environment
11. White-box testing (when is it needed? What has to be observed?)
12. Integration testing
13. System testing
14. Recommendations for the application of SW tests
15. SPICE requirements for SW testing

Copyright 2010 Eberhard De Wille

Page 2

SW Testing for Microcontroller Applications

Motivation:
Some Famous Software Bugs

Copyright 2010 Eberhard De Wille

Page 3

SW Testing for Microcontroller Applications


Software glitches leave Navy Smart Ship dead in the water
The Navys Smart Ship technology may not be as smart as the service contends.
Navy brass have called the Yorktown Smart Ship
pilot a success in reducing manpower, maintenance
and costs. The Navy began running shipboard
applications under Microsoft Windows NT so that
fewer sailors would be needed to control key ship
functions.
But the Navy last fall learned a difficult lesson about
automation: The very information technology on
which the ships depend also makes them vulnerable.
The Yorktown last September suffered a systems failure when bad data was fed into its
computers during maneuvers off the coast of Cape Charles, Va. A navigation officer entered
the speed of the ship as -0,0 knots. The input was not checked by the software and the minus
sign caused the problem.
The ship had to be towed into the Naval base at Norfolk, Va., because a database
overflow caused its propulsion system to fail, according to Anthony DiGiorgio, a
civilian engineer with the Atlantic Fleet Technical Support Center in Norfolk.
Copyright 2010 Eberhard De Wille

Page 4

SW Testing for Microcontroller Applications


On 5 April 1992 we experienced a shattering computer error during a German
election of the parliament for the state of Schleswig- Holstein.
At first it looked like the Green party was hanging on by
their teeth to a vote percentage of exactly 5%.
In the constellation at that time this meant that the Social
Democrats (SPD) won all precincts and only have direct
elected members of parliament. No extra seats were
needed for them and they could not have anyone from
their list seated. This was most unfortunate, as the
candidate for minister president was number one on the
list!
After midnight (and after the election results were published) someone discovered that the
Greens actually only had 4,97% of the vote. The program that prints out the percentages only
uses one place after the decimal, and had *rounded the count up* to 5%! This software had
been used for *years*, and no one had thought to turn off the rounding at this very critical
region!
So 4,97% of the votes were thrown away, the seats were recalculated, the SPD got to seat one
person from the list, and now have a one seat majority in the parliament. And the newspapers
are clucking about the "computers" making such a mistake.

Copyright 2010 Eberhard De Wille

Page 5

SW Testing for Microcontroller Applications


On 23 August 1991 a "Sleipner A" North Sea oil and gas drilling platform sank off the
Norwegian coast. This economic loss of 700 US$ was caused by a bug in the design SW.

The sea depth is 220m at this place. The platform was supported by tanks at 89m depth.
The wall of a tank cracked. The pumps could not cope with the water and the platform
tilted over and sunk. The event was noticed as a level 3.0 earthquake at the seismic
stations in the area!
The failure was cause by the simulation and design software NASTRAN. The stress on
the tanks was underestimated by 47% due to a calculation error in a complex formula.
Therefore the tanks were build in a way that they only could last at a maximum depth of
62m.
Copyright 2010 Eberhard De Wille

Page 6

SW Testing for Microcontroller Applications


On 31. March 2005 Daimler Chrysler recalled 1.3 million vehicles of the E- SL and CLS
classes.

Among other HW related failures, some SW bugs were detected which lead to this
biggest recall in the history of DC.
Failures were detected in the software of the power control module
Other SW failures were detected in the brake system control units.
There are already some minor accidents which customers claim to be caused by these
bugs.
The whole cost and dimension of the problem is currently not know. But it seems to be
of a big financial impact. Suppliers (mainly BOSCH are also affected).
SPIEGEL ONLINE - 31. Mrz 2005, 16:04 URL: http://www.spiegel.de/auto/werkstatt/0,1518,349049,00.html
Copyright 2010 Eberhard De Wille

Page 7

SW Testing for Microcontroller Applications

Some Definitions
related to SW Testing

Copyright 2010 Eberhard De Wille

Page 8

SW Testing for Microcontroller Applications

What Test is not


Test is not debugging. Debugging has the goal to remove errors.
The existence and the approximate location of the error is known.
Debugging is not documented. There is no specification and there will be
no record (log) or report. Debugging is done prior to commissioning of
software and debugging is the result of testing but never a substitution for it.
Test can never find 100% of the included errors. There will be always
a rest of remaining errors which can not be found. Each kind of test will
find a different kind of errors.
Test has the goal to find errors and not their reasons.

Copyright 2010 Eberhard De Wille

Page 9

SW Testing for Microcontroller Applications


V- Cycle of SW development
Documents and Actions
System Specification
System Test
Specification

System
Tests
System Test Report

SW Requirement
Specification
Software Req. Test
Specification

SW Requirements Tests
SW Requirements
Test Report
Integration
(Integrated Modules) Integration Tests

SW Design
Document(s)
Module Test
Specification
s

Debugging and / or
some form of
Module Tests

SW Implementation
(Source Code)

Copyright 2010 Eberhard De Wille

Integration Test
Reports
Module Tests
Module Test
Reports

Static SW Tests
(Lint + Review)

Static SW Test
Reports

Page 10

SW Testing for Microcontroller Applications

What is Test?
Test is a formal activity. It involves a strategy and a systematic approach.
The different stages of tests supplement each other. Tests are always
specified and recorded.
Test can be planned. The workflow and the expected results are specified.
Therefore the duration of the activities can be estimated. The point in time
where tests are executed is defined.
Test is the required formal proof of software quality.

Copyright 2010 Eberhard De Wille

Page 11

SW Testing for Microcontroller Applications


Some definitions:
Validation:
The process of evaluating a system or component during or at the end of
the development process to determine whether it satisfies specified
requirements.

Verification:
(1) The process of evaluating a system or component to determine whether
the products of a given development phase satisfy the conditions imposed at
the start of that phase. Contrast with: validation.
(2) Formal proof of program correctness. See: proof of correctness.

Proof of correctness:
(1) A formal technique used to prove mathematically that a computer
program satisfies its specified requirements.
(2) A proof that results from applying the technique in (1).
Source: IEEE Std 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology
Copyright 2010 Eberhard De Wille

Page 12

SW Testing for Microcontroller Applications


Some definitions:
Fault:
An incorrect step, process, or data definition in a computer program.
Note: This definition is used primarily by the fault tolerance discipline. In
common usage, the terms "error" and "bug" are used to express this
meaning.

Failure:
The inability of a system or component to perform its required functions
within specified performance requirements. Note: The fault tolerance
discipline distinguishes between a human action (a mistake), its
manifestation (a hardware or software fault), the result of the fault (a
failure), and the amount by which the result is incorrect (the error).

Source: IEEE Std 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology
Copyright 2010 Eberhard De Wille

Page 13

SW Testing for Microcontroller Applications

Conclusion:
Fault:
Is a static property and can be identified before running the SW
It can be identified in the source code or design as a mistake
Not all faults fail

Failure:
Is a dynamic property and can be identified at runtime
It is the difference between expected and actual runtime behavior
Every failure is caused by at least one fault

Copyright 2010 Eberhard De Wille

Page 14

SW Testing for Microcontroller Applications


Approximately 8% of all faults lead to a serious failure
Reducing the number of faults is the aim of testing. It will reduce the
number of failures
It is important to measure the number of faults in SW
There is no methodology to predict if a fault will fail and how serious
it will fail
The language choice is not changing the situation!

All faults

Faults which
will fail

Copyright 2010 Eberhard De Wille

Page 15

SW Testing for Microcontroller Applications


Something to observe for microcontroller applications:

Probability that a failure appears in a system

100%

System operation time

5000h

10000h

Copyright 2010 Eberhard De Wille

15000h

Page 16

SW Testing for Microcontroller Applications


Measuring the number of faults:
Measure the fault density as the number of faults in thousand lines
of code
The number of lines of codes can be defined differently: e.g. total
new lines, non-blank pre-processed lines or executable lines
Use the executable lines to measure!

Faults/KLOC

Grade

0.1 - 1

very good

1-5

good (average state of the art)

5 - 10

o.k. (reasonable commercial system)

> 10

Bad (poor systems like windows)

Copyright 2010 Eberhard De Wille

Page 17

SW Testing for Microcontroller Applications


Which improvements can you expect?
Faults
per
KLOC

Errors per KLOC at NASA Goddard

12
10
8
6
4
2

Some improvements can be seen but they are not overwhelming!

0
1976 1978 1980 1982

1984 1986 1988 1990 1992

Copyright 2010 Eberhard De Wille

Page 18

SW Testing for Microcontroller Applications

SW Testing Overview

Copyright 2010 Eberhard De Wille

Page 19

SW Testing for Microcontroller Applications

System
Tests
typically done by system team
System Integration Tests
regression tests by SW
team

Software Tests
Component Tests
Software Integration Tests

dynamic
tests

Test environment for execution e.g. CPPUnit, Perl


Module Tests

static
tests

functional
tests
(black box)
structural
tests
(white box)

Automatic code checkers, reviews, inspections, etc.

Copyright 2010 Eberhard De Wille

Page 20

SW Testing for Microcontroller Applications

Overview of Tests (1)


Static tests
The software is not executed but analyzed offline. In this category belong
code inspections (e.g. Fagan inspections), Lint checks, etc.

Dynamic tests
This requires the execution of the software or parts of the software (using stubs).
It can be executed in the target system, an emulator or simulator, or a test bench e.g.
on a PC. Within the dynamic tests the state of the art distinguishes between
structural tests (white box) and functional tests (black box).

Copyright 2010 Eberhard De Wille

Page 21

SW Testing for Microcontroller Applications


Overview of Tests (2)
Structural tests
These are so called "white-box tests" because they are performed with the
knowledge of the source code details. Structural tests are first of all static tests,
but can also be dynamic white-box tests, where software is executed and test
cases are selected according to the known structure of the software.
Functional tests
These are the so called "black-box" tests. The software is regarded as a unit
with unknown content. Inputs are stimulated and the values of the output results
are recorded and compared to the expected and specified values. The software is
also stressed with critical values at the boundaries of the input values and even
with illegal input values.

Copyright 2010 Eberhard De Wille

Page 22

SW Testing for Microcontroller Applications


Overview of Tests (3)
Module test or Unit test
A module is the smallest unit of source code to be compiled. If it is too small
to allow functional tests (black-box tests) it is recommended to combine modules
to components. A module test will usually require the employment of test stubs.
Component test
This is the black-box test of modules or groups of modules which represent a
certain functionality. There are no rules about what can be called a component.
It is just what the tester defined to be a component, however it should make
sense and be a testable unit. Components can be step by step integrated
to bigger components and tested as such.

Copyright 2010 Eberhard De Wille

Page 23

SW Testing for Microcontroller Applications


Overview of Tests (4)
Integration test
The software is step by step completed and tested by functional tests covering
different levels of components, preferably on a test bench. This is followed by
integration in the target hardware. There should be integration tests which
have to be executed during this integration into the target hardware, followed
by system tests of the completely integrated product.
System test
This is a black-box test of the complete software in the target system.
The environmental conditions have to be realistic (complete original hardware
in an appropriate environment). Other system tests will follow (sensors, mech.)

Copyright 2010 Eberhard De Wille

Page 24

SW Testing for Microcontroller Applications


Which Test finds which Error ?
Possible error

Can be found by

Example

Operating
system errors,
architecture and
design errors

Design inspection, Disturbances by OS interruptions or hardware interrupts,


integration tests
timing problems, lifetime and duration problems.

Integration errors

Integration tests,
system tests

Resource problems (runtime, stack, registers, memory, etc.)

System errors

System tests

Wrong system behavior, specification errors

Copyright 2010 Eberhard De Wille

Page 25

SW Testing for Microcontroller Applications


Which Test finds which Error ?
Possible error

Can be found by

Example

Syntax errors

Compiler, Lint

Missing semicolons, Values defined but not initialized or


used, order of evaluation disregarded.

Data errors

Software
inspection,
module tests

Overflow of variables at calculation, usage of inappropriate


data types, values not initialized, values loaded with wrong
data or loaded at a wrong point in time, lifetime of pointers.

Algorithm errors
Logical errors

Software
inspection,
module tests

Wrong program flow, usage of wrong formulas and


calculations (e.g. not precise enough)

Interface errors

Software
inspection,
module tests,
component tests

Overlapping ranges, range violation (min. and max. values


not observed or limited), unexpected inputs, wrong
sequence of input arguments.

Copyright 2010 Eberhard De Wille

Page 26

SW Testing for Microcontroller Applications


Define Test End Criteria !

Faults
per
KLOC

Asymptotic fault density

found
faults per
day

Copyright 2010 Eberhard De Wille

Time of testing (days)


Page 27

SW Testing for Microcontroller Applications


Other possible Test End Criteria
100% free of PC-Lint errors and warnings
100% C1 Code Coverage achieved
100% known Requirements tested ok
Etc.

Copyright 2010 Eberhard De Wille

Page 28

SW Testing for Microcontroller Applications

Test Documentation according


to IEEE 829

Copyright 2010 Eberhard De Wille

Page 29

SW Testing for Microcontroller Applications

Test documentation is the vital element which raises any


"try out" activities to the level of a proper test.
Even if the way something is tested is good, it is
worthless if it is not documented!
Test documentation is important for product liability
Test documentation is the proof of the quality of your
software!

Copyright 2010 Eberhard De Wille

Page 30

SW Testing for Microcontroller Applications

Test plan

Test plan
Scope of the test activities
Test Methods
Test Tools
Schedule and sequence
Test objects
Test attributes
Responsibilities
Risks

Copyright 2010 Eberhard De Wille

Definition of Test Specifications


Definition of Test Reports
Definition of Test End Criteria
Type of Test Inputs / Data
Which Test is applied in which
sample phase

Page 31

SW Testing for Microcontroller Applications


Effort Estimation and Schedule of Test Activities

Use a standard tool for the schedule of test activities,


e.g. MS-Project

Copyright 2010 Eberhard De Wille

Page 32

SW Testing for Microcontroller Applications

Test design
specification
Method
Approach
Test environment
(test stubs, make
files, recording
facilities, etc.).

Copyright 2010 Eberhard De Wille

Test case /
procedure
specification
Test object(s)
Test Attributes
Steps to be executed
Expected results

Page 33

SW Testing for Microcontroller Applications


Test Case Specification
Test Case Description
Description of the purpose, scope and intention of the test case
Test Precondition:
Prerequisites for the execution of this test case e.g.:
successful execution of other test cases
availability of additional test software (e.g. stubs, test harness)
availability of test hardware (e.g. emulator)
Test Steps:
Description how to set the test object (e.g. module, function) in a test relevant state
Description of Inputs (if necessary with input sequence and timing)
Input File

Expected
Result

Copyright 2010 Eberhard De Wille

Description / Comment

Page 34

SW Testing for Microcontroller Applications


Test Case Specification other possible Test Steps
Test Steps: (Example)
Connect the CANAlyzer to the bus where the control unit is connected
Load the configuration file test-01.cfg to configure the CANAlyzer
Run the CANAlyzer so that it transmits the configured CAN messages
Check for the transmission of CAN frames $102 and $132 by the CANAlyzer
Reset the control unit and re-start it
Check in the INCA diagnostic system for the expected results
Expected Results: (Example)
Variable

Hex-value

Represents

DP_ub_VehSpeed

0xA3

185 km/h

DP_ub_VehAcc

0x12

0.5 m/s

For manually executed tests the Test Case Specification can be also used as Test Log
Copyright 2010 Eberhard De Wille

Page 35

SW Testing for Microcontroller Applications

Test log
Test recording
Documentation of detailed results
Data for each test case
are recorded
Manual (log) or
Automatic (recording)

Copyright 2010 Eberhard De Wille

Test Reporting
Identification of test
objects
Identification of used
test specification
Document the results in
condensed form
Clear statement of
passed or failed

Trend

Page 36

SW Testing for Microcontroller Applications


Test Documentation Mapping (recommendation)
IEEE

Recommended Solution

Test Plan

Test Plan (generic)


Test Schedule

Test Design Specification


Test Procedure Specification
Test Case Specification

Test Specification

Test Item Transmittal Report

Release Note

Test Log
Test Incident Report
Test Summary Report

Test Report

Copyright 2010 Eberhard De Wille

Page 37

SW Testing for Microcontroller Applications

Defect Data Collection

Copyright 2010 Eberhard De Wille

Page 38

SW Testing for Microcontroller Applications

Collect your defect data to receive test end criteria if required (different
collection than the one for process improvement)
Collect your defect data to improve your development and testing process
Set up a database with search and filter functions (Excel may be sufficient)
Classify your bugs, sources of bugs, possible prevention, etc.
Design your reporting templates and test logs to support an easy defect
data collection
Name a person who is responsible to maintain the defect database!

Copyright 2010 Eberhard De Wille

Page 39

SW Testing for Microcontroller Applications


Defect Data Collection of the following data:

Where did the problem occur (portion of the code, interaction of components)
When did the problem occur (at an early test or inspection, at a later test, in
the field)
Why did the problem occur (what was the reason for the problem, what failed
and why did it fail, why was it not discovered previously)
How severe was the problem (range from formal to serious system failure)

How could the problem have been prevented earlier (what can be done to
improve the process to make it not happen again)

Copyright 2010 Eberhard De Wille

Page 40

SW Testing for Microcontroller Applications


Example of a possible database layout:
----- once per test object and test session
Test object identification (file name, version, VM label)
Date of Test
Name of Tester
Lines of code
Total number of bugs
Duration of test

----- repeated per bug


Description
Bug severity
Bug found in test #
Bug could have been prevented by #

Copyright 2010 Eberhard De Wille

Page 41

SW Testing for Microcontroller Applications


Example of possible classifications:
Bug severity:
1 = highly critical, eventually leads to a failure of the SW
e.g. real overflows, division by zero, access to array out of boundaries
2 = critical
overflow etc. will most likely not happen under normal operating conditions but a
potential problem may occur if defines or parameters change or limitations in other SW
parts (not under our control) change. I.e. currently not a real problem but SW is not
robust
3 = medium
e.g. problem related to design. It works currently, but may not be portable to other
CPUs/compilers, hardly testable code (e.g. too many inputs)
4 = low
things not according to design or coding standards but unlikely to create problems,
formal things like naming convention

Copyright 2010 Eberhard De Wille

Page 42

SW Testing for Microcontroller Applications

Example of possible classifications:

Bugs found in:

Bug could have been prevented by better:

1 = Requirements Inspection / review


2 = Design Inspection / review
3 = Code inspection / review
4 = PC-Lint
5 = Module / Component tests
6 = Integration tests
7 = System tests
8 = Customer System tests
9 = Field
10 = Other means (add comment)

0 = Needs further analysis / discussion


1 = Requirements engineering
2 = High Level Design
3 = Module design
4 = Coding
5 = Test definition of earlier tests
6 = Test conduct of earlier tests

Copyright 2010 Eberhard De Wille

Page 43

SW Testing for Microcontroller Applications

Test Organization

Copyright 2010 Eberhard De Wille

Page 44

SW Testing for Microcontroller Applications

Why should you have a Test Team / Department ?


Better adherence to the defined process
Independent institution
Concentrated know how
Uniform document outputs
Best results with least effort
Constant watch and improvement of the process

IEEE 1012 p.90 Testing by a separate instance not by the developer!


Copyright 2010 Eberhard De Wille

Page 45

SW Testing for Microcontroller Applications

Setup of the Test Team


Project Leader
product lifecycle
main schedule
test inputs

Test Team Members


test conduct
support of test process

Test Team Leader


effort estimation
test planning
test process
supervision of team
test conduct

Copyright 2010 Eberhard De Wille

Page 46

SW Testing for Microcontroller Applications

State of the Art


Testing Techniques

Copyright 2010 Eberhard De Wille

Page 47

SW Testing for Microcontroller Applications

White-Box
Testing Techniques

Copyright 2010 Eberhard De Wille

Page 48

SW Testing for Microcontroller Applications


Overview of White-Box Test Coverage
C0 All statements in a module are executed.
C1 All branches in a module are executed at least once.
C1+ All branches in a module are executed at least once and loops are run with their
min and max values.
C1p All branches in a module are executed at least once and all logical conditions
are covered at least once.
C2

C0, C1, C1+ and C1p have to be covered 100% plus a combination of the contained
loops has to be achieved: 1. Loop not executed, 2. Loop executed with a low counter
value, 3. Loop executed with a high counter value.

Cik C0, C1, C1+, C1p and C2 have to be covered 100% plus the test of the contained
loops has to be achieved: i = 1,2,3...k.
Ct The combination of all possible paths through the test objectis covered by white-box tests
Source: Georg Erwin Thaller (2002), Software-Test, Verification and Validation, Heise Verlag
Copyright 2010 Eberhard De Wille

Page 49

SW Testing for Microcontroller Applications


C0 coverage - Statement Testing
Basic White-Box testing technique: for each test case, specify:
the inputs of the test object,
identification of statements to be executed by the test case,
the expected outcome of the test case
Test completeness criteria: achievement of the test coverage e.g. 100% of the statements.
Application: Module test, especially for new or strongly modified software.
Comment: More effective for unstructured (e.g. Assembler) than for structured (C) software.

Copyright 2010 Eberhard De Wille

Page 50

SW Testing for Microcontroller Applications

C0 Statement coverage each program line will be executed at least once

if (a == 1)
{
/* statement coverage tests the following line */
b = c * x;
}
else
{
/* this branch will not be tested by statement coverage! */
}

Copyright 2010 Eberhard De Wille

Page 51

SW Testing for Microcontroller Applications


C1 coverage Branch or Decision Coverage
Basic White-Box testing technique:
It is designed in order to execute each outcome of all or selected decision points in a test
object. It also applies for loops. For each test case, specify:
the inputs of the component,
identification of decision outcomes to be executed by the test case,
the expected outcome of the test case.
Test completeness criteria: achievement of the
test coverage e.g. 100% of the statements.
Application: Module test, especially for new or
strongly modified software.
Comment: Refinement of statement testing.
Permit to determine if a branch is reachable
or not (decision outcome not achievable).

Copyright 2010 Eberhard De Wille

Page 52

SW Testing for Microcontroller Applications


C1 Branch coverage: each branch will be run through at least once
if (a == 1)
{
/* C0 statement coverage tests the following line */
b = c * x;
}
else
{
/* with C1 coverage now this branch will be tested as well */
}

for (a = 0; a < b; a++)


{
/* for C0 and C1 one test case to enter the loop */
b = c * x;
// more code lines
}
/* for C1 one test case to skip the loop, e.g. b = 0 */

Copyright 2010 Eberhard De Wille

Page 53

SW Testing for Microcontroller Applications


C1+ Branch coverage: additional test cases compared to C1
for (a = 0; a < b; a++)
{
/* for C0 and C1 one test case to enter the loop */
b = c * x;
// more code lines
}
/* for C1 one test case to skip the loop, e.g. b = 0 */
Required test cases for C1+:
b = 0
b = 1
b = maximum specified value (if known), otherwise use the maximum of
the data type

Copyright 2010 Eberhard De Wille

Page 54

SW Testing for Microcontroller Applications


C1p Condition coverage:
Condition coverage means that every condition which will result in the run through a
branch has to be tested.
if (a == 1 || z == 0 || y == 3)
{
/* With condition coverage the following line will be executed */
/* once for a=1 , once for z=0 and a third time for y=3 */
b = c * x;
}
else
{
/* with branch coverage now this branch will be tested as well */
}
Test Case 1

a != 1

z != 0

y != 3

Test Case 2

a=1

z != 0

y != 3

Test Case 3

a != 1

z=0

y != 3

Test Case 4

a != 1

z != 0

y=3

Copyright 2010 Eberhard De Wille

Page 55

SW Testing for Microcontroller Applications


Using multiple condition coverage means to check all possible combinations of the
conditions.
Test Case 1

a != 1

z != 0

y != 3

Test Case 2

a=1

z != 0

y != 3

Test Case 3

a != 1

z=0

y != 3

Test Case 4

a=1

z=0

y != 3

Test Case 5

a != 1

z != 0

y=3

Test Case 6

a=1

z != 0

y=3

Test Case 7

a != 1

z=0

y=3

Test Case 8

a=1

z=0

y=3

This is an extension of the C1p coverage, but not required to achieve C1p.

Copyright 2010 Eberhard De Wille

Page 56

SW Testing for Microcontroller Applications


C2 Basic Path Coverage
Advanced White-Box testing technique:
Path testing (transaction flow testing) uses a model of the systems processing (the control
flow graph), and intends to exercise transaction flows. For each test case, specify:
the inputs of the component,
identification of decision outcomes to be executed by the test case,
the expected outcome of the test case.
Test completeness criteria: achievement of the test coverage: e.g., 100% coverage of the
transaction flows .
Application: Functional SW Validation.
Comment: Test execution should be automated from the start e.g., via capture/playback-tool.
Required for safety relevant systems of SIL3 and higher.

Copyright 2010 Eberhard De Wille

Page 57

SW Testing for Microcontroller Applications


C2 Basic Path Coverage
All other coverage (C0, C1, C1+, C1p) has to be achieved
Plus in C2 now is the focus on the combination of loops.
for (a = 0; a < b; a++)
{
// more code lines
}
for (c = 5; c < x; c++)
{
for (d = 0; d < y; d++)
{
// more code lines
}
}

1.Each loop not entered


2.Each loop run with min value
3.Each loop run with max value
4.Combine all possibilities!!

Loop 1

Loop 2

Loop 3

b=0

x=5

n.r.

b=0

x=6

y=0

b=0

x=6

y=1

b=0

x=6

y = max

b=0

x = max

y=1

b=0

x = max

y = max

b=1

x=5

n.r.

b=1

x=6

y=0

b=1

x=6

y=1

b=1

x=6

y = max

b=1

x = max

y=1

b=1

x = max

y = max

cont. with b = max


Copyright 2010 Eberhard De Wille

Page 58

SW Testing for Microcontroller Applications


Cik Basic Path Coverage with exhaustive loop testing
All other coverage (C0, C1, C1+, C1p, C2) has to be achieved
Plus in Cik now is the focus on the use of loops with all values in range.

for (a = 0; a < b; a++)


{
// more code lines
}
for (c = 5; c < x; c++)
{
for (d = 0; d < y; d++)
{
// more code lines
}
}

Copyright 2010 Eberhard De Wille

1.Each loop not entered


2.Each loop run with all posssible
values from min to max
3.Combine all possibilities!!

Page 59

SW Testing for Microcontroller Applications


Ct Total Path Coverage
Using total path coverage means that all possible combinations of paths through the
software are tested, including all logical combinations and all combinations for loops.
if (a < 5)

if (a < 5)
/* then */
b = 30;

/* else */
b = 15;

/* then */
b = 30;

/* else */
b = 15;

if (a == 1 || x == 5 || y > 3)

if (a == 1 || x == 5 || y > 3)

/* then */
b = a * c;

/* then */
b = a * c;

/* else */
/* do nothing */

if (a < 5)

if (a < 5)
/* then */
b = 30;

/* else */
b = 15;

if (a == 1 || x == 5 || y > 3)
/* then */
b = a * c;

/* else */
/* do nothing */

/* else */
/* do nothing */

Copyright 2010 Eberhard De Wille

/* then */
b = 30;

/* else */
b = 15;

if (a == 1 || x == 5 || y > 3)
/* then */
b = a * c;

/* else */
/* do nothing */

Page 60

SW Testing for Microcontroller Applications


White-Box Testing Techniques: State transition testing
Definition and objectives: State transition testing uses a model of the states the
component may occupy, the transitions between those states, the events which cause
those transitions, and the actions which may result from those transitions.
For each test case specify:
the starting state
the input of the component
the expected output from the component
the expected final state

For each transition within a test case specify:


the starting state of the component
the expected next state
the event which cause transition to the next state
the expected action caused by the transition

Copyright 2010 Eberhard De Wille

Page 61

SW Testing for Microcontroller Applications


White-Box Testing Techniques: State transition testing
Test completeness criteria: achievement of the test coverage: e.g., 100% coverage of
the state transition diagram
Application: Component and system testing. However it only applies in cases where clear
states and transitions within a system or component can be identified.
Comments: Best pay-off in the early phase of design. Not an effective technique for very
small (<=4) or big (>256) state graphs.
Transition From State

To State

Condition (derive test cases from here!)

t1

Off

Init

Power up of System, uC reset

t2

Init

Run

Init was run successful

t3

Run

Run

All diagnostics are o.k.

t4

Run

Init

Diagnostic failed, SW Watchdog

t5

Run

Power Dn

KL15 down

t6

Power Dn

Init

SW Watchdog, diagnostic failed

t7

Power Dn

Off

All power down activities successful

t8

Init

Init

Startup diagnostic failed

Copyright 2010 Eberhard De Wille

Page 62

SW Testing for Microcontroller Applications


White-Box Testing Techniques: Data flow testing
Definition and objectives: Data flow testing uses a model of the interactions between
parts of a component connected by the flow of data as well as the flow of control.
A data flow is an abstract representation of the possible changes of data objects
(creation or assignment: define, usage: reference, uninitialized/lifetime ended: undefined).
Test cases shall be designed to execute control flow paths between definitions and uses of
the variables in the components. Data usage occurs in calculation or predicate.

For each test case, specify:

Possible anomaly pattern:

the inputs of the component

ur reference without init

location of relevant variable definition, use


and rendering invalid,

dd written multiple times before referencing

control flow sub-paths to be checked

du written but never referenced


rd using an old value (in a cyclic task context)

the expected outcome of the test case.

Copyright 2010 Eberhard De Wille

Page 63

SW Testing for Microcontroller Applications


White-Box Testing Techniques: Data flow testing

Copyright 2010 Eberhard De Wille

Page 64

SW Testing for Microcontroller Applications


White-Box Testing Techniques: Data flow testing
Test completeness criteria: achievement of the test coverage e.g. all uses of a variable.
Application: Mainly in unit test. For modules with static or global data.
Comments:
Depending on the data flow a refinement of branch/decision testing can be performed.
Effectiveness rapidly deteriorates as the size of the software under test increases.
Use tool support with the ability to visualize the use of data.
The implementation of strictly object oriented design makes this test easier because the
scope is only module wide.
Should be a mandatory test executed with diligence if global variables are used
across several modules.

Copyright 2010 Eberhard De Wille

Page 65

SW Testing for Microcontroller Applications

Black-Box
Testing Techniques

Copyright 2010 Eberhard De Wille

Page 66

SW Testing for Microcontroller Applications


Black-Box Testing Techniques: Equivalence Partitioning
Definition and objectives:

Equivalence partitioning has the aim to reduce test cases to the really useful test
cases.

Equivalence partitioning has the aim to ensure that test cases have been selected
which cover also unexpected inputs.

Test Theory says: A partition contains a set or range of values of which can be
reasonably expected to be treated by the component in the same way (i.e. they may
be considered equivalent).

Both valid and invalid values are partitioned in this way.

The input and output values are derived from the specification of the components
behavior.

For components with many input parameters it might be useful to select appropriate
combinations of values for each parameter partition. I.e., to test with combinations of
inputs that interfere with each other.
Copyright 2010 Eberhard De Wille

Page 67

SW Testing for Microcontroller Applications


Black-Box Testing Techniques: Equivalence Partitioning
Example:
Input: day of a month (integer value)
1 >=

day

<= 31

-4

15
1

Invalid partition < 1


Dirty Test Cases

43
31

valid partition

Copyright 2010 Eberhard De Wille

Invalid partition > 31


Dirty Test Cases

Page 68

SW Testing for Microcontroller Applications


Black-Box Testing Techniques: Equivalence Partitioning
Recommended test completeness criteria:
1. Achievement of C1 coverage.
2. Test at least one input/output pair from each equivalence partition.
3. If a component has multiple inputs a permutation of the test cases is
recommended.
Application:
1. Recommended for all test levels from module test to big components.
2. It is the major black-box testing technique.
Comments:
1. Equivalence partitioning significantly reduces the number of conditions (input and
output) to be tested.
2. One test case out of each partition is sufficient !
3. Unexperienced testers would test 1..31 and forget the dirty test cases
4. Equivalence partitioning and boundary value analysis supplement each other
and have to be applied at the same time!

Copyright 2010 Eberhard De Wille

Page 69

SW Testing for Microcontroller Applications


Black-Box Testing Techniques: Boundary Value Analysis
Example:
Input: day of a month (integer value)
1 >=

day

<= 31

-4

0 1

15

Invalid partition < 1


Dirty Test Cases

31 32

43

31

valid partition

Copyright 2010 Eberhard De Wille

Invalid partition > 31


Dirty Test Cases

Page 70

SW Testing for Microcontroller Applications


Black-Box Testing Techniques: Equivalence Partitioning
Recommended test completeness criteria:
1. Boundary-value analysis that partitions the input and output values of the
component into a number of ordered sets with identifiable boundaries.
2. Boundary-value analysis is variant and refinement of equivalence partitioning for
which each edge of an equivalence class has to be tested.
Application:
1. Recommended for all test levels from module test to big components.
2. It is a major black-box testing technique which always has to be combined with
equivalence partitioning.
Comments:
1. The natural limits of data types e.g. 0 and 255 for an "unsigned char" have also
to be regarded as boundaries.
2. For signed data types the values around zero i.e. -1, 0, +1 have to be considered
as boundaries. Test cases for these boundaries have to be set up as well.

Copyright 2010 Eberhard De Wille

Page 71

SW Testing for Microcontroller Applications

Integration
Testing Techniques

Copyright 2010 Eberhard De Wille

Page 72

SW Testing for Microcontroller Applications


Integration Testing Techniques: Interface testing
Definition and objectives:
Interface testing checks the inter-component interfaces in order to verify the
consistency of its implementation.
The easiest solution is to apply SW inspection.
Functional tests (equivalence partitioning and boundary value analysis) can be
specified in order to exercise:

the data range between components,


the type compatibility,
the number and order of parameters,
the data transfer method
In case of dynamic tests you should use the same test cases to check both
components which have to be integrated.

Copyright 2010 Eberhard De Wille

Page 73

SW Testing for Microcontroller Applications


Integration Testing Techniques: Interface testing

Test completeness criteria: achievement of 100% coverage of all partitions and


boundaries.
Application: general SW Integration.
Comments:
1. Strongly typed languages e.g. ADA or Java prevent from many of the errors found by
the method.
2. Tests should be automated and used as regression tests.
3. This kind of test is usually also used as Acceptance Test for third party software

Copyright 2010 Eberhard De Wille

Page 74

SW Testing for Microcontroller Applications


Integration Testing Techniques: Memory testing
Definition and objectives:
Memory testing aims at detecting:

memory problems,
overwriting and reading outside array bounds,
memory allocated but not freed, (usually no problem in embedded systems)
reading and using not initialized memory.
Test completeness criteria: achievement of 100% requirements coverage.
Application: All test level but esp. SW Validation, System Integration
Comments: Applied methods are inspection for out of bounds access and un-initialized
memory (variables). Alternatively tools like PolySpace can be used. For un-initialized
variables Perl scripts can be used. Un-freed memory can be detected with DevPartner.

Copyright 2010 Eberhard De Wille

Page 75

SW Testing for Microcontroller Applications


Integration Testing Techniques: Real time testing
Definition and objectives:
Real time testing aims to check that:

The different real time tasks are executed in the specified and designed order
(chronology rule)
That tasks that exclude each other are not executed at the same time (exclusion rule)
That tasks are synchronized in the specified way (synchronization rule)
That tasks are executed with the specified priorities (priority rule)
That tasks are not delayed
That tasks do not overrun
That RAM usage fits the available RAM (stack, C-stack and if applicable heap)
Processor load is acceptable
Worst case load scenarios are considered

Copyright 2010 Eberhard De Wille

Page 76

SW Testing for Microcontroller Applications


Integration Testing Techniques: Real time testing

Test completeness criteria: No particular criteria


Application: Essential test for embedded software
Comments:
1. Should be done with measuring methods during runtime (e.g. stack supervision,
task runtime and overall system load
2. Special load situations and task behavior has to be checked with test cases in the
laboratory

Copyright 2010 Eberhard De Wille

Page 77

SW Testing for Microcontroller Applications

Static White-Box Tests

Copyright 2010 Eberhard De Wille

Page 78

SW Testing for Microcontroller Applications


Static White-Box Tests:
Precondition: Inspected SW-Design Document
Precondition: Source Code compiles and is free of warnings
Precondition: Static Checking tool was applied on daily basis
Precondition: Code Reviews have been performed (functional focus)
Performing the Static White-Box Test:
Run the static checking tool (PC-Lint) and document the results
Perform a Code Inspection and document the results

Copyright 2010 Eberhard De Wille

Page 79

SW Testing for Microcontroller Applications


Static White-Box Tests: - Automatic checking tools
Use tools prior to inspections to reduce the number of faults. E.g.
PC-Lint
PC-Lint with MISRA checker
QA-C or QA-C++
QA-C / MISRA

Use tools to support inspections E.g.


CodeSurfer
SNIFF+
Proprietary Perl Scripts

Copyright 2010 Eberhard De Wille

Page 80

SW Testing for Microcontroller Applications


Static White-Box Tests: - Why should you use automatic checkers?
Example of a program for the C164: What is going wrong?
int main(void)
{
unsigned char x;
x = ~0xFF;

The ~ will promote the constant to 00FF


then the bits are turned to FF00
at last the value is cast to 00
00 is promoted to 0000
0xFF is promoted to 00FF and complemented to FF00

if (x == ~0xFF)
{
printf("I should come out here\n\n");
}
else
{
printf("I should not be here\n\n");
}
getchar();

0000 == FF00

Copyright 2010 Eberhard De Wille

Page 81

SW Testing for Microcontroller Applications


Static White-Box Tests: - Why should you use automatic checkers?
Example: What is going wrong?
int main(void)
{
unsigned char x;
unsigned short y;
y = 1000;

The ++ will promote x to 16 bit


then the value is incremented
at last the value is cast back to 8 bit
Thus the value can never be bigger than 255

for(x=0; x<=y; x++)


{
if (x == 199)
{
printf("I should exit soon but
don't\n\n");
}
}
}

255 <= 1000 will never end the loop

Copyright 2010 Eberhard De Wille

Page 82

SW Testing for Microcontroller Applications


Static White-Box Tests: - Examples of PC-Lint checks - configuration
-e783
-e818
-e956

// Line does not end with new-line


// Pointer parameter could be declared as pointing to const
// Non Const, non volatile static or external variable

//global
+e900
+e914
+e916

message enabling
// Always produce an output message
// Implicit adjustment of function return value from type to type
// Implicit pointer assignment conversion

Copyright 2010 Eberhard De Wille

Page 83

SW Testing for Microcontroller Applications


Static White-Box Tests: - Examples of PC-Lint checks - output
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 377: Error -> Warning 639 Strong
type mismatch for type 'T_ULONG' in binary operation
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 377: Error -> Warning 639 Strong
type mismatch for type 'T_SLONG' in binary operation
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 377: Error -> Note 912 Implicit
binary conversion from long to unsigned long
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 379: Error -> Note 912 Implicit
binary conversion from int to unsigned int
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 421: Error -> Note 912 Implicit
binary conversion from int to unsigned int
C:\Lint_testing\test_objects_lib\AAL_gCLib.c 422: Error -> Note 912 Implicit
binary conversion from int to unsigned int

Copyright 2010 Eberhard De Wille

Page 84

SW Testing for Microcontroller Applications


Static White-Box Tests: - Examples of PC-Lint checks problem and solution
Version with bugs:
377 uw_Temp = (T_UWORD)(((((T_ULONG)uw_nom<<1)+1) *
(T_SLONG) AL_uw_Lookup [ (T_UBYTE)uw_div ]) >> 17);
378
379 if (uw_Sign == 1)

Clean version:
377 uw_Temp = (T_UWORD)(((((T_ULONG)uw_nom<<1)+(T_ULONG)1) *
(T_ULONG) AL_uw_Lookup [ (T_UBYTE)uw_div ]) >> 17);
378
379 if (uw_Sign == 1u)

Copyright 2010 Eberhard De Wille

Page 85

SW Testing for Microcontroller Applications


Static White-Box Tests: - PC-Lint MISRA extentions

Motor Industry Software Reliability Association


Free consortium of members of the automotive industry
Deals with the known language insecurities and common programmer errors
Deals with the known problems of the C - standard
Aims to define a subset of C for safety related microcontroller applications
SIL 2 and SIL 3
Defines proprietary coding guidelines for licensees
Tools like PC-Lint and QA-MISRA support the automatic check of SW

Copyright 2010 Eberhard De Wille

Page 86

SW Testing for Microcontroller Applications


Static White-Box Tests: Inspection, Review, Walk trough and other checks
Confusing!: Various words are commonly used, everybody has a different
understanding of it, but only a few methods are really defined and make sense.
The following terms and concepts are known:
Document Check
4 Eye Inspection
Walk trough defined (IEEE 1028)!
Review (object review or document review but also milestone review)
Technical review defined (IEEE 1028)!
Management review defined (IEEE 1028)!
Inspection (also called SW or Fagan Inspection) defined (IEEE 1028)!
Various derivates of technical reviews and inspections

Copyright 2010 Eberhard De Wille

Page 87

SW Testing for Microcontroller Applications


Static White-Box Tests: - Definitions
Document Check: a document or other work product is simply checked by
another person at his desk. A feedback is given to the author.
4 Eye Inspection: a document or other work product is checked by the author
and another expert at his desk.
Walktrough: is an educational method to increase awareness. In a meeting the
work product is read and presented to the participants. IEEE 1028 says:
"The purpose of a systematic walk-through is to evaluate a SW product. A SW
walk-trough may be held for the purpose of education of an audience regarding
a software product."
Review: is a method to improve general distribution and generate agreement. It
may be almost the same as a walk through. Another variant is to ask each
participants to prepare and then collect only comments in the meeting.
All these methods may use check lists and some form of documentation of the
activity.
Copyright 2010 Eberhard De Wille

Page 88

SW Testing for Microcontroller Applications


Static White-Box Tests: - Definitions
Inspection: is a statistically controlled procedure whos output is a document
of measurable quality.

review

walkthrough

document

inspection

Document of
measurable quality

The procedure is defined


Defined and mandatory participants
Checklists are defined and defect data are collected
Copyright 2010 Eberhard De Wille

Page 89

SW Testing for Microcontroller Applications


Static White-Box Tests: - Inspection Facts

Source: Michael Fagan, July 1986, Advances in Software Inspecton, IEEE Trans. Software Engineering, Vol12, No 7, p745
Copyright 2010 Eberhard De Wille

Page 90

SW Testing for Microcontroller Applications


Static White-Box Tests: - Inspection Facts

Source: Wheeler David, 1996, Software Inspection: an industy best practice, IEEE Computer Society Press, p7
Copyright 2010 Eberhard De Wille

Page 91

SW Testing for Microcontroller Applications


Static White-Box Tests: - The Inspection Process according to IEEE 1028 (The
same as a Fagan Inspection)
Author assembles material
Work Product,
Standards, Specs

1. Planning

Educate inspectors
conducted by moderator
presented by author

2. Overview
Checklist

Get familiar with work product


use checklist

3. Preparation

4. Examination /
Meeting /
Inspection

Report
Defect List

5. Rework

Moderator ensures entry criteria met


Select 3-5 people & their roles
Schedule & distribute material

Defect
Summary

check if everybody is prepared


reader presents work product
defect detection no discussion of solutions
record and classify defects
max. 2 hours
decision (accept, verify rework, re-inspect)
by author
Moderator verifies rework

6. Follow-up

Copyright 2010 Eberhard De Wille

Page 92

SW Testing for Microcontroller Applications


Static White-Box Tests: - The Inspection Process according to IEEE 1028 (The
same as a Fagan Inspection)

Role

Description

Moderator

Chief planner, meeting manager, and reporter

Recorder

Document defects at examination, may also be moderator

Reader

Present work product

Tester

Examine for test issues

Other

Examine for special issues

Author

Developer of the work product. Must not assume any other role

Copyright 2010 Eberhard De Wille

Page 93

SW Testing for Microcontroller Applications


Static White-Box Tests: - Technical Review according to IEEE 1028
Comparison between Inspection and Technical Review:
Issue

Technical Review

Inspection

Objective

Determine the suitability of a work product


for its intended use.

Determine the suitability of a work product for


it's intended use, but beyond that search for
anomalies by examination through educated
inspectors.

Roles

A minimum of two persons is required.


Each one of them can assume multiple
roles. Since the scope is different
additional persons e.g. management or
customer representative can participate,
but this is not regarded as a role in the
process.

Additionally required roles are the Author and


the Reader. The roles are explicitly separated
and can not be assumed by one person.

The inputs are very similar than the ones of


the inspection. It has to be observed that
checklists are not mentioned i.e. not
required.

Additional inputs are: Inspection reporting


forms, Inspection checklists, Hardware
product specifications, Hardware performance
data. Some of these inputs are optional. The
reporting form is mandatory as well as the
inspection checklist. The wording of the
standard would suggest that checklists are not
mandatory, however a table in the appendix of
the standard makes them mandatory.

Input

Copyright 2010 Eberhard De Wille

Page 94

SW Testing for Microcontroller Applications


Static White-Box Tests: - Technical Review according to IEEE 1028
Comparison between Inspection and Technical Review:
Issue

Output

Entry Criteria
For the
Meeting

Meeting

Outcome of the
Meeting

Technical Review

Inspection

The only output is an Action Item List


and a record (meeting minutes) of the
technical review.

The outputs are a formal inspection report, a


formal defect summary and a defect list with
classified defects. The emphasis is on
providing a standard output of found errors
which
also
would
allow
statistical
evaluations.

No special criteria mentioned.

It is explicitly stated that the meeting has to


be re-scheduled if the inspection leader
finds that the participants are not well
prepared.

Keep the described rules of a review


meeting.

The defined roles have to be explicitly kept.


The reader and not the author will present
the material. The other roles also have to be
formally kept.

Generate of the defined review report.

At the end of the inspection the decision has


to be made to accept the work product,
close the inspection, but require a rework,
reject the work product and require a reinspection after the rework is done.

Copyright 2010 Eberhard De Wille

Page 95

SW Testing for Microcontroller Applications


Static White-Box Tests: Practical Problems with Inspections
It is difficult to get 6+ people who fulfill the "requirements" to attend in
numerous meetings (we are not in 1976 any more, time pressure is high
an development teams are global)
The formalism is heavy (although needed to be classified in higher CMMI
or SPICE levels). However, the formalism does not guarantee that bugs are
detected!
The preparation period is pretty vague and the checklists may not cover
everything that is necessary (too general). However, this activity IS THE
BUG FINDING ACTIVITY!
Recommendation: emphasize the "preparation period" and make it a
specified activity with clearly expected work steps.
Recommendation: question the formalism and reduce it to a level that can
be lived.
Copyright 2010 Eberhard De Wille

Page 96

SW Testing for Microcontroller Applications


Static White-Box Tests: - The Adapted Inspection Process

1. Planning

Work Product,
Standards, Specs

Inspectors get support from the


author and can ask questions any time

2. Overview
3. Inspection

Detailed
specification
Report

4. Problem
resolution
meeting

5. Rework

Author assembles material


Inspector ensures entry criteria met
use fixed staff
Schedule by test/inspection manager

Defect List

Defect
Summary

Action List

Check the work product


Use specification
record and classify defects
observe the general rules (LOC/h)
discuss findings with the author
decision on actions concerning
each defect (action list)
involve other project members if
necessary
by author
Handled by a problem resolution process
e.g. PVCS tracker

6. Follow-up

Copyright 2010 Eberhard De Wille

Page 97

SW Testing for Microcontroller Applications


Static White-Box Tests: - The Adapted Inspection Process

Role

Description

Test Manager

Schedules the work and arbitrates in case of upcoming problems


in the work flow

Inspectors

If possible have 2 independent inspectors. Check the work


product and document defects at examination.

Author

Developer of the work product. Must not assume any other role.
Is responsible to rework the work product

Copyright 2010 Eberhard De Wille

Page 98

SW Testing for Microcontroller Applications


Static White-Box Tests: Code inspections should build on automatic checks
1976 Code inspections inspected for:
Style transgressions
C - Standard transgressions
Dangerous use of the language (automatically detectable e.g. data type mix)
Dangerous use of the language (not detectable by a tool e.g. overflows)
Difference between requirements and behavior
Compliance to proprietary design and coding guidelines

Copyright 2010 Eberhard De Wille

Page 99

SW Testing for Microcontroller Applications


Static White-Box Tests: Code inspections should build on automatic checks

>2011 Automatic code checkers check for:


Style transgressions
C - Standard transgressions
Dangerous use of the language (automatically detectable e.g. data type mix)

>2011 Code inspections have to inspect for:


Dangerous use of the language (not detectable by a tool e.g. overflows)
Difference between requirements and behavior
Compliance to proprietary design and coding guidelines
The first 3 points can be detected by automatisms. Therefore apply them
prior to inspections. This reduces inspection time and increases the efficiency
of inspections.
Copyright 2010 Eberhard De Wille

Page 100

SW Testing for Microcontroller Applications


Static White-Box Tests: - Inspections
Inspections find faults not failures.
Inspections can be used for various purposes: Requirements, Design, Code,
other documents and specifications.
Inspections find approximately 80% of all faults. The total number
of faults that can be found is estimated around 96%
Slow down inspections otherwise they do not find anything! Stay < 100 LOC
per hour and < 400 LOC per day!
Defects
Detected

LOC / hour
100 LOC / hour
Copyright 2010 Eberhard De Wille

Page 101

SW Testing for Microcontroller Applications


Static White-Box Tests: - Code inspection areas

Data Check
Control Flow Check

Design

Data Flow Check

Calculation and
numeric check

Copyright 2010 Eberhard De Wille

Formal Aspects

Page 102

SW Testing for Microcontroller Applications


Static White-Box Tests: - Code inspection areas
Calculation and Numeric Check
Overflow / Underflow, signed / unsigned mixed calculation treated correctly?, division by zero,
array access range limits, etc.

Control Flow Check


pointer access (misalignment, null, lifetime), GOTO, endless loops,
recursive function calls, etc.

Data Check and Data Flow Check


data flow correct?, definition at correct place?, static vs. local vs. global variables,
variable names match the data type?, data init?, etc.

Design
nesting too deep?, too many or too less sub-functions, for every if an else,
include structure, interfaces, etc.

Formal Aspects / Other Issues


Naming conventions, special operators avoided?, coding guidelines adhered to?, etc.

Copyright 2010 Eberhard De Wille

Page 103

SW Testing for Microcontroller Applications


Static White-Box Tests: - Code inspection example of specification
Checked Item

How to perform the check

Is a calculation overflow /
underflow possible?

For every variable in a calculation check which could be


the maximum value which it possibly can take on. Then
use the worst case scenario (e.g. for the multiplication of
two variables take their highest possible value) and
check if the result is fitting in the result variable without
overflow. Do the same for negative values (underflow).
This applies to multiplications, additions, subtractions
and shifting operations.

Are critical parameters limited


in the init function?

In case the overflow / underflow check found a situation


where a fault could happen due to a certain parameter
setting this parameter has to be limited in an init function
of the cross compiled source code.

Is a division by zero possible?

Some CPUs have no problem with this. The result will be


simply the highest possible value of the data range.
Other CPUs crash. In order to be portable it has to be
made sure before the actual calculation takes place that
the divider is unequal to zero.

Copyright 2010 Eberhard De Wille

Page 104

SW Testing for Microcontroller Applications

Functional Tests
(Black Box Tests)

Copyright 2010 Eberhard De Wille

Page 105

SW Testing for Microcontroller Applications


Black-Box Tests: - principles

SW
component

Output - Interface

Input- Interface

Stimulation / Pre-set

Performance check

A component is any unit which can be tested from a functional point of view
Thus a component can be typically a module or a group of modules
up to the complete software
A component has defined interfaces at input and output
The requirements (functionality) of the interfaces has to be clearly defined

Copyright 2010 Eberhard De Wille

Page 106

SW Testing for Microcontroller Applications


Black-Box Tests: - principles
Black Box Tests are always dynamic tests. This means that they have
to be executed.
Executing the tests can be on a test bench or in the target system.
The interiors and structures of a component are disregarded
The tests are entirely focusing on the check of the behavior of a
component at its interfaces compared to the requirements specification
The classical functional test techniques have to be employed: i.e.
equivalence partitioning and boundary value analysis.

Copyright 2010 Eberhard De Wille

Page 107

SW Testing for Microcontroller Applications


Black-Box Tests: - principles
Specify and log / record and report the tests
Use a suitable test bench to perform the tests, as e.g. the Perl environment,
simulation tools, other commercial test tools (Test Director, Cantata, etc.)
Set up test scripts which allow you to perform regression testing
Try to keep the tests as long as possible free from using original hardware!
Your SW Design has to support this but using a HW Abstraction layer.
Modern Compilers do not rely on proprietary syntax for control register
access. They provide structures and defines for this. This enables you to
even test / substitute the HW Abstraction Layer in a test bench.

Copyright 2010 Eberhard De Wille

Page 108

SW Testing for Microcontroller Applications


Black-Box Tests: - Recommended Prerequisites

Check the matching of data types (best done in an inspection!)


Used data types have to match each other. Caution with compatible data types!
Check number and sequence of arguments (best done in an inspection!)
Caution with multiple arguments of the same data type or compatible types.

Copyright 2010 Eberhard De Wille

Page 109

SW Testing for Microcontroller Applications

Dynamic Testing
with a
Perl Testing Environment

Copyright 2010 Eberhard De Wille

Page 110

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment

Dynamic White-Box Tests


for certain portions of the
software.
Component tests using boundary
value analysis and equivalence
class partitioning.
Direct development of terms
and formulas in the Perl test
environment.

Copyright 2010 Eberhard De Wille

Page 111

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
The Perl part
#! c:/perl/bin/perl
use warnings;
use strict;
use Inline "C";
&test_start();
&test_001();
&test_002();
.
.
.

Enables the use of C subroutines i.e.


the test objects
Make Perl subroutines for easier test design

Make your script verbose to see what is going on

printf "DONE! \n";

__END__
__C__

Copyright 2010 Eberhard De Wille

Page 112

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
The Perl part
sub test_start {
my $uid = 0; my $year = 0; my $month = 0; my $day = 0; my $hour = 0; my $minute = 0;
$uid = getlogin();
$minute = (localtime(time()))[1];
$hour = (localtime(time()))[2];
$day = (localtime(time()))[3];
$month = (localtime(time()))[4] + 1;
$year = (localtime(time()))[5] + 1900;

Write the results to a log file

open(AUS, ">test-ws1.log");
printf AUS "** place some identifying text here ****************************\n";
printf AUS "** Date:
";
printf AUS ($year);
printf AUS ("-%02.0d",$month);
printf AUS ("-%02.0d \n",$day);
printf AUS "** Time:
";
printf AUS ("%02.0d",$hour);
printf AUS (":%02.0d \n",$minute);
printf AUS "** Username: ";
printf AUS ($uid,"\n");
printf AUS "\n\n";

Provide a standard date and time


stamp, including an user ID

Call a standard init function in the


C - part

calling C routine to initializte some important values


test_init();

Copyright 2010 Eberhard De Wille

Page 113

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment

Beware of the following restrictions due to the


Inline C at the interface of the DLL:
data types char (char, unsigned char, signed char) are not
handled correctly as arguments and return values (handled as
ASCII characters but not as integer numbers. Solution: make
wrapper functions which type cast it to short or unsigned short.
data types signed are not handled correctly as argument or
return value. The signed keyword has to be left away.
Solution: work with redefined data types as e.g. T_SWORD
instead of signed short and re-define it to short for the test
environment.

Copyright 2010 Eberhard De Wille

Page 114

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment

Beware of the following restrictions due to the


Inline C at the interface of the DLL:
the function definition has to observe:
T_UWORD CLASS_NAME function (void) only works if:
T_UWORD function ()
use wrappers if your interfaces do not comply with this:
T_UWORD CLASS_NAME function (void)
unsigned short Wrap_function ()
{
return function();
}
Copyright 2010 Eberhard De Wille

Page 115

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Definition of Global Values in the C-part
If the SW design of the test object uses global variables they have to be defined
in the C-part prior to the place where the test object will be placed
Necessary defines have to made here as e.g. to switch on a test mode in the SW

The C - Test Object


One of the main goals of this test method was to use the test object without
any changes as far as possible.
If function definitions do not comply (e.g. CLASS_NAME), use wrapper functions
It has to be observed that the test object has to be coded according to
ISO-C standards to make tests easy.
Make the code in a way that a test mode e.g. "#ifdef TEST" is supported to
enable the tests. E.g. making variables global for the test.

Copyright 2010 Eberhard De Wille

Page 116

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
The "test_init()" Function
This function is needed to set the SW preconditions for the test.
E.g. parameter structure (coming from an EEPROM) is set to defined
values. This "reading the values out of EEPROM" is substituted by this
function.

The Test Stubs


The test stubs are the sub-functions which are usually called by any of the
test object functions. They are not available because they are located in
other modules.
Thus they have to be supplied by the test environment.
If the test stubs have to return special values there have to be further functions
to set these values to their defined figure prior to the execution of the calling
function in the test object.
Copyright 2010 Eberhard De Wille

Page 117

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
The "get_" Functions
"get_" functions are the interface functions which enable the Perl test scripts
to read out data from the test object.
The recommended syntax for the naming of the interface functions is to use the
actual name of the accessed variable and the prefix get_ in front of it. Example:
unsigned short get_MOD_us_MyVar () {return (MOD_us_MyVar );}

The "set_" Functions


The "set_" functions are the interface functions which enable the Perl test
scripts to write data into the test object.
The syntax for the naming of the interface functions has to be the actual name
of the accessed variable and the prefix set_ in front of it. Example:
void set_MOD_us_MyVar ( unsigned short value) {MOD_us_MyVar

Copyright 2010 Eberhard De Wille

= value; }

Page 118

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Can be used for:
Development support (e.g. the output of CSV for the development of
a filter function)

Copyright 2010 Eberhard De Wille

Page 119

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Can be used for:
Permutation of input values by nested for loops e.g. for overflow testing
division by zero testing, interface and dirty testing, etc.
my @ARrsw_AccZ_array

= ($SSHORT_MIN-1, $SSHORT_MIN, $SSHORT_MIN+1, -1, 0, 1, .....

my @ARrsw_RollRate_array

= ($SSHORT_MIN-1, $SSHORT_MIN, $SSHORT_MIN+1, -1, 0, 1, .....

my @ARP_ARrb_sut_min_array

= (0, 255);

....
for (@ARP_ARrb_sut_min_array) {
$setval10 = $ARP_ARrb_sut_min_array [$i10];
$i10++; $i11 = 0;
for (@ARP_ARrb_sut_nom_array) {
$setval11 = $ARP_ARrb_sut_nom_array [$i11];
$i11++; $i12 = 0;
for (@ARP_ARrb_sut_max_array) {
$setval12 = $ARP_ARrb_sut_max_array [$i12];
$i12++; $cnt++;
&test_overflow_1_1();

Copyright 2010 Eberhard De Wille

Page 120

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Can be used for Regression Tests:
Comparison of an old program version with a new program version
especially for redesign and refactoring purposes.

Perl
Test Scripts
OLD
Test Object

log

log

Compare the results

Test Stubs,
etc.

Perl
Test Scripts
NEW
Test Object
Test Stubs,
etc.

Variant 1
Copyright 2010 Eberhard De Wille

Page 121

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Can be used for Regression Tests:
Comparison of an old program version with a new program version

Read and compare


output data

Perl
Test Scripts
OLD
Test Object

Log
differences

Feed with same


input data

NEW
Test Object
Test Stubs,
etc.

Variant 2
Copyright 2010 Eberhard De Wille

Page 122

SW Testing for Microcontroller Applications


Dynamic Testing Perl Script based test environment
Can be used for:
Individual test cases for the classical test techniques:

Branch / Decision Testing


Condition Coverage
Path Testing
Equivalence Partitioning (including Dirty Testing)
Boundary Value Analysis
Interface Testing / Acceptance Tests
State Transition Testing

Note: almost 95% of all detectable errors are detected while setting
up the tests! The execution of the test has more a formal meaning
and its biggest value is the possible regression testing.

Copyright 2010 Eberhard De Wille

Page 123

SW Testing for Microcontroller Applications

White - Box Testing


When is it needed?
What has to be observed?

Copyright 2010 Eberhard De Wille

Page 124

SW Testing for Microcontroller Applications


White Box Testing:
Older Testing Methodology was focusing heavily on Dynamic WhiteBox Testing as an own discipline, but:
It is very time consuming and the benefit is doubtful.
Test cases may be generated which may not reflect the real
life situation. I.e. test cases with input combinations that never
will happen.
If you need White-Box Tests most likely your requirements and
functional specification is not complete.

Copyright 2010 Eberhard De Wille

Page 125

SW Testing for Microcontroller Applications


White Box Testing:
The following White-Box Testing is recommended:
Static White-Box Testing: automatic code checks, reviews and
inspections (as already outlined).
Data Path Testing as a part of inspection, preferably with tool
support.
State Transition Testing: if state machines are contained in the SW
identify them from the design and code (they may not be reflected
in the requirements and other specifications), then execute them
with dynamic tests.
Perform other dynamic white-box tests as outlined in the following
pages.

Copyright 2010 Eberhard De Wille

Page 126

SW Testing for Microcontroller Applications


White Box Testing:
Use White Box Tests if code inspection indicates that they
need to be set up, e.g. when calculations are too complex to
easily cover them in an inspection.
Use White Box Testing Techniques as metrics for your
functional tests
The metric in focus is the C1 coverage.
All other metrics, e.g. C2 coverage makes not much sense, because
for applications where it is required nowadays model based
development is used with the ability to generate test cases
automatically and achieve the required coverage (SIL 4 applications).

Copyright 2010 Eberhard De Wille

Page 127

SW Testing for Microcontroller Applications


White - Box Testing C1 Code Coverage Analysis:
The use of a dedicated tool is recommended for automatic
instrumentation and measurement, e.g. DevPartner.
Test cases have to be defined on the go to achieve the
required coverage. I.e. by looking into the code to find the reasons
for missing coverage and defining more test cases.
Safety standards require certain coverage e.g. C0 + C1 for SIL2
and for SIL3.
To cover each branch and line gives assurance that there is not a
hidden pitfall in a rarely run through code parts.
To cover each branch constitutes a kind of review to find senseless
code portions.

Copyright 2010 Eberhard De Wille

Page 128

SW Testing for Microcontroller Applications


White Box Testing C1 Code Coverage Analysis:

Branch coverage each branch will be run through at least once

if (a == 1)
{
/* statement coverage tests
b = c * x;
bit_branch1 = 1; // special
}
else
{
/* with branch coverage now
bit_branch2 = 1; // special
}

the following line */


instrumentation

this branch will be tested as well */


instrumentation

To measure the C0 and C1 coverage a code instrumentation is


required.

Copyright 2010 Eberhard De Wille

Page 129

SW Testing for Microcontroller Applications

Integration Testing

Copyright 2010 Eberhard De Wille

Page 130

SW Testing for Microcontroller Applications


Integration Testing: General
Make a clear distinction between Software Integration and
Software Integration Testing
Make a clear distinction between System Integration and
System Integration Testing
Software Integration is the process to build an error free compiling SW
out of individual modules.
Software Integration Test is the functional test of components and
finally the integrated software against a specification. This is best done on a
test bench.
Software Integration Tests usually are done in several steps, each time testing
the next integration step, i.e. the next bigger integrated component

Copyright 2010 Eberhard De Wille

Page 131

SW Testing for Microcontroller Applications


Integration Testing: General

IEC 61508-3 (safety critical software) knows 3 types of system


integrations:
7.5.2.4 The specified integration tests for programmable electronics
(hardware and software) shall distinguish between the following activities:
a) merging of software system on to the target programmable electronic
hardware;
b) E/E/PE integration, i.e. adding interfaces such as sensors and actuators;
c) total integration of the EUC and E/E/PE safety-related system.

Copyright 2010 Eberhard De Wille

Page 132

SW Testing for Microcontroller Applications


Integration Testing: General
System Integration is the process to build a completed system, consisting
of all its required electrical, software and mechanical components.
System Integration Test is the functional test of component interactions in
a partially and finally in the fully integrated system against a specification.
System Integration Tests are usually very intense for the initial integration.
For subsequent integrations of unchanged components the regression tests
have to be used to reduce the effort and to guarantee the successful
integration.
System Integration Test has to be specified and its execution has to be
documented in a test report.

Copyright 2010 Eberhard De Wille

Page 133

SW Testing for Microcontroller Applications


Example of an airbag
software and system
integration

System Tests

Runtime
RAM Usage,
System behavior

Integration
Tests

SYSTEM

Component
Tests

Signal Chain
Complete SW
functionality

SW Integration
Tests

Component
Tests
ROLLOVER
ALGORITHM

Complete
SW

HW

Component
Tests

Component
Tests

Component
Tests

Component
Tests

SIDE
ALGORITHM

FRONT
ALGORITHM

OC / AWS
ALGORITHM

BASIC
SW

OS, HW
Abstraction Layer

Component
Tests

Data Preparation

Make an integration strategy! Define what to test


and integrate in which sequence!

Copyright 2010 Eberhard De Wille

Page 134

SW Testing for Microcontroller Applications


Integration Testing: Prerequisite a good SW design which allows easy
integration

Frame Work

Operating System

OS Abstraction Layer

Functional
Library
Function
Block 1

Function
Block 2

Function
Block 3
C
Library

Physical Layer
HW Abstraction Layer
Microcontroller Hardware
Copyright 2010 Eberhard De Wille

Page 135

SW Testing for Microcontroller Applications


Integration Testing: BottomUp component tests (Software Integration)
First of all perform an inspection of the interfaces to check if they match
the specification (data types, arguments) and if they match each other (for
internal interfaces between components). This is highly recommended for
first integrations for sub-sequent integrations with unchanged interfaces
it can be omitted.
Define testable sub-components and components and design the respective
component tests as regression tests.
Try to distinguish this from the system tests by using test-benches rather
than original hardware. E.g. because of easier accessibility of interfaces
and test results (test harness and test stubs)

Copyright 2010 Eberhard De Wille

Page 136

SW Testing for Microcontroller Applications


Integration Testing: Bottom up component tests (Software Integration!)
First test each individual component with a black-box test

Performance
check

Output - Interface

SW
component 2

Copyright 2010 Eberhard De Wille

Output - Interface

Input- Interface

SW
component 3

Input- Interface

Stimulation /
Pre-set

Performance
check

Output - Interface

SW
component 1

Input- Interface

Stimulation /
Pre-set

Stimulation /
Pre-set

Performance
check

Page 137

SW Testing for Microcontroller Applications


Integration Testing: BottomUp component tests (Software Integration!)
Next, test bigger components until the complete SW is integrated

Output - Interface

Output- Interface

Output - Interface

Input- Interface

SW
component 3

SW
component 2

Input- Interface

Output - Interface

SW
component 1

Input- Interface

Input- Interface

Stimulation /
Pre-set

Performance
check

SW component 4
Copyright 2010 Eberhard De Wille

Page 138

SW Testing for Microcontroller Applications


Integration Testing: Integration into hardware (System Integration)
Perform step by step integrations of Basic SW components into the
target hardware to check the overall system performance.

Actuation

SW
SW
comp2 comp1

Output - Interface

Rest of SW

Test in target

Input- Interface

Stimulation

Actuation

Output - Interface

SW
SW
comp2 comp1
comp1

Input- Interface

Stimulation

Target System

Target System

Finally integrate the pre-integrated and pre-tested SW which was


checked on the test bench.
Copyright 2010 Eberhard De Wille

Page 139

SW Testing for Microcontroller Applications


Integration Testing: Integration of the control unit with the rest of the System

This may be e.g. connection of real external sensors and test on a


special test bench.
This may be e.g. mounting the control unit and other external devices
e.g. sensors in the overall system e.g. a vehicle.
At this stage integration tests tend to be difficult and may be reduced to a
number of basic checks (e.g. if the sensors are connected correctly), before
the system tests are executed to validate the completely integrated system.
However, if system tests are not possible (e.g. rockets, oil platforms) or very
expensive (e.g. airbag systems), integration tests have to be continued as
far as possible .

Copyright 2010 Eberhard De Wille

Page 140

SW Testing for Microcontroller Applications

System Testing

Copyright 2010 Eberhard De Wille

Page 141

SW Testing for Microcontroller Applications


System Testing:
Perform the tests ONLY in the original environment.
The special challenge is to perform measurements in the original hardware
without influencing the system.
Each system requires a very specific test. No standardization is possible
and tools are proprietary and usually very expensive.
System tests are to a large degree for the purpose of validating the
original requirements. I.e. to check on the one hand if the requirements
were implemented correctly, but also to check if the correct requirements
have been defined.

Copyright 2010 Eberhard De Wille

Page 142

SW Testing for Microcontroller Applications

Recommendations for the


Application of Software Tests

Copyright 2010 Eberhard De Wille

Page 143

SW Testing for Microcontroller Applications


Recommended Test Documentation
1.

Test Plan: can be generic if you always perform the same testing
and reference the Test Schedule

2.

Test Schedule: time planning of your tests in a specific project

3.

Static Test Specification: will contain the test specifications for


automatic code checks (e.g. Lint messages enabled / disabled),
specifications (check lists) for code inspections. This can be generic if
tests are always performed the same for all projects.

4.

Static Test Report

5.

Dynamic White-Box Test Specification: This is optional and only filled


in if these tests are required for certain code (e.g. indicated by code
inspections)

Copyright 2010 Eberhard De Wille

Page 144

SW Testing for Microcontroller Applications


Recommended Test Documentation
1.

Dynamic White-Box Test Report: This is optional (see above)

2.

Functional Test Specification: this should specify all levels of


component tests up to the complete SW package.

3.

Functional Test Report

4.

Integration Test Specification: There may be several ones. E.g. one for
the integration of the SW into the target hardware. E.g. a second one
for system integration. Theses specifications are system integration
specifications. Software integration has to be covered by functional test
specifications.

5.

Integration Test Report

Copyright 2010 Eberhard De Wille

Page 145

SW Testing for Microcontroller Applications


Have an Added Value for your Coding Phase!
1.

Apply automatic code checkers on a daily basis during the coding


phase and fix the bugs immediately.

2.

Set up functional tests in parallel with implementing the source code


e,g, in the Perl testing environment. The code is only complete when
the tests are executed o.k.

3.

For the functional tests observe that you cover all equivalence
partitions and boundary checks.

4.

Make sure with an automatically instrumented code, that your


functional tests give you the required C1 coverage.

5.

Perform code reviews (with a focus on the functionality) after a


module is finished and tested o.k. by the functional tests (2).
Copyright 2010 Eberhard De Wille

Page 146

SW Testing for Microcontroller Applications


Make your Testing Phase Easy!
1.

Make sure that all source code to be tested fulfills the "added value"
requirements of the coding phase.

2.

First run the automatic code checker and cancel the testing in case
there are still bugs in the code. Restart testing after the bugs were
fixed. Documentation and entry in the defect database is not
required.

3.

In case the automatic checker was run o.k. you can formally
document it in the test report and proceed with a code inspection.

4.

The code inspection has to be formally documented and results have


to be entered in the defect database.

5.

During the inspection you have to determine if dynamic white-box


test are needed.
Copyright 2010 Eberhard De Wille

Page 147

SW Testing for Microcontroller Applications


Make your Testing Phase Easy!
1.

In case dynamic white box tests are indicated, now specify and
execute them, followed by filling in a test report.

2.

Collect the functional tests as performed during the coding phase.


Check the test cases for completeness and compliance with the
equivalence class and boundary value analysis rules. Document the
test cases in a functional test specification. Since the tests are coded
in test scripts a prose form of specification can be omitted. Use only
some general descriptions.

3.

Integration test specifications have to be set up by the SW and


system integrators in parallel with the coding phase.

4.

Integration tests have to be executed during integration and


reported in the appropriate report documents.

Copyright 2010 Eberhard De Wille

Page 148

SW Testing for Microcontroller Applications

SPICE Requirements
for Software Testing

Copyright 2010 Eberhard De Wille

Page 149

SW Testing for Microcontroller Applications


ENG. 6 Software Development
1.BP1 Develop verification methods for SW modules: This is covered by
the set up of a test plan where the methods for software verification have
to be described (static code analysis, reviews, inspections, functional test
e.g. with Perl testing environment.
2.BP2 Develop SW modules: Not in focus here.
3.BP3 Assure consistency: Not in focus here, but can be achieved by
reviews which check consistency between design and code.
4.BP4 Verify the SW modules: Ensure that the modules comply with
their requirements. This is covered by the already presented testing
methods (static checks, functional tests)

Copyright 2010 Eberhard De Wille

Page 150

SW Testing for Microcontroller Applications


ENG. 6 Software Development Work Products
1.Unit test plan: This is covered by the set up of a test plan and the test
schedule.
2.SW unit: Not in focus here. I.e. software modules.
3.Test incident report: Covered by the various test reports.
4.Test case specification: Covered by the various test specifications

Copyright 2010 Eberhard De Wille

Page 151

SW Testing for Microcontroller Applications


Introductory Note: SPICE states that for embedded systems the SW integration
may coincide with the integration into the target HW. Therefore integration tests
may be covered by pure SW component tests when performed on a test bench or
system integration test when performed on the target hardware.

ENG. 7 Software Integration


1.BP1 Develop a strategy for SW integration: The sub-components and
components and their sequence of integration and testing have to be
defined and documented in the functional test specification of the final
component (SW integration). For system integration this has to be done
in the system integration test specification
2.BP2 Develop tests for integrated SW units: This is covered by defining
functional tests for components and bigger components, ending with the
completely integrated SW as the last component. For system integration
this has to be documented in the integration test specification.

Copyright 2010 Eberhard De Wille

Page 152

SW Testing for Microcontroller Applications


ENG. 7 Software Integration
1.BP3 Integrate the SW units: This is part of the activity when bigger
components are build and made ready for test, or when they are step by
step integrated into the target hardware.
2.BP4 Test the integrated SW units: This is part of the activity when subcomponents and components are tested and test reports are generated
(test bench or target HW).
3.BP5 Assure consistency: This means that traceability between SW
design and SW units has to be proved. It can be achieved by review /
inspection where the consistency between design and code is checked.
There is hardly any other way.
4.BP6 Perform regression tests for integrated SW units: This is
implicitly covered by using an automated and script based testing
approach. For system integration also try to automate.
Copyright 2010 Eberhard De Wille

Page 153

SW Testing for Microcontroller Applications


ENG. 7 Software Integration Work Products
1.SW Integration Test Plan: E.g. a section at the beginning of the
functional test specification of the biggest (final) component, which
contains all integrated sub-components and at the beginning of the
system test specification.
2.Regression Test Plan: See above. Add a description about the sequence
in which the individual component tests have to be executed.
3.Test procedure: We called it "Test Specification" because it contains
the "Test Design Specification", the "Test Case Specification" and the
"Test Procedure Specification"
4.Test incident report: This is what we called "Test Report" because it
contains the Test Log, the Test Incident Report and the Test
Summary Report .

Copyright 2010 Eberhard De Wille

Page 154

SW Testing for Microcontroller Applications


ENG. 8 Software Test

1.BP1 Develop tests for the integrated SW product: This is covered by


the set up of functional tests and the integration tests.
2.BP2 Test the integrated SW product: This is simply the execution of
the defined tests and the reporting
3.BP3 Execute regression tests for the integrated software: This covered
by automating the tests and defining which ones are mandatory to be
executed repeatedly for new SW versions.

Copyright 2010 Eberhard De Wille

Page 155

SW Testing for Microcontroller Applications


ENG. 8 Software Test Work Products
1.SW test plan: This is covered by the set up of a test plan and the test
schedule.
2.Test incident report: Covered by the various test reports.
3.Test case specification: Covered by the various test specifications

Copyright 2010 Eberhard De Wille

Page 156

Você também pode gostar