Software Testing an enabler for Software

Quality Assurance

Abstract:

Today, the biggest challenge to software development organizations is to

ensure consistent delivery of quality software to the clients within the
estimated time and cost.

Organizations have come to realize that in order to expand globally and

profitably they must have quality product and services.

In order to control risk, time, cost and quality, organizations have to test
their software systems thoroughly to ensure that these systems fulfill
industry requirements and meet their customer’s needs. Thorough testing is
therefore beneficial to ensure that newly developed software systems
contain attributes of correctness, reliability, performance, security, usability
etc. The principal aim of testing is to detect faults so that they can be
removed before the product is made available to customers. Faults in
software are made for a variety of reasons, from misinterpreting the
requirements through to simple typing mistakes. It is the role of software
testing and quality assurance, to reduce those faults by identifying the
failures.Software Quality assurance and testing are an essential component
that produces software applications. Testing has thus become a serious
undertaking in most organizations. A well defined quality assurance
procedure that operates throughout the development cycle is the key to
software quality and cost management and to integrate this, implementation
of international standards can help companies reach their quality goals.

There are a set of International standards and frameworks which can

provide the software industry with the means to build quality systems into
their processes. Examples of such standards and frameworks include the ISO
9000 series , the CMM ,Six Sigma,CMMI, MBNQA,TickIT, SPICE etc.
Currently many nations are participating in significant consolidation and
coordination efforts to develop international standards and frameworks.

International standards thus contribute to increasing the reliability and

effectiveness of the products and services we used and produce.

Introduction:

What is Quality?

Software Quality
High quality software meets the needs of users while
being reliable, well supported, maintainable, portable,
and easily integrated with other tools.

Making changes to improve software quality requires time

and money to:

• Spot the problem

• Isolate its source

• Connect it to the real cause

• Fix the requirements, design, and code

• Test the fix for this problem

• Test the fix has not caused new problems

• Change the documentation

For a given change to make sense, the improvement needs to pay for all
these tasks, plus the revenue lost during the delay in the product
release.However, given a fixed amount of development time and money,
adding features adds bugs and reduces time for Testing.

Organizations often misunderstand that increasing quality is always more

expensive,in terms of total cost of production and maintenenance.But,infact,
quality is focused from start then:

• You tend to produce components with more (hidden)

defects, so

• You have to spend more time fixing these (late), so

• You have little time for anything else, so

• You produce poor quality software even though you

put huge amounts of effort into defect checking.

Thus quality is something that has to be considered

throughout the product lifecycle and it cannot be added in later.
 Quality Delays are Expensive

A
Design
0
Code
C
T
v
Review
Revie
o
e
g
d
sm
t
e
p
f
i
i
lx
e
t
i
m
e

Thus it makes sense to focus on improving component quality before testing,

to catch difficult defects early.
Testing:
[IEEE]

Testing is the process of exercising or evaluating a system of system

component by manual or automated means to verify that it satisfies specific
requirements.

Better Quality Through Testing?

Humphrey (2002) estimates that experienced software engineers normally

inject 100 or more defects per KLOC. Half of these are detected
automatically (e.g. by the compiler).So a 50 KLOC program contains around
2500 defects to be found manually. Suppose we need about five hours to
find each of these defects by testing. That’s over 20000 hours for the whole
program which is considered to be as a bad news(regarding the quality of
the software).

Better Quality via Standards?

Most products have safety standards, and many have usability standards,
but computer software rarely has such standards.

Can quality be improved by enforcing standards?

• It is very difficult to enforce standards on actual program behavior

• Standardizing the process can help make sure that no steps are skipped

•But, standardizing to an inappropriate process can reduce productivity, and

thus leave less time for quality.
Software Engineering Standards:

According to the IEEE Comp. Soc. Software Engineering Standards

Committee a standard can be:

• An object or measure of comparison that defines or represents the

magnitude of a unit

• A characterization that establishes allowable tolerances or constraints for

categories of items

• A degree or level of required excellence or attainment.

Why Bother with Standards?

• Prevents idiosyncrasy: e.g. Standards for primitives in

programming languages)

• Repeatability: e.g. repeating complex inspection processes.

• Consensus wisdom: e.g. Software metrics.

• Cross specialization: e.g. Software safety standards.

• Customer protection: e.g. Quality assurance standards.

• Professional discipline: e.g. V & V standards.

• Badging: e.g. Capability Maturity Model levels.

 Some Standards Organizations

ANSI:
American National Standards Institute.
Does not itself make standards but approves them.

AIAA:
American Institute of Aeronautics and Astronautics (e.g.
AIAA R-013-1992 Recommended Practice for Software Reliability).

EIA:
Electronic Industries Association (e.g. EIA/IS-632 Systems
Engineering)

IEC:
International Electrotechnical Commission (e.g. IEC 61508
Functional Safety - Safety-Related Systems)

IEEE:
Institute of Electrical and Electronics Engineers Computer Society
Software Engineering Standards Committee (e.g.IEEE Std 1228-1994
Standard for Software Safety Plans)

ISO:
International Organization for Standardization (e.g.ISO/IEC 2382-
7:1989 Vocabulary-Part 7: Computer Programming).
Quality Assurance
[NASA]

The function of software quality that assures that the standards, processes,
and procedures are appropriate for the project and are correctly
implemented.

What is QA?
QA refers to planned and systematic production processes that provide
confidence in a product's suitability for its intended purpose. It is a set of
activities intended to ensure that products (goods and/or services) satisfy
customer requirements in a systematic, reliable fashion. Two key principles
characterize QA: "fit for purpose" (the product should be suitable for the
intended purpose) and "right first time" (mistakes should be eliminated). QA
includes regulation of the quality of raw materials, assemblies, products and
components; services related to production; and management, production
and inspection processes.

When should QA testing start in a project? Why?

QA is involved in the project from the beginning. This helps the teams
communicate and understand the problems and concerns, also gives time to
set up the testing environment and configuration. On the other hand, actual
testing starts after the test plans are written, reviewed and approved based
on the design documentation.
 Software Quality Assurance Standards

(I) ISO
(International Organization of Standards)
ISO 9126 is an international standard for the evaluation of software quality.
The fundamental objective of this standard is to address some of the well
known human biases that can adversely effect the delivery and perception of
a software development project.

The standard is divided into four parts:

• Quality model
• External metrics
• Internal metrics
• Quality in use metrics.

[1]Quality Models
The quality model established in the first part of the standard, ISO 9126-1,
classifies software quality in a structured set of characteristics and sub-
characteristics as follows:
Functionality - A set of attributes that bear on the existence of a set of
functions and their specified properties.
Reliability - A set of attributes that bear on the capability of software to
maintain its level of performance under stated conditions for a stated period
of time.
Usability - A set of attributes that bear on the effort needed for use, and on
the individual assessment of such use, by a stated or implied set of users.
Efficiency - A set of attributes that bear on the relationship between the
level of performance of the software and the amount of resources used,
under stated conditions.
Maintainability - A set of attributes that bear on the effort needed to make
specified modifications.
Portability - A set of attributes that bear on the ability of software to be
transferred from one environment to another.

[2]Internal Metrics
Internal metrics are those which do not rely on software execution (static
measures).

[3]External Metrics
External metrics are applicable to running software.
[4]Quality in Use Metrics
Qualities in use metrics are only available when the final product is used in
real conditions.

ISO 9001
ISO 9001 is a standard based on quality management principles.It covers
product design,development,production,installation and services.ISO 9001
places great emphasis on documentation,control,traceability,measurement
and accountability.Corrective action and follow-through are also key themes.
ISO 9001 states what must be done to achieve a quality environment,it
does not say how to do this.Definition of the actual procedures is left to each
individual organization,with the goal that the procedures meet the
customer’s needs.Thus,the standard states key functional requirements for
which you must “say what you do,do what you say,and be able to
show what you have done”.The guideline ISO 9000-3 and the British
guide TickIT describe how to apply the ISO 9001 standard to the software
development environment.

Effects on software testing:

• Test documentation that conforms to company defined standards
• Existence of a test plan
• Control of all test environment
• Verification of design
• Defined audit and sign-off procedures for final test activities
• Evidence that problems are logged and tracked to resolution
• Maintenance of quality records
• Performance of internal quality system audits
ISO 9000 [1] proposed by International Standards Organization,
introduced the latest version ISO 9000:2000 in the year 2000. It is a
generic standard which can be applied to all types of industry including
software. ISO 9000:2000 has five major sections with focus on
customer satisfaction, data analysis and on continual improvement.
ISO 9000 is broad and general framework, which is very well
understood. ISO 9000 enables any organization to define the basic set
of processes, their interconnectivity, and needs measurement and
analysis of software metrics for continual improvement. For application
to computer software, a set of guidelines have been provided in ISO
90003, by International Standards Organization.

(II) IEEE
(Institute of Electrical and Electronics Engineers)
The Institute of Electrical and Electronics Engineers(IEEE) issues a
collection of standards that pertain to software engineering.Each
individual standard describes a particular aspect of software
development and provides specific directions on performing that
task.The IEEE software engineering standards most directly describe
how to perform software testing activities.
The IEEE standards relevant to software testing include:
• Well defined requirements.
 • Configuration management
• Reviews and audits
• Requirements traceability

Standards that may be referred to according to IEEE include:

IEEE 1008: a standard for unit testing
IEEE 1012: a standard for Software Verification and Validation
IEEE 1028: a standard for software inspections
IEEE 1044: a standard for the classification of software
anomalies
IEEE 1044-1: a guide to the classification of software anomalies
IEEE 830: a guide for developing system requirements
specifications
IEEE 730: a standard for software quality assurance plans
IEEE 1061: a standard for software quality metrics and
methodology
IEEE 12207: a standard for software life cycle processes and
life cycle data
BS 7925-1: a vocabulary of terms used in software testing
BS 7925-2: a standard for software component testing

Effects on software testing:

• Defines format and specifies the contents for the Test Plan
• Defines format and specifies the contents for the Test Design
Specifications
• Defines format and specifies the contents for the Test Case
Specifications
• Defines format and specifies the contents for Test Procedure
Specifications
• Defines format and specifies the contents for Test Item Transmittal
Report
• Defines format and specifies the contents for Test Log
• Defines format and specifies the contents for Test Incident Report
• Defines format and specifies the contents for Test Summary.
 (III) CMM
(Capability Maturity Model)
The capability Maturity Model for software produced by the software
Engineering Institute focuses primarily on management issues. the model
assigns a maturity level designation of 1 through 5 based on the
capability demonstrated by an organization.
The core principles of the SW-CMM include:
• Well-designed steps for performing each task
• Proper training for each person to perform job
• Existence of management support for all efforts performed
• Measurement of all steps performed
Levels included in the SW-CMM:
• Level1:Initial
• Level2:Repetbale
• Level3:Defined
• Level4:Managed
• Level5:Optimizing

Effects on software testing:

The CMM does not provide much guidance on the subject of software testing,
however,level3 addresses some testing-related issues.CMM addresses the
following activities:
• Defined process for performing software testing
• Peer reviews for test plans,test procedures and test cases
• Configuration management for test procedures,test plans and test
cases
• Traceability of product requirements from software requirements
through design,code and test cases
• Adequate resources and funding for performing software testing tasks
• Use and availability of tools
• Adequate training for software testers to perform their tasks
• Test readiness criteria established for unit,integration,system and
acceptance testing
• Problem tracking procedures.

(IV)CMMI:
CMMI (CMM Integration) [2] was proposed by SEI in 2000 to facilitate
integration of the contents of SW-CMM, SECM (for Systems Engineering) and
IPD-CMM (for Product Development). SW-CMM is withdrawn and SEI only
assesses companies for CMMI. CMMI has two representations: Staged and
Continuous. The continuous Representation deals with developing maturity
in selected group of process areas: Project Management, Support,
Engineering and Process Management. Staged Representation has five levels
with level 5 being the highest mark of maturity. The practices prescribed in
CMMI framework are very detailed and organizations intending to move up
the maturity levels are to show objective evidences of following the
prescribed practices.