Pittsburgh, PA 15213-3890

Software Quality Requirements and Evaluation, the ISO 25000 Series

PSM Technical Working Group February 2004 Dave Zubrow
Sponsored by the U.S. Department of Defense 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Background
This presentation reviews the latest developments with the ISO 25000 SQuaRE series of standards. This series on Software Quality Requirements and Evaluation (SQuaRE) is an effort to harmonize ISO 9126 and ISO 14598. The information presented here is based on the current state of the standards/

Note: Many of the slides in this presentation are taken from WG 6 documents and materials, especially slides from Professor Azuma, WG convener.

page 2 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Outline
Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation

page 3 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Relationship between ISO/IEC 9126 and ISO/IEC 14598

R e s o u rc e s a n d e n v ir o n m e n t E v a lu a tio n p ro c e s s S o ftw a re p ro d u c t E ffe c t o f th e s o ftw a re p ro d u c t

E v a lu a tio n s u p p o rt

E v a lu a tio n p ro c e s s

In te rn a l m e tr ic s

E x te rn a l m e t r ic s

Q u a lity in u s e m e tr ic s

1 4 5 9 8 -1 1 4 5 9 8 -2 1 4 5 9 8 -6 1 4 5 9 8 -3 1 4 5 9 8 -4 1 4 5 9 8 -5 9 1 2 6 -3 9 1 2 6 -1 9 1 2 6 -2 9 1 2 6 -4

page 4 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Relationship and transition process between ISO/IEC 9126, ISO/IEC 14598 and SQuaRE series of standards
C U R R E N T 9 1 2 6 : P r o d u c t q u a lity -1 : Q u a lity m o d e l -2 : E x te rn a l m e trics - 3 : In t e r n a l m e t r i c s -4 : Q u a lity in u s e m e trics N ew P rop osal G u id e s to u s e 9 1 2 6 & 1 4 5 9 8 B a s e m e tric s Q u a lity r e q u ir e m e n ts 1 4 5 9 8 : P r o d u c t e v a lu a tio n -1 : G e n e ra l o v e rv ie w -2 : P la n n in g a n d m a n a g e m e n t -3 : P ro c fo r d e v e lo p e rs -4 : P ro c fo r a c q u ire rs -5 : P ro c fo r e v a lu a to rs -6 : D o c o f e v a lu a tio n m o d u le s S Q u a R E 2 5 0 0 0 : Q u a lit y M a n a g e m e n t D iv is io n 2 5 0 0 0 : G u id e to S Q u a R E (N P ) 2 5 0 0 1 : P la n n in g a n d m a n a g e m e n t 2 5 0 1 0 : Q u a lit y M o d e l D iv is io n 2 5 0 1 0 : Q u a lity m o d e l a n d g u id e (R e v ) 2 5 0 2 0 : Q u a lity M e a s u r e m e n t D iv is io n 2 5 0 2 0 : M e a s u re m e n t re fe re n c e m o d e l a n d g u id e (N P ) 2 5 0 2 1 : M e a s u re m e n t p rim itiv e s (N P ) 2 5 0 2 2 : M e a s u re m e n t o f in te rn a l q u a lity 2 5 0 2 3 : M e a s u re m e n t o f e x te rn a l q u a lity 2 5 0 2 4 : M e a s u re m e n t o f q u a lity in u s e 2 5 0 3 0 : Q u a lit y R e q u ir e m e n t s D iv is io n 2 5 0 3 0 : Q u a lity re q u ire m e n ts a n d g u id e (N P ) 2 5 0 4 0 : Q u a lit y E v a lu a t io n D iv is io n 2 5 0 4 0 : Q u a lity e v a lu a tio n o v e rv ie w a n d g u id e 2 5 0 4 1 : E v a lu a tio n m o d u le s 2 5 0 4 2 : P ro c e s s fo r d e v e lo p e rs 2 5 0 4 3 : P ro c e s s fo r a c q u ire rs 2 5 0 4 4 : P ro c e s s fo r e v a lu a to rs

page 5 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

State of the Standards

25000, Guide to the SQuaRE series Ballot out for FCD 25020, Software Product Quality Measurement Reference Model Ballot out for 2nd CD 25021, Measurement Primitives Ballot out for 1st CD 25030, Quality Requirements Ballot out for 2nd CD All Ballots closing Mid-April to Early May Editors assigned but no drafts out 25010, Quality Model 25023, External Quality Measures

page 6 2004 by Carnegie Mellon University

SQuaRE: Architecture
ISO/IEC 2501n ISO/IEC 2501n Quality Model Quality Model Division Division ISO/IEC 2500n ISO/IEC 2500n Product Quality Product Quality General Division General Division
General Overview and General Overview and Guide to the SQuaRE Guide to the SQuaRE Planning and Planning and Management Management

Carnegie Mellon Software Engineering Institute

ISO/IEC 2503n ISO/IEC 2503n Quality Quality Requirement Requirement Division Division

ISO/IEC 2504n ISO/IEC 2504n Quality Quality Evaluation Evaluation Division Division

ISO/IEC 2502n ISO/IEC 2502n Quality Metrics Quality Metrics Division Division
2004 by Carnegie Mellon University

page 7

Carnegie Mellon Software Engineering Institute

SQuaRE general reference model

Target of process

Software Product

(Effect)

User Task

Process

Requirements Specification Planning Measurement Evaluatioin

Requirements Specification Planning Measurement Evaluation

Execution

25030 (QR)

25001 (PM)

25021( MP) 25022 (MfIQ) 25023 (MfEQ)

25041(EM) 25042 (PD) 25043 (PA) 25044 (PE)

25030 (QR)

25001 (PM)

25021( MP) 25022 (MfIQ) 25024 (MfQU) 25025(DEM)

25041(EM) 25042 (PD) 25043 (PA) 25044 (PE)

25020 (Measurement Reference Model and Guide)

25030 (Quality Requirements and Guide) 25040 (Evaluation Process Overview and Guide)

Particular Guidance

Legend : applied to 25010 (Quality Model and Guide) used by MP Measurement Primitives MfIQ Metrics for Internal Quality MfEQ Metrics for External Quality MfQU Metrics for Quality in Use EM Evaluation Modules PM Planning and Management PA Process for Acquirers PE Process for Evaluators PD Process for Developers QR Quality Requirements

General Guidance
25000 (Guide to SQuaRE)

page 8 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Outline
Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation

page 9 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

The Product Quality Measurement Reference Model

page 10 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Software Product Quality Life-Cycle and Quality Measures

Requirements
Needs

Products
Quality Quality In Use In Use Validation

Quality In Use Measures External Quality Measures Internal Quality Measures

Users Quality Users Quality Needs Needs

External Quality External Quality Requirement Requirement

External External Quality Quality

Verification And Internal Quality Validation Internal Internal Quality Internal Requirement Quality Requirement Quality Verification
Implementation
page 11

2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Quality Model
Quality model

Characteristic 1

Characteristic 2

Characteristic n

Subcharacteristic 1

Subcharacteristic 2

Subcharacteristic m

Quality Attributes

page 12 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

ISO/IEC 9126-1 Quality In Use

Effectiveness Effectiveness

Quality In Use Quality In Use

Productivity Productivity

Safety Safety

Satisfaction Satisfaction
page 13 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

ISO/IEC 9126-1 - Quality Model

Quality Characteristics Functionality
Suitability Accuracy

Subcharacteristics
Interoperability Security

Compliance

Reliability
Maturity Fault tolerance Recoverability

Compliance Attractiveness

Usability
Understandability Learnability Operability Comp

Efficiency
Time behavior Resource utilization Changeability Installability Stability

Compliance
Testability

Maintainability
Analyzability

Compliance Comp
page 14

Portability
Adaptability Co-existence Replaceability

2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Process and Product Measurement

External Quality Measure External Quality Measure Internal Quality Measure Internal Quality Measure Quality in Use Measure Quality in Use Measure

Software

Computer System Information System Business System Computer Centered System

page 15 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Software product quality measurement reference model (SPQM-RM)

page 16 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Measurement concept of the SQuaRE model

page 17 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Relationship between the SQuaRE SPQM-RM and the ISO/IEC 15939 Information Model
Q u a lit y N e e d s / Q u a lit y E v a lu a tio n R e p o rt A s s e s s m e n t A n a ly s is R a tin g Q u a lity M e a s u r e s A n a ly s is M o d e l F u n c tio n (F o r m u la ) D e r iv e d M e a s u r e M e a s u re m e n t F u n c tio n B a s e M e a s u re M e a s u re m e n t M e th o d M e a s u re m e n t M e th o d A ttrib u te s In fo r m a tio n P ro d u c t

In te r p r e ta tio n In d ic a to r

M e a s u re m e n t P r im itiv e s

A ttr ib u te s fo r Q u a lity M e a s u re

S Q u a R E

S P Q M -R M

IS O /IE C

1 5 9 3 9

page 18 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Outline
Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation

page 19 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Titles of the Measurement Standards: 2502n

Common: Software Engineering Software product Quality Requirements and Evaluation (SQuaRE) Division: Software Quality Measurement (For Explanation Only) 25020 Measurement Reference Model and Guide 25021 Measurement Primitives 25022 Measurement of Internal Quality 25023 Measurement of External Quality 25024 Measurement of Quality In Use

page 20 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Quality Measure and Measurement Method (Concept)

Quality Measures: A variable, which shows satisfactory levels of a quality characteristic, subcharacteristic or sub-sub-characteristic, to which a value is assigned as the result of measurement Quality Measure Set: A set of a Quality Measure, Measurement Primitives that are used for deriving the quality measure, associated Scales and Measurement Methods, a Formula to combine them to generate the value of Quality Measure, and Guide to use them and analyze the results. for a Quality Characteristic or Subcharacteristic. (A line of Quality Measure Table) Quality Measurement Table: A set of Quality Measure Set for each Quality Subcharacteristic

page 21 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Application of SQuaRE
Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8
Phase Requirement analysis (Software and systems) Architectural design (Software and systems) Software detailed design Software coding and testing Software integration and software qualification testing System integration and system qualification testing Software installation Software acceptance support

25000 (SQuaRE) series model reference

Required quality in use, Required internal quality, Required external quality

Predicted quality in use, Predicted external quality, Measured internal quality

Predicted quality in use, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Measured internal quality

Predicted quality in use, Measured external quality, Measured internal quality

Measured quality in use, Measured external quality, Measured internal quality

Key deliverables of activity

Quality in use requirements (specified), External quality requirements (specified), Internal quality requirements (specified)

Architecture design of Software / system

Software detailed design

Software code, Test results

Software product, Test results

Integrated system, Test results

Installed system

Delivered software product

Applicable measures

Internal measures (External measures may be applied to validate specifications)

Internal measures

Internal measures

Internal measures External measures

Internal measures External measures

Internal measures External measures

Internal measures External measures

Quality in use measures Internal measures External measures

page 22 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

User needs, characteristics & weights

Q u a lit y in u s e
C H A R A C T E R IS T IC E ffe c tiv e n e s s P r o d u c tiv it y S a fe ty S a tis f a c ti o n

W E IG H T
H H L M

E x te rn a l &

in t e r n a l q u a lit y S U B C H A R A C T E R IS T IC
S u ita b ilit y A c c u ra c y In t e r o p e r a b ility S e c u r it y C o m p lia n c e

C H A R A C T E R IS T IC
F u n c t io n a lit y

W E IG H T ( H ig h / M e d iu m / L o w )
H H L L M L L H H M L H M H H H H H M L M H H L H M H

R e lia b ilit y

M a tu r ity F a u lt t o le r a n c e R e c o v e r a b ilit y C o m p lia n c e

U s a b ilit y

U n d e r s ta n d a b ilit y L e a r n a b ilit y O p e r a b ility A ttr a c tiv e n e s s C o m p lia n c e

E f f ic ie n c y

T im e b e h a v io u r R e s o u r c e u tiliz a tio n C o m p lia n c e

M a in t a in a b ilit y

A n a ly z a b ilit y C h a n g e a b ilit y S t a b ilit y T e s t a b ilit y C o m p lia n c e

P o r t a b ilit y

A d a p t a b ilit y In s ta lla b ilit y C o - e x is t e n c e R e p la c e a b ilit y C o m p lia n c e

page 23 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Quality measurement tables

Q u a lity in u s e m e a s u r e m e n t c a te g o r y
R E Q U IR E D L E V E L A S S E S S M E N T A C T U A L R E S U L T C H A R A C T E R IS T IC M E A S U R E S
E f fe c tiv e n e s s P r o d u c tiv it y S a fe ty S a tis f a c tio n

In te r n a l q u a lit y m e a s u r e m e n t c a te g o r y

E x te r n a l q u a lit y m e a s u r e m e n t c a t e g o r y
R E Q U IR E D L E V E L A S S E S S M E N T A C T U A L R E S U L T
F u n c t io n a lit y S u i ta b ilit y A c c u ra c y In t e r o p e r a b ility S e c u r it y C o m p lia n c e

C H A R A C T E R IS T IC

S U B C H A R A C T E R IS T IC

M E A S U R E S

R E Q U IR E D L E V E L

A S S E S S M E N T A C T U A L R E S U L T

C H A R A C T E R IS T IC

S U B C H A R A C T E R IS T IC

M E A S U R E S

F u n c t io n a lit y

S u ita b ilit y A c c u ra c y In t e r o p e r a b ility S e c u r it y C o m p lia n c e

R e lia b ilit y

M a tu r ity F a u lt t o le r a n c e R e c o v e r a b ilit y ( d a t a , p r o c e s s ,

R e li a b ilit y

M a tu r ity F a u lt to le r a n c e R e c o v e r a b ilit y ( d a t a , p r o c e s s , C o m p lia n c e

U s a b ilit y

C o m p lia n c e U n d e r s ta n d a b ilit y L e a r n a b ilit y O p e r a b ility A ttr a c tiv e n e s s C o m p lia n c e E f f ic ie n c y T im e b e h a v io u r

U s a b il it y

U n d e r s ta n d a b ilit y L e a r n a b ilit y O p e r a b ility A ttr a c tiv e n e s s C o m p lia n c e

R e s o u r c e u tilis a tio n

E ffic ie n c y

T im e b e h a v io u r
C o m p lia n c e

R e s o u r c e u tilis a tio n
M a in t a in a b ilit y A n a ly z a b ilit y C h a n g e a b ility S t a b ilit y

C o m p lia n c e

M a in t a in a b i lit y

A n a ly z a b ilit y C h a n g e a b ilit y S t a b ilit y T e s t a b ilit y C o m p lia n c e

P o r t a b ilit y

T e s t a b ilit y C o m p lia n c e A d a p t a b ilit y In s ta b ilit y C o - e x is t e n c e R e p la c e a b ilit y C o m p lia n c e

P o r t a b il it y

A d a p t a b ilit y In s ta b ilit y C o - e x is t e n c e R e p la c e a b ilit y C o m p lia n c e

page 24 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Measurement plan implications

S U B C H A R A C T E R I S T IC D E L IV E R A B L E S T O B E IN T E R N A L M E A S U R E S T O A P P L IE D
1 . 2 . 3 . ( N o t A p p lic a b le ) 1 . 2 . 3 . ( N o t A p p lic a b le ) 1 . 2 . 3 .

E X T E R N A L B E M E A S U R E S T O A P P L IE D B E

Q U A L IT Y IN

U S E B E

M E A S U R E S T O A P P L IE D
( N o t A p p lic a b le )

E V A L U A T E D
1 . S u it a b ilit y 1 . 2 . 3 . 2 . S a t is f a c t io n 1 . 2 . 3 . 3 . 4 . 5 . 6 .

page 25 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Measurement Primitive
A measure, either a base measure or a derived measure, that is commonly used for deriving internal quality measures, external quality measures and quality in use measures. No definition (25000) measure collected during Software Product Lifecycle from which Internal, External and Quality in Use Measures are derived. (25020 and 25021) Single value of measurement primitive generally does not indicate the quality of the measured entity. NOTE The Quality is measured afterwards by calculating the Quality Measures. (25020 and 25021)

page 26 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Measurement Primitives in SPQM-RM

2004 by Carnegie Mellon University

Measurement primitives concept

page 27

Carnegie Mellon Software Engineering Institute

Consideration views

Considerations
Measurement Concept Base Derived Measurement Scale Type Nominal Ordinal Interval Ratio Absolute Measurement Focus Internal External Q in use Measurement Method Type Subjec tive Objec tive

Measurement

primitive

page 28 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Measurement Primitives Hyperlink Format example

C o n s id e r a t io n V ie w s M e a s u re m e n t c o n c e p t B a se D e r iv e d M e a s u r e m e n t s c a le N o m in a l : R a t io A b s o lu te M e a s u re m e n t fo c u s In te rn a l E x te rn a l Q u a lit y in U s e M e a s u re m e n t m e th o d O b je c t iv e S u b je c tiv e M e a s u r e m e n t P r im itiv e s : N u m b e r o f f u n c tio n s : : T a s k t im e : : T im e o r n u m b e r o f t a s k s : Q u a lity M e a s u re s : P h y s ic a l a c c e s s ib ilit y : : W a itin g t im e : : T a s k e ffic ie n c y :

page 29 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Outline
Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation

page 30 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Relationship between Needs and Requirements

Stakeholders Needs in their Minds Stated, Implied or Unaware Needs Collected and Identified Stakeholders (Business) Needs Solicit & Identify Select & Specify Selected and Specified Needs & QIU Requirements Functional Requirements Functional Design & Internal Quality Requirements

External Quality Requirements

Non Functional Design & Internal Quality Requirements

page 31 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

ISO/IEC 15288 System Life Cycle Processes to appear in 25030

ISO/IEC 25030 Quality requirements
Elicit and define quality requirements Formalise identified quality requirements Identify & formalise internal quality requirements

Stakeholder requirements definition process Requirements analysis process Architectural design process Implementation process Integration process Verification process Transition process Validation process Operation process Maintenance process

SQuaRE

ISO/IEC 15288 (Technical Processes)

page 32 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Evaluation process view according to ISO/IEC 14598-1

E s ta b lis h ev a lu a tio n req u irem en ts E s ta b lis h p u r p o s e o f e v a lu a tio n I d e n tif y ty p e s o f p r o d u c t( s ) S p e c if y q u a lity m o d e l 9 1 2 6 -1 Q u a l i ty C h a r a c te r is tic s

S e le c t m e t r ic s S p ecify th e ev a lu a tio n E s ta b lis h r a t in g le v e ls f o r m e tr ic s E s ta b lis h c r i te r ia f o r a s s e s s m e n t

9 1 2 6 - 2- 6E E t ear ln aal t i o n x v 14596 u - M e o rr i u s e s ri M td cl 1 1 2 6 8 3 6I n t v a n u a t i o E er l a l 9 M e tdr ucls s M o i e 9 1 2 6 - 4 Q u a lity in u s e M e tr ic s 1 4 5 9 8 - 6 E v a lu a tio n M o d u les

D es ig n th e ev a lu a tio n

P r o d u c e e v a lu a tio n p la n

M e a s u r e c h a r a c te r is tic s E x ecu te th e ev a lu a tio n C o m p a r e w it h c r i t e r i a A s s e s s r e s u lts

page 33 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Summary - 1
SQuaRE series addresses requirements and evaluation of software product quality Internal, External, and Quality in Use Measures are the link between requirements and evaluation Software Product Quality Measurement Reference Model is a specific instance of 15939 information model Additional standards to create catalogues of measures for quality attributes and measurement primitives Need for validated measures to populate catalogues
page 34 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Summary - 2
Working to reconcile and harmonize SQuaRE series with other standards Concern over number of standards and fragmentation of content Reviewers sought

page 35 2004 by Carnegie Mellon University