Software Process Reviews/Audits

Process Overview
by

Tom Gilchrist, CSQA, CSQE,

Before we start
Information in this presentation are
my opinions and not necessary
those of my employer.

SQA Context
Overview of SW Audit Process
SW Audit Examples

SASQAG

Some Terms/Ideas

Process
Deterministic vs. Non
Deterministic
Quality vs. Value

SASQAG

Software Quality Assurance

Check software products and processes to verify that they
comply with the applicable procedures and standards.
(Process Reviews or Audits)
Review and measure the quality of software products and
processes throughout development. (Dynamic & Static
Testing)
Provide software project management (and other appropriate
parties) with the results of reviews and process checks.
Work with the software project during early stages to establish
plans, standards, and procedures to keep errors from
occurring in the first place.

SASQAG

Formal Definition
Audits provide an independent evaluation of
software products or processes to ascertain
compliance to standards, specifications, and
procedures based on objective criteria that included
documents that specify:
The form or content of the product to be
produced
The process by which the products shall be
produced
How compliance to standards or guidelines
shall be measured.
IEEE STD 1028, (1988)

SASQAG

Audit Types

First Party Audit

Within you company or organization

Second Party Audit

Sometimes called external audits
By a Customer on his Supplier
By a Supplier on you.

Third Party Audit

Outside third party is contracted to do
the audit.
SASQAG

Audit/Process Review Principles

Conducted by individuals who are
organizationally independent of the developers.
Begin early in the requirements phase and
continue throughout the development process.
Professionally planned, conducted and
documented.
Follow-up on corrective action.
Project Management is involved in the Audit
process and is responsible for rework and
process improvements.
SASQAG

What Software Audit Should Do

Determine:
Compliance to requirements
Conformance to plans, policies, procedures, and
standards
Drive process improvement based on:
Adequacy of plans, policies, procedures, and
standards
Effectiveness and efficiency of plans, policies,
procedures, and standards
Assess personnel familiarity to requirements and
documentation
Assure availability, use and adherence to software
standards

SASQAG

What Triggers an Audit?

Quality Assurance Plan
Event
Date
Requests from management
Requests from developers
Requests from customers
Integration with process improvement activities
Outside requirements regulatory
Gut feel

SASQAG

Scope: Requirements, Time, and Target

External
Standards

Spread around
organization
Audit
Target

Organizational
Procedures and
Methods

SASQAG

Cover all functions and

activities
Try to hit things early
Move towards process
audits

10

Process Review/Audit Process

Developers

Project Manager

Auditor
Plan
(Requirements,
Scope, & Checklist)

Start

Conduct
Audit

Prepare
Audit

Write-up
Report &
Findings

Review
with
Manager

NO

Findings?
YES

Corrective
Actions
OK

Closeout
Audit &
File

Re-Work

SASQAG

END

Follow-up
Audit

11

Identify Requirements
Policies/Standards Corporate, Group, IEEE
Processes/Plans SCMP, SQAP, SDP, Project Plan
Procedures
Change Management, Design
Reviews, Document Standards,
Testing
Task Instructions Library updates, unit testing, peer
reviews
Success of an audit is directly proportional to preparation,
research and analysis conducted before the audit is
performed.
SASQAG

12

Requirement Types

Functional (ascertainably true or false)

Quality (range of acceptable values)

SASQAG

13

Types of Audits (Internal)

SASQAG

Quality System Audits

Product Audit
Process Audit
Project Audit
CM Audit

14

Evidence Collection

Collect Factual Information

Analyze and Evaluate the Evidence
Draw Conclusions
Generate Findings

SASQAG

15

Corrective Action of Findings

Determine Action
Immediate Remedial Action
Process Improvement/Fix
Acceptable Risk
Identify Root Cause
Corrective Actions Plan
Manage CA Plan to completion
Analyze Effects of CA

SASQAG

16

Develop Audit Checklist

Focus on clear requirements (or

unclear to fix)
Select subset of requirements
Focus on important steps/products
Write clear concise questions
Canned checklist vs. straw horse

SASQAG

17

Checklist Sample
Requirement

Checklist Item

Company
Standard ABC234, page 7

Does project QA plan

will have a list of
deliverables subject to
Peer Reviews?

Check SQA document for a list

of approved peer reviews and
which documents are to be
reviewed. (if no documents are
found, then fail. If no peer
review procedures are
referenced, then fail)

Project SQA
Plan

Were the number of

audits completed
equal to the number
planned?

Check to see which audits were

planned for the last 60 days.
Check for evidence that the audit
was completed and if there were
findings, that a CA plan was
signed.

Project SQA
Plan

Were the number of

peer reviews
completed equal to the
number planned?

For each peer review type, check

the CM records for the past 60
days to see if the document type
specified in the QA plan was
checked into CM for the first
time. If so, check for records of
the peer review being completed
as per peer review process cited
in SQA plan.

SASQAG

Details

Observations

Results (P/F)

18

Interviewing

Ask open-ended questions

Know the types of answers expected
Focus on Process and not People
Seek Corroboration and Evidence

SASQAG

19

Sample Interview Questions

How do you track your progress?
Do you have a CM Plan?
Tracing
What are you working on?
Is it a configured item?
Do you have an approved CR or PR?
Is the version you are working on
checked out of CM?

SASQAG

20

Desirable Auditor Characteristics

Emotional
Interviews
Group
dynamics
Oral reports
Empathy
Dont take
things
personally

SASQAG

Mechanical
Sampling
Root Cause
Analysis
Intellectual
Writing
Planning
Speaking
Detail
Oriented
Concise
21

Desirable Auditor Characteristics

(Cont.)

Knowledge of Audit process

Knowledge of target (SW) processes
Knowledge of techniques
Professional attitude
Good listener
Inquisitive/analytical
Communicates at all levels
Detailed Notes and Observations
Diplomatic

SASQAG

22