TEST METHOD

TEST METHOD
VALIDATION
VALIDATION
Randall J. Troutman CPP
Senior Engineering Manager
Oliver Medical
Presentation Disclaimer
Oliver Medical
Oliver Medical
is devoted to building awareness while
is devoted to building awareness while
enhancing educational value among medical packaging
enhancing educational value among medical packaging
professionals through seminars such as this.
professionals through seminars such as this.
The views, topics, and suggestions discussed today are
The views, topics, and suggestions discussed today are
for educational purposes only and therefore should not
for educational purposes only and therefore should not
be considered the last word on these issues.
be considered the last word on these issues.
It is the sole responsibility of the attendee and his or her
It is the sole responsibility of the attendee and his or her
company to determine the suitability of information
company to determine the suitability of information
presented today. As well as to investigate all other
presented today. As well as to investigate all other
sources of information in regards to application of the
sources of information in regards to application of the
content.
content.
Presentation Overview
Background
ASTM Guidance / Definitions
ASTM F88-05 (overview)
ASTM F1929-98 (overview)
Getting Started in your lab
References
What is all the buzz?
FDA hot button issue -
Recent release of revised 11607 documents have caused
confusion in regards to listed test methods
Test method validation statement in both parts
Part 1 - Clause 4.4 Test Methods
Part 2 - Clause 4.3 Test Methods
This statement was included in the previous revision
ISO 11607 Clause 4.3.2
11607 part 1
4.4 Test Methods
4.4.1 All test methods used to show compliance with this part of ISO 11607 shall be
validated and documented.
NOTE Annex B contains a list of suitable test methods.
4.4.2 The test method validation shall demonstrate the suitability of the method used. The
following elements shall be included:
- establishment of a rationale for the selection of the appropriate test for the packaging system;
- establishment of acceptance criteria;
NOTE Pass/fail is a type of criteria;
- determination of test method repeatability;
- determination of test method reproducibility; and
- establishment of test method sensitivity for integrity tests.
11607 part 2
4.3 Test Methods
4.3.1 All test methods used to show compliance with this part of ISO 11607 shall be
validated and documented.
NOTE Annex B in ISO 11607-1 contains a list of suitable test methods.
4.3.2 The test method validation shall demonstrate the suitability of the method used. The
following elements shall be included:
- establishment of a rationale for the selection of the appropriate test for the packaging system;
- establishment of acceptance criteria;
NOTE Pass/fail is a type of criteria;
- determination of test method repeatability;
- determination of test method reproducibility; and
- establishment of test method sensitivity for integrity tests.
11607 part 1 Annex B
What does the list include?
Standard guides
Standard practices
Standard test methods
Including some that can not be validated
From various organizations..
11607 part 1 Annex B
Includes standards from the following organizations;
ASTM (Examples peel, dye, bubble emission, etc.)
ISO (Examples air permeance, biological evaluation, etc.)
EN (Example sterilization wrap requirements and test methods)
USP (Examples Biological reactivity test, in vitro)
TAPPI (Example dirt in paper)
DIN (Example testing of paper board bending stiffness)
General Definitions
Validation
1
<general> confirmation by examination and provision of objective
evidence that the particular requirement for a specific intended use
can be consistently fulfilled
Verification
2
confirmation by examination and provision of objective evidence that
specified requirements have been fulfilled
Qualification
referred to as the process of events used to establish feasibility of a
concept, process, or method
1. 11607 part 1
2. 21 CFR Part 820
ASTM Guidance
ATSM E 177
Standard Practice for Use of Terms
Precision and Bias in ASTM Test Methods
ATSM E 1169
Standard Guide for Conducting Ruggedness Tests
ATSM E 691
Standard Practice for Conducting an Interlaboratory Study
to Determine the Precision of a Test Method
ASTM Guidance cont.
ASTM E 2282
Standard Guide for Defining the
Test Result of a Test Method
ATSM E 1488
Standard Guide for Statistical Procedures to Use in
Developing and Applying Test Methods
ATSM E 456
Standard Terminology Relating to Quality and Statistics
ASTM E 177
Standard Practice for Use of Terms the
Precision and Bias in ASTM Test Methods
Purpose is to provide a basis for;
Precision & Bias statements (for quantitative methods)
Describes methods of expressing precision and bias
Gives examples of how P&B statements may be written
Definitions
precision
the closeness of agreement among test results obtained under
prescribed conditions
repeatability
addresses variability between independent test results gathered
from within a single laboratory (otherwise known as intralaboratory
testing)
reproducibility
addresses variability among single test results gathered from
different laboratories (otherwise known as interlaboratory testing)
Definitions
bias
a generic concept related to a consistent or systematic difference
between a set of test results from the process and an accepted
reference value of the property being measured
accuracy
a generic concept of exactness related to the closeness of
agreement between the average of one or more test results and an
accepted reference value
Accuracy should be used when comparing test results, not in terms
of the output result generated from execution of test a process. But
as - is this test result accurate, verses what is the accuracy of these
results compared to previously generated results.
ASTM E 1169
Standard Guide for
Conducting Ruggedness Tests
Purpose is to provide a basis for;
Finding the variables that influence measurements
experimental factors using Plackett-Burman designs
Determining how closely these factors need to be controlled
The test does not determine optimum conditions for
the test method
Definitions
ruggedness (robustness)
determination of variables associated with the performance of a test
method that is ultimately critical to the accuracy of the precision
statistical control
point when the test results obtained vary in a predictable manner,
showing no unassignable trends, cycles, abrupt changes, excess
scatter, or other unpredictable variations as determined by
application of appropriate statistical methods
ASTM E 691
Standard Practice for Conducting an Interlaboratory
Study to Determine the Precision of a Test Method
Purpose is to provide a basis for;
Addressing inherent variability in test methods
Includes (a.) Operator, (b.) equipment used, (c.) calibration
of equipment, (d.) environment (temp, humidity, & etc.)
Developing protocol w/ details
Calibration, specific methods of testing, etc.
Sample preparation
Sample preparation tips, including conditioning
Statistically valid sampling plans
Statistical evaluation methods
ASTM E 2282
Standard Practice for
Defining the Test Result of a Test Method
Purpose is to provide a basis for;
Identifying the elements that comprise the test
results of a test method and to illustrate how
these elements combine into the test result.
Definitions
test method
a definitive procedure for the identification, measurement and evaluation of one or
more qualities, characteristics, or properties of a material, product, system, or service
that produces a test result
observation or observed value
the most elemental single reading or corrected reading obtained in the process of
making a measurement
test determination
(1) the process of calculating from one or more observations a property of a single
test specimen, or (2) the value obtained from the process
test result
the value obtained by carrying out the complete protocol of the test method once,
being either a single test determination or a specified combination of a number of test
determinations
ASTM E 1488
Standard Guide for Statistical Procedures
to Use in Developing and Applying Test Methods
Purpose is to provide a basis for;
Using statistical procedures for use in
developing new test methods or
revising or evaluating existing test methods
Types of data
Numerical vs. Categorical / Attribute Data
Statistical Analysis Methods
Analysis of Means (ANOM)
Analysis of Ranges (ANOR)
ASTM E 456
Standard Terminology Relating to Quality and
Statistics
Purpose is to provide a basis for;
Terminology used in quality and statistical based ASTM
methods in which standard definitions appear
attribute data: observed values or determinations which
indicate the presence of absence of specific characteristics.
variables data: measurements which vary and may take any
of a specified set of numerical values.
Formal Process to Evaluate
As taken from 11607;
Suitability of particular test method
Robustness (ruggedness)
Precision and / or bias
Repeatability (within lab precision)
Reproducibility (between lab precision)
Range and / or Sensitivity of Detection
Document the Exercise
The engineering exercise should include:
Protocol
Objective / Scope
Sampling plan, w/ statistical rationale
Predetermined acceptance criteria
Report w/ supporting data
Demonstrates all requirements have been met
Statement of validation / successful execution
Includes proof of work
Applying the process.
ASTM F88-05
Quantitative test method application
Statistical comparison of test results w/ controlled variation
ASTM F1929-98
Qualitative test method application
Determining sensitivity levels for pass / fail tests
In house test method validation
Modified standard test method for use by MDM
Internal test method developed by MDM
1. Scope
1.1 method covers the measurement of the strength
of seal in flexible materials
1.2 method may be conducted on seals between a
flexible and rigid material
1.3 the seals tested may be from any source,
laboratory, or commercial
1.4 method measures the force required to separate
a test strip of material containing a seal. Also
identifies the mode of specimen failure
Conducting a Test Method Evaluation
Evaluate / Read the Test Method
Scope meets intended use
Understand the sources of variation
Material combinations
Flexible vs. rigid application
Number and types of equipment involved
Equipment calibration details
What, how, and when
Test sample conditioning
Conducting a Test Method Evaluation
Conduct an evaluation to expose how these
factors affect the test result
ruggedness test
Objective is to evaluate how the test method variables
affect the test determination
The results of this test will provide you with an
accurate sample to use in the repeatability
phase of your evaluation
Conducting a Test Method Evaluation
Conduct reproducibility studies with other pieces of like
equipment in other locations using samples proven in
the ruggedness and repeatability studies
Use a statistician to help determine the predetermined
Precision acceptance criteria for the method in your lab
Based on the data compiled within your R&R study you
may use a one-way analysis of variance (ASTM E 691) to
evaluate the data
Compare your data to the Precision data recorded in
the actual standard (Section 11)
Applying the process.
ASTM F88-05
Quantitative test method application
Statistical comparison of test results w/ controlled variation
ASTM F1929-98
Qualitative test method application
Determining sensitivity levels for pass / fail tests
In house test method validation
Modified standard test method for use by MDM
Internal test method developed by MDM
1. Scope
1.1 defines materials and a procedure that will
detect and locate a leak equal or greater
than a channel of 0.002 in.
1.2 intended for use on packages with an
edge seal formed between a transparent
film and a porous material
1.3 requires a contrasting dye penetrant
solution be used
Conducting a Test Method Evaluation
Evaluate / Read the Test Method
Scope meets intended use
Objective is to determine if the method that
you conduct internally meets the sensitivity
claim in the standardized method
For this particular method the sensitivity level is
a channel down to 0.002 in. in seal
Conducting a Test Method Evaluation
Overall evaluation should consider;
Sufficient sample sizes
Variation in dye mixture methods
Variation in sourced ingredients
Variation in exposure times
5 seconds vs. 20 seconds
Variation in the actual procedure / process
Variation in voids detected
Conducting a Test Method Evaluation
While considering those variables you
should prove the sensitivity of the method
Establish a range, w/ control limits
Example range
Control (no void)
0.001 void (min limit)
0.002 void (detectable)
0.003 void (detectable)
0.005 void (detectable)
Applying the process.
ASTM F88-05
Quantitative test method application
Statistical comparison of test results w/ controlled variation
ASTM F1929-98
Qualitative test method application
Determining sensitivity levels for pass / fail tests
In house test method validation
Modified standard test method for use by MDM
Internal test method developed by MDM
Non-Standardized Test
Methods
Acceptable for use but validation is more
difficult because there is no data for
comparison
You must ensure that the test is measuring the
intended attribute
US regulators expect more data from the
validation process
Getting started in your lab.
Review the standard test methods that you
currently execute within your company
How was the standardized method evaluated?
Review the ASTM standards mentioned in this
presentation for guidance
Develop a plan to validate (evaluate) your
methods separately and be prepared to employ
varying strategies to fulfill the requirement
Thank You
Randall Troutman CPP
Oliver Medical
800.253.3893 ext. 7914
616.456.7711 ext. 7914
rtroutman@olivermedical.com
www.olivermedical.com
References
Tyvek Rx Newsletter (Vol. 15 Issue 2 October 2006)
Fact vs. Fiction, ASTM Standardization News, Pat Picariello
Validation and Regulatory Acceptance of Toxicological Test Methods,
ad hoc Interagency Coordinating Committee on the Validation of
Alternative Methods, 1997
ISO EN 11607-1 2006
ISO EN 11607-2 2006
ASTM F1929-98
ASTM F88-05
ASTM E 177
ASTM E 1169
ASTM E 691
ASTM E 2282
ASTM E 1488
ASTM E 456
Additional Research
Test Method Validation Techniques
Analytical Chemistry Test Methods
Toxilogical Test Methods
Forensic Test Methods