Journal of Chromatography A, 1158 (2007) 158–167

Review

Set-up and evaluation of interlaboratory studies

Yvan Vander Heyden, Johanna Smeyers-Verbeke ∗
Vrije Universiteit Brussel (VUB), Pharmaceutical Institute, Department of Analytical Chemistry and Pharmaceutical Technology,
Laarbeeklaan 103, B-1090 Brussels, Belgium
Available online 22 February 2007

Abstract
Interlaboratory comparison by means of method performance precision and bias studies and proficiency testing schemes are described. The set-up
of the experiments as well as the evaluation of the data by means of graphical and statistical methods are considered. The use of interlaboratory
data for the estimation of measurement uncertainty is also addressed.
© 2007 Elsevier B.V. All rights reserved.

Keywords: Interlaboratory studies; Method performance; Proficiendy testing; Bias evaluation; Uncertainty

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
2. Method performance studies—precision experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
2.1. Aim and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
2.2. Organisation of a precision experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
2.3. The components of precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
2.4. Evaluation of the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
2.4.1. Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
2.4.2. Calculation of the variances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
2.4.3. Repeatability and reproducibility as a function of concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
2.5. What precision to expect? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
3. Method performance studies—bias experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
3.1. Aim and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
3.2. Organisation of the bias experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
3.3. The evaluation and interpretation of the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
4. Proficiency studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
4.1. Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
4.2. Organisation of a proficiency study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
4.3. Evaluation of the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5. Interlaboratory studies and uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

∗ Corresponding author. Tel.: +32 2 477 4737; fax: +32 2 477 4735.
E-mail address: fabi@vub.ac.be (J. Smeyers-Verbeke).

0021-9673/$ – see front matter © 2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.chroma.2007.02.053
 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
1. Introduction
Interlaboratory studies, which are sometimes also called col-
laborative studies or trials or ring tests, are studies in which
several laboratories analyse the same material(s). Depending
on the focus of the study three main types can be distin-
guished.
Collaborative trials or method-performance studies assess
the performance characteristics of a specific method. In the ISO
5725 guidelines [1] they are called accuracy experiments and
consider the evaluation of the precision as well as the true-
ness from the interlaboratory trial. The ISO 5725-2 guideline
[1] specifically describes precision experiments for the determi-
nation of the repeatability and the reproducibility. The second
part of accuracy is trueness, which, in an interlaboratory context,
measures the bias of the measurement method. ISO describes the
bias experiments in ISO 5725-4 [1].
Laboratory-performance or proficiency studies focus on the
laboratory with the aim to assess the proficiency of the individual
laboratories. In such studies, sometimes called round-robin stud-
ies, test samples of which the concentrations are known or have
been assigned, often from the interlaboratory experiment itself
is analysed by a group of laboratories. The laboratories apply
whatever method in use in their laboratory. Proficiency testing is
an essential part of the accreditation of analytical laboratories.
The International Harmonized Protocol for the proficiency test-
ing of analytical chemistry laboratories [2], which is the result
of a cooperation between AOAC International, ISO and IUPAC,
has recently been revised to include the experience gained in
the last 10 years [3]. In contrast to collaborative trials, which are
generally organised by one of the participating laboratories, pro-
ficiency tests that are planned a few times per year are managed
by an external body.
The objective of material-certification studies is to provide
(certified) reference materials. A group selected laboratories
analyses, preferably with different methods, a material to deter-
mine the most probable value of the concentration of a certain
analyte with the smallest uncertainty possible.
Since the latter studies for the production and evaluation of
reference materials are very specialised [4] and generally organ-
ised by institutions created for that purpose we will only focus
on the method-performance and proficiency studies.
2. Method performance studies—precision experiments
2.1. Aim and definitions
The aim of a precision experiment is to evaluate the
repeatability and the reproducibility of an analysis method. By
convention everything related to repeatability is reported by r
and everything related to reproducibility by R.
ISO 5725-1 [1] defines repeatability as “the precision under
repeatability conditions”. This means “conditions where inde-
pendent test results are obtained with the same method on
identical test items in the same laboratory by the same operator
using the same equipment within short intervals of time”. Short
intervals of time mean that the measurements are carried out
159
without recalibration of the instrument, unless the recalibration
is an inherent part of the method. The word “independent” means
that all steps of the method must be carried out for each repli-
cate measurement. For instance, it is not acceptable to extract
one sample and then carry out replicate determinations on it by
chromatography. This results only in an estimate of the injec-
tion repeatability and cannot be considered an estimate of the
method repeatability.
An analyst can measure the repeatability of a method as
practised by him/herself in his own laboratory. When the repeata-
bility of a standard measurement method as carried out by
qualified laboratories in general is measured, this must be done
by a collaborative experiment.
The repeatability can be described by the repeatability stan-
dard deviation sr . Another measure is the repeatability limit, r.
This is defined by ISO as “the value less than or equal to which
the absolute difference between two test results obtained under
repeatability conditions may be expected to be with a probability
of 95%”. It can be shown that r = 2.8sr .
The repeatability variance and relative repeatability standard
deviation are written as sr2 and RSDr , respectively.
Reproducibility is defined by ISO as the “precision under
reproducibility conditions”. The latter are “conditions where
test results are obtained with the same method on identical test
items in different laboratories with different operators using
different equipment”. It follows that the reproducibility of a
method is necessarily obtained from a collaborative precision
study.
The reproducibility limit R is defined by ISO as “the value
less than or equal to which the absolute difference between two
test results obtained under reproducibility conditions may be
expected to be with a probability of 95%”. It is obtained as
R = 2.8sR .
The repeatability and reproducibility limits, r and R, can
be used to include precision clauses in the description of the
method. A typical precision clause is: “The absolute difference
between two single test results obtained under repeatability con-
ditions should not be greater than 0.7 mg/L”. An experimental
difference between two values larger than r therefore indicates
that the laboratory’s repeatability for the method investigated is
not up to standard. Part 6 of the ISO standard 5725 [1] describes
a procedure on how to judge on the acceptability of results when
the repeatability limit is exceeded.
Notice that the GUM [5] defines reproducibility as “the
closeness of agreement between the results of measurements
of the same measurand carried out under changed conditions
of measurement”. The Guide continues to specify that the
changed conditions may include different principles or meth-
ods of measurement, different observers, different measuring
instruments, different locations, different conditions of use or
different periods of time and a note specifies that “a valid state-
ment of reproducibility requires specification of the conditions
changed”. Reproducibility is then no longer linked to a specific
method and therefore, in a GUM context, an interlaboratory
study as described in this article measures reproducibility in
a measurement set up in which all possible conditions change
except the method applied.
160 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
2.2. Organisation of a precision experiment
Guidelines and protocols to set up and interpret method per-
formance studies are available from among others ISO [1] and
IUPAC [6]. The latter is a revision of an earlier protocol [7],
which is also adopted by the AOAC as the guideline for the
AOAC Official Methods Program [8].
ISO [1] specifies that the method to be investigated shall be
standardised, which means that the procedure is described in
detail and that in general it is in use in a number of laboratories.
ISO adds however that the same approach can also be used to
study the precision of non-standardised methods. In any case a
clearly written protocol has to be available. This should, where
relevant, include system suitability checks and the way the test
items are prepared before the actual analysis starts.
Since collaborative studies are very time-consuming AOAC
[8] warns not to conduct such experiments with unoptimised
methods. A preliminary pilot trial can be useful to familiarise
with the measurement method and to evaluate the protocol for
errors and ambiguities. For the latter CIPAC [9] and AOAC [8]
suggest performing a pilot trial with three or four laboratories.
The study should be organised and supervised by a panel
that consists of experts familiar with the method. They usually
delegate certain responsibilities to an executive officer. The exec-
utive officer, which takes care of the actual organisation of the
study, usually belongs to one of the laboratories participating
in the study. Moreover, a statistical expert should be involved
in the design of the experiments, the analysis and reporting of
the data. Within each laboratory a supervisor selects a proficient
analyst, organises the actual performance of the measurements,
taking into account the instructions from the executive officer
and the reporting of the results.
The laboratories that participate are representative for those
laboratories that will apply the method later. This also means
that the laboratories should achieve a representative level of
competence. There may be a tendency to include only the very
best laboratories, but this will lead to an underestimation of the
precision measurements.
The objective of a precision experiment is to get an estimate
s of the true standard deviation σ. Enough measurements must
be carried out for the values of s to approach σ sufficiently well.
The ISO standard 5725-1 [1] includes a complex equation that
allows computing how many measurements should be carried
out and concludes that the number of laboratories p participat-
ing in the experiment should be at least 8. It does not appear
useful to include more than 15 laboratories (although this is not
discouraged). The AOAC guideline [8] states that enough labo-
ratories should participate to ensure that at least 8 valid results
would be obtained. The older version of the ISO standard [10]
added that when only one single level of concentration is of inter-
est at least 15 laboratories should be included. AOAC states that
at least 15 laboratories are needed for collaborative studies of
qualitative analyses.
The number of laboratories mentioned above is needed if one
wants to develop an internationally accepted reference method.
This does not mean that method-performance studies with a
smaller number of participants are not useful. They are then an
additional step in the intralaboratory validation of a few labo-
ratories with common interests or a preliminary step in a true
interlaboratory study (see [11]).
The basic experimental design is a balanced uniform level
experiment that requires p laboratories to analyse the material n
times at q levels. The analysis is carried out at q levels of con-
centration. Indeed, precision can (and very often does) depend
on concentration and the validation of a method has to be done
over the whole range concerned. It is considered that in general it
should be possible to cover the expected range by 5 materials, i.e.
q = 5, where a material is a specific combination of matrix and
level of analyte. However there are cases where this is neither
possible nor useful. For instance, when a method is validated for
the analysis of a drug in formulations, its concentration may be
situated in the relatively small range at which it is pharmacolog-
ically active but not toxic so that drugs will not be formulated at
largely different concentrations. A smaller number can then be
accepted. AOAC states that it should not be less than 3.
For qualitative analyses AOAC [8] recommends 2 analyte
levels per matrix, 6 samples per level and 6 negative controls
per matrix.
For the evaluation of the repeatability n independent replicate
test results have to be obtained under repeatability conditions at
each level. The number of replicates is most often recommended
to be n = 2. Several organisations, among which ISO and AOAC,
prefer not to repeat the experiment but recommend a blindly
coded distribution of duplicate samples so that the operator does
not know which samples are replicates and therefore is not influ-
enced to produce results which are as alike as possible. However
this set-up is only possible if the qn measurements can be per-
formed within a short time interval. Alternatively a split-level
replication can be applied. In the split-level experiment a mate-
rial with (usually two) slightly different concentrations is used.
Split-level materials are also known as Youden matched pairs.
Each of the levels is analysed once. ISO discusses this approach
in part 5 of the 5725 standard [1]. IUPAC [6] and AOAC [8] also
propose this design as an alternative.
The materials investigated and the test items that are sent to
the laboratories should be representative for those that will be
encountered in normal use. If it is not possible to cover all types
of material in the procedure then the materials for which the pre-
cision measurements are considered to apply should be clearly
mentioned, e.g. water in cheese. For a given type of material the
precision is usually related to the concentration of the analyte. In
such cases the whole range of concentration should be studied
and the precision should be reported as a function of concen-
tration. The ISO standard puts the emphasis very much on the
effect of concentration on precision and much less on the effect
of matrix effects that can occur between materials belonging to
the same type of material, which is a pity. It is for instance far
from evident that the same kind of results will be obtained for
the determination of water content in all types of cheese.
A difficult question is how to account for possible heterogene-
ity between the test items distributed to the laboratories (or even
within the test items). If it is not possible to avoid heterogeneity,
for instance when the items are solids, such as powders, the final
report should make it clear that the precision measures reported
 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
include possible heterogeneity between samples as a source of
variation. It is possible to assess heterogeneity to some extent but
this requires more extensive experiments. Indications on how to
do this can be found in [12]. Sometimes it is not possible at all to
send identical items to the participating laboratories. ISO 5725-
5 [1] gives as example leather: no two hides are identical and
between-hide variation could therefore be a serious source of
variation. To distinguish it from sources of variation due to the
measurement a hierarchical design, which requires more sam-
ples to be tested, should be used instead of the balanced uniform
level design.
Measures should also be taken to ensure stability of the
samples. For food samples for instance it may be necessary
to transport them in a deep-frozen state, to include directions
for storage of the sample if it is not analysed immediately on
receipt by the participating laboratory and on how to carry out
the thawing of the sample.
2.3. The components of precision
Each test result y can be represented as follows [1]:
y =m+B+e (1)
where m is the general average for the material analysed, B
represents the laboratory component of bias and e is the random
error, representing the residual variance. The latter is estimated
by sr2 , the repeatability variance, while the variance of B gives
2 , the between-laboratory variance.
rise to sL
The reproducibility variance is the sum of the repeatability
variance and the between-laboratory variance
2
= sr2 + sL
2
(2)
sR
Eqs. (1) and (2) state that laboratory components of biases are
systematic errors from the point of view of a single laboratory
(Eq. (1)) but random errors from the interlaboratory point of
view (Eq. (2)).
2.4. Evaluation of the data
The basic statistical model is a random effects one-way anal-
ysis of variance (ANOVA) model from which for each sample
the components of variance sr2 and sL 2 can be extracted and by
2 . The IUPAC/AOAC
using Eq. (2) it is then possible to obtain sR
protocol [6,8] states this as such. The ISO standard 5725-2 [1]
does not use this terminology, probably because it prefers to
apply a calculation method, which can be carried out by hand.
It does not explicitly follow the usual calculation scheme for
ANOVA but yields the same results.
2.4.1. Outliers
Before computing sr , sL and sR it is necessary to clean the data
from outliers. There are two possible types of outliers, namely:
(i) outlying laboratories: they deviate from the others in pre-
cision (repeatability), meaning that they deliver work of a
lesser standard than the others or the means of their results
161
are too low or too high compared to the others, indicating
unacceptable laboratory bias;
(ii) outlying results for individual laboratories at a given level:
these too can deviate either in precision or in mean value
compared to the other results at the same level.
From a statistical point of view unequal repeatabilities vio-
late a fundamental assumption of ANOVA namely equality of
variances, also called homogeneity of variances, and too low or
too high results lead to distributions that are no longer normal,
another assumption of ANOVA.
2.4.1.1. Procedures for outlier removal. The ISO 5725-2 [1]
and IUPAC/AOAC [6,8] standards describe a procedure for
outlier removal. Both procedures are very similar, the main dif-
ference being the confidence level at which outliers are detected
(see further).
Visual evaluation for consistency of results: It should be noted
that outlier rejection is not applied here (and should never be
applied) as an automatic procedure, but rather that the statistical
conclusions are only one aspect of the whole context leading to
the final decision. A graphical way to evaluate the consistency
of results and laboratories is therefore useful. ISO recommends
Mandel’s k and h statistics. These statistics can also be used to
evaluate the quality of laboratories in laboratory-performance
studies. They are obtained as follows:
si
ki =  (3)
p
s 2 /p
i=1 i
and
ȳi − ȳ
hi =  p (4)
1/(p − 1)( i=1 (ȳi − ȳ)2 )
The statistic ki is a within-laboratory consistency statistic
that compares for a selected level the standard deviation within
one laboratory to the mean standard deviation of the different
laboratories for that level.
The statistic hi is a between-laboratory consistency statistic
that measures for a selected level the standardised deviation of
the mean value obtained by laboratory i (ȳi ) from the grand mean
for that level (ȳ). It is therefore a measure of the laboratory bias.
The ki and hi values obtained in that way for the different lev-
els (samples) can be plotted grouped per laboratory or grouped
per sample together with the critical values of the k and h statis-
tics at the significance levels of 1% and 5%. These critical values,
as a function of the number of laboratories and the number of
replicates per laboratory (for k), can be found in the ISO guide-
line 5725-2 [1]. An example is given in Fig. 1 which is adapted
from ref [13]. It can be observed from Mandel’s k plot that lab-
oratory 4 almost consistently shows the highest repeatability
while Mandel’s h plot indicates that laboratory 7 tends to report
higher values than the other laboratories. However a decision is
not taken before the statistical tests described in the following
sections have been performed.
Test for homogeneity of variances: The homogeneity of vari-
ances is tested with the Cochran test. This test investigates if the
162 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
Fig. 1. Mandel’s k (a) and h (b) statistics (adapted from [12]). The lines represent
the 1% (—) and 5% (- - -) limit of the statistics.
variances obtained by the p laboratories at a certain level (si2 )
are not too dissimilar by calculating the following statistic:
s2
C = pmax 2
i=1 si
2
with smax the largest of the p variances. Critical C-values are
tabulated [1,8].
This test can be recycled, which means that if, in a first round,
laboratories or values in an individual laboratory are rejected,
the Cochran test is applied again to the remaining laboratories.
Test for outlying laboratory averages: The averages of the
results for the same level obtained by the p laboratories are tested
for outliers by the Grubbs’ tests. There are several variants of
these tests. First the single outlier test is applied, which means
that it is investigated if there is one average, which is too high
or too low, compared to the others. The test can be performed
by calculating:
(ȳi − ȳ)
G=
s sr2 =
with ȳi the mean value for the suspected outlying laboratory
(either the lowest or highest result), ȳ the grand mean and s the
standard deviation of the p mean values. The absolute value of
G is compared with the critical values for this test, which are
tabulated [1].
IUPAC/AOAC perform the single Grubbs’ test by calculat-
ing the percentage reduction in the standard deviation when the
suspect laboratory is deleted:
 
1 − s1
R = 100 (7)
s standard [1].
with s as defined above and s1 the standard deviation obtained
by removal of the suspected outlying laboratory (either the
lowest or the highest result). This value is compared in the
usual way with tabulated R-values. This is equivalent with
the test of Eq. (6) because their critical values are related
[14].
If the single outlier test is negative, the Grubbs’ test for
two outlying observations, which can detect two simultaneous
high or two simultaneous low values, is applied. Again different
equivalent tests are applied by ISO and IUPAC/AOAC. More-
over a procedure proposed in the IUPAC/AOAC standard also
detects one high and one low value occurring simultaneously.
The reader is referred to the guidelines for more information on
these tests.
Stragglers and outliers: In contrast with most other statistical
tests, the decision is not taken at the 5% level of confidence, nor
is it taken automatically. In this context, outliers are eliminated at
the 1% level (ISO [1]) or 2.5% (IUPAC/AOAC [6,8]). ISO calls
outliers detected at the 5% level stragglers and they are flagged,
but are included in the analysis except when an assignable cause
for the outlying result is found.
In the procedure for outlier treatment in both the ISO and
IUPAC/AOAC guidelines, the Cochran and Grubbs’ tests are
sequentially applied. The procedure stops if no further outliers
are flagged or if too many laboratories have to be eliminated.
IUPAC/AOAC specify that no more than 2/9 laboratories should
be eliminated since this might indicate problems with the
method.
Robust statistics: The outlier tests described higher are based
(5) on a normal distribution of errors, which can be assumed but not
proven and decisions have to be taken by the panel. To avoid this,
it is possible to apply so called robust statistics. ISO describes
them in part 5 of the 5725 standard [1] as an alternative to the
above statistical procedures. In this case outliers are not deleted,
but the statistics are such that extreme values have lower weights
in the calculations.
2.4.2. Calculation of the variances
The ANOVA table as well as the calculation of the differ-
ent variances to be performed for each material (sample) is
summarised in Table 1.
Simple hand calculations are also possible and are carried out
as follows when all ni = n = 2.
1  2
p
(6)
di (i = 1, . . . , p) (8)
2p
i=1
and
 p 
1  s2
2
sL = (ȳi − ȳ)2 − r (9)
p−1 2
i=1
where di is the difference and ȳi the mean of the tworesults
obtained by laboratory i and ȳ the grand mean, i.e. ȳi /p.
2 have been computed, s2 can be obtained from
When sr2 and sL R
Eq. (2). Similar equations for n ≥ 2 can be found in the ISO
 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167 163

Table 1
ANOVA table
Source Mean squares Estimate of
p 2
ni (ȳi −ȳ¯ )
Laboratory MSL = i=1
σr2 + n̄σL2
p (p−1)
ni
(yij −ȳi )2
Residual (=repeatability) MSr = i=1 j=1
σr2
p p 2

(N−p)
 
p

ni − i=1
ni
with n̄ = 1
(p−1) p and N = ni
ni
i=1
i=1 i=1
Calculation of variances
The repeatability variance
sr2 = MSr d.f. = (N − p)
Variance component between laboratories (between-laboratory variance)
sL2 = MSL −MSr 2 < 0 set s2 = 0
if sL
n̄ L
Reproducibility variance
sR2 = s2 + s2
r L

2.4.3. Repeatability and reproducibility as a function of
concentration
A last step is to investigate if there is a relationship between
sr and the concentration m and between sR and m. Indeed, it is
known that precision measures depend on concentration. The
ISO document recommends testing the following models (b0
and b1 are the regression coefficients):
sr = b1 m
sr = b0 + b1 m
log sr = b0 + b1 log m
Model (10), a straight line through the origin, means that
the coefficient of variation is constant. No formal procedures
are described to decide which of the equations fits best. The
simplest of the models that is found to fit the data sufficiently
well is adopted for further use. For all models weighted regres-
sion methods, which however are quite complicated, are also
described. Similar models are developed for sR .
Other relations than those described by Eqs. (10)–(12) can
of course also be established. If the precision is found not to
depend on the concentration the reported precision measures, sr
and sR, are obtained from the variance pooled for the different
concentrations.
2.5. What precision to expect?
An important question in the evaluation of the precision is
what values of sr and sR can be considered acceptable. Horwitz
et al. [15] and Boyer et al. [16] carried out interesting work in
this context. They found from the evaluation of a large number of
collaborative studies that there was a strong relationship between
the concentration of an analyte and the observed precision. Inter-
laboratory collaborative studies on various commodities ranging
in concentration from a few percent (salt in foods) to the ppb
(ng/g) level (aflatoxin M1 in foods) but also including stud-
ies on, for example, drug formulations, antibiotics in feeds and
pesticide residues were included in their study.
They found the following relationship between the relative
reproducibility standard deviation RSDR (%) and the concentra-
tion expressed as a decimal fraction:
RSDR (%) = 2(1−0.5 log C) (13)
This equation states that RSDR approximately doubles for
every 100-fold decrease in concentration, starting at 2% for
C = 1, i.e. for a 100% pure sample. This means, for instance,
(10)
that when an assay is carried out by determining the content of
(11) a single component sample, a relative reproducibility standard
deviation of 2% is to be expected.
(12) Eq. (13) can be rewritten as:
RSDR (%) = 2C−0.1505 or as σR = 0.02C0.8495
or as log σR = 0.8495 log C − 1.699 (14)
It was also concluded from the Horwitz’s initial study that
the RSDR only depends on the concentration and not on the
analyte or method. The author [15] concludes that RSDR values
are suspect, when they exceed by more than a factor 2 what is
expected from Eq. (13). The ratio between the reproducibility
obtained and the one expected from Eq. (13) is sometimes called
the HORRAT (short for Horwitz ratio) and it should not exceed
2.
Another interesting result of the Horwitz study is that the
corresponding repeatability measure (RSDr ) is generally one-
half to two-thirds of the reproducibility measure RSDR .
In a recent article the Horwitz function (Eq. (13)) has been
revisited [16] and the authors conclude that it is a useful tool to
summarise historical data but that it is an unsuitable criterion
to assess interlaboratory comparisons. Horwitz himself already
showed [17] that for very low concentrations the estimates of the
reproducibility standard deviation, σ R, are somewhat better, i.e.
lower, than expected from the above equations. He concluded
that for such concentrations σ R = C/3 or log σ R = log C − 0.477.
Thompson came to a similar conclusion [18], but with different
coefficients, and, moreover, also found a divergence from the
first Horwitz equation at high C (>0.138, i.e. 14%) [19]. He
164 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167

proposes the following set of equations: The bias is estimated as δ̂ = ȳ¯ − μ which is the difference
between the grand mean (ȳ¯ ) and the accepted reference value
0.22C if C < 1.2 10 − 7
(μ) and the standard deviation associated with this bias estimate
σR = 0.02C0.8495 if 1.2 10 − 7 < C < 0.138 (14) is given by [1]:
0.01C0.5 if C > 0.138
3. Method performance studies—bias experiments
3.1. Aim and definitions
Method performance studies are mainly conducted with the
aim to evaluate the precision of the measurement method. They
are much less applied for the evaluation of the bias. However
interlaboratory experiments are required if one wants to estimate
the method bias which is the systematic error inherent to the
method. Indeed there are two components of bias, the method
bias and the laboratory bias which is the bias introduced by the
way a specific laboratory applies an unbiased method. A single
laboratory that by an internal method validation evaluates the
bias cannot distinguish the method bias from the laboratory bias.
A collaborative trial is needed for this. Part 4 of the ISO 5725
guidelines [1] describes how to evaluate the bias of a standard
measurement method.
3.2. Organisation of the bias experiment
The experimental set-up described in the ISO guidelines is
very similar to that of precision experiments. The main differ-
ences are (i) the requirement that a reference value is known
for example by using reference materials or materials whose
properties are known or are obtained from analysis by another
method, known to be unbiased and (ii) the number of laborato-
ries and of replicates per laboratory. The ISO standard 5725-4
[1] includes a table that allows to derive the minimum number
of laboratories, p, and of replicates, n, in order to be able to con-
clude that there is no bias with 95% probability (α = 0.05) but
also to be able to detect a predefined bias with 95% probability
(β = 0.05). Taking the β-error into account is important to avoid
that a relevant bias goes unnoticed due to a too low number of
measurements [20,21].
IUPAC/AOAC [6,8] do not consider separate bias experi-
ments and specify that if a true or assigned value is not known a
consensus value must be obtained from the collaborative study
itself.
3.3. The evaluation and interpretation of the data
The data are evaluated for outliers as described for the
precision experiment and the repeatability variance (sr2 ) and
reproducibility variance (sR 2 ) are estimated from the experi-
ments. They are compared with the variances, σr2 and σR2 ,
obtained from an independent precision experiment, if avail-
able. The latter variances σr2 and σR2 are used in the assessment
of the bias if the test applied for the comparison is not significant.
In the other case it is necessary to examine the reason for the
discrepancy and it might be necessary to repeat the experiments.
 
2 − (1 − 1/n)s2
sR r
sδ̂ = (15)
p
A 95% confidence interval is calculated that allows testing
the significance of the bias.
4. Proficiency studies
4.1. Aim
Proficiency studies assess the performance of individual lab-
oratories by comparing their measurement results with those
from other laboratories or with known or assigned values. They
are generally operated on a regular basis, ranging most often
from 2 to 6 rounds per year. Proficiency studies are also one
of the elements that are considered in the assessment for the
accreditation of laboratories and as a result of this develop-
ment many proficiency-testing providers or specific schemes
have now themselves been accredited.
4.2. Organisation of a proficiency study
There are several guidelines and protocols for the set-up
and evaluation of proficiency studies from among others ISO
[22,23], Eurachem [24] and ILAC (International Laboratory
Accreditation Cooperation) [25]. The most recent is the Inter-
national Harmonized Protocol for the Proficiency Testing of
Analytical Chemistry Laboratories, which is the result of a coop-
eration between ISO, AOAC and IUPAC [3]. It is a revision of
an earlier Harmonized Protocol [2].
The proficiency scheme is organised by the scheme provider
in cooperation with an advisory group consisting of persons
with knowledge of the methods and procedures involved and
including a statistical expert.
The materials investigated or the test items that are sent to
the laboratories should be representative for those that are rou-
tinely tested by the participating laboratories. The material must
be sufficient homogeneous and stable. The revised Harmonized
Protocol [3] provides improved homogeneity testing and also
includes requirements concerning the repeatability for the ana-
lytical methods used to check homogeneity. It also describes
pros and cons of several methods to determine an assigned value
and particularly pays a lot of attention to the estimation of the
assigned value from the consensus of the participant’s results
since this is the most frequently used and less costly method.
Participating laboratories are generally free in the choice of
the analytical method but it should be consistent with their nor-
mal routine practice. It is obvious that the method should not be
adapted for the proficiency exercise.
ISO [22] nor the revised Harmonized Protocol [3] explicitly
specify the number of laboratories to be included in the scheme
but the latter warns that with a small number of participants (<15)
 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
the uncertainty on the consensus value might be unacceptably
high and consequently jeopardise the interpretation the z-scores
(see below).
4.3. Evaluation of the data
When the assigned value is determined as a consensus value
the influence of extreme values should be minimised and there-
fore the data should be cleared from outliers. Notice that outliers
should only be removed for the calculation of summary statistics
but they should be considered in the evaluation of the per-
formance of the individual laboratories (see below). Graphical
techniques (such as Mandel’s h statistic but also box plots [26])
and statistical outlier tests such as the Grubbs’ tests are applied
(see Section 4.2).
The most widespread used performance indicator of the lab-
oratories is the z-score that is calculated as:
x−X
z= (16)
s
with x the result obtained by a laboratory, X the assigned value
(see above) and s a standard deviation. The revised Harmonized
Protocol describes several ways in which the latter parameter
can be obtained. It can for example be determined in a profi-
ciency testing as the standard deviation of the laboratory results,
after the elimination of outliers. However it is to be preferred
that s is a set target value of precision that could e.g. be derived
from the precision to perform a certain task, from method perfor-
mance studies or from the Horwitz curve [15]. The Harmonized
Protocol [3] calls this s the “fitness-for-purpose based standard
deviation for proficiency assessment”.
With good estimates of X and s, z corresponds to a standard-
ised variable and the interpretation is based on the assumption
that the z-scores are normally distributed with a mean of zero and
a standard deviation of one. Consequently one expects that |z|>2
in only 4.55% of the cases and |z|>3 in only 0.27%. Therefore
the z-scores are generally interpreted as follows:
|z| < 2 satisfactory performance
2 ≤ |z| ≤ 3 questionable performance
|z| > 3 unsatisfactory performance
It is not unusual that more than one test item is analysed within
one round of a proficiency test. The combination of several z-
scores to summarise the overall performance of a laboratory is
however not recommended to avoid an individual significant
score to be systematically masked [3].
In a split-level experiment useful information can also be
obtained from a Youden plot, which consists of plotting the
results of two samples with similar concentration against each
other [27,28]. An example, adapted from ref [29] is given in
Fig. 2.
The median values for both samples are also plotted. They
divide the plot into four quadrants and their intersection point
is accepted as the most probable value. With only random error,
one expects the points to be equally distributed over the four
quadrants. The circle represents the 95% limit around the origin
165
Fig. 2. Youden plot for data adapted from [28]. y1 and y2 are the concentrations
of clenbuterol as determined by 10 laboratories in two urine samples (Youden
pairs).
of the plot. As occurs frequently, here too there are more points
in the upper right and lower left quadrant which indicates that
several laboratories report results that are too high or too low,
which means that there is a systematic error. If all laboratories
apply the same method these systematic errors are laboratory
biases.
Proficiency testing can also serve other purposes than assess-
ing the proficiency of laboratories as illustrated by Thompson et
al. [30]. Historical data collected over several years in a profi-
ciency scheme in which many laboratories used the Kjeldahl or
the Dumas method for the determination of protein in various
foodstuffs were used to examine the bias between both methods.
The findings were confirmed later in a designed interlaboratory
experiment [31].
5. Interlaboratory studies and uncertainty
Analysts and metrologists in general know that the measure-
ment result is only an approximation of the true value. To be
complete an uncertainty statement should therefore accompany
the measurement result. Although this has been accepted for a
very long time, it is only relatively recently that guidelines about
how to produce a quantitative statement of uncertainty have been
published. The basic document concerning the uncertainty of
measurement results is the Guide to the Expression of Uncer-
tainty [5], often called the GUM. It was first published in 1993
and corrected in 1995 by ISO. It is the result of a collaboration
from several organisations in a task force called ISO/TAG4.
The GUM proposes an error-propagation or error-budget
approach to estimate the uncertainty related to a measurement
result. The analytical process is decomposed into its compo-
nents, the uncertainty related to these components are separately
quantified in the form of a standard deviation and then com-
bined into an error-budget. This also includes an estimation of
166 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
all sources of systematic error which according to the GUM
should be corrected for if significant. The advantage of this
approach is that once the contributions of individual sources of
uncertainty have been quantified it may be possible to adapt
the method used such that the more important contributions
become smaller, finally leading to a method that yields results
with less uncertainty. However it soon became clear that the
principles of the GUM could relatively easily be applied to phys-
ical measurements (such as weights and volumes, or electrical
measurements), but much less easily to chemical ones.
As an alternative method to measure uncertainty for chemical
measurements the Analytical Methods Committee of the Royal
Society of Chemistry (AMC) [32] proposed an approach, which
is based on precision data assessed in an interlaboratory study.
It is based on reproducibility data and fulfils the requirement to
include also systematic errors, because as explained in Section
2.3 systematic errors within a laboratory, such as the laboratory
bias, become random errors if a population of laboratories is
considered. Ellison has discussed several issues related to the
use of interlaboratory data in the estimation of measurement
uncertainty [33].
The reproducibility standard deviation from a method-
performance experiment includes all uncertainty contributions
that randomly occur in a population of laboratories but it does
not include the uncertainty associated with the method bias.
This uncertainty consists of the variation of the bias estimate
as expressed by its standard deviation, sδ̂ (see Eq. (15)), and the
uncertainty associated with the accepted reference value, uref . It
follows from Eq. (15) that the former will be small compared
to the reproducibility standard deviation if a large number of
laboratories are included in the interlaboratory trial.
The uncertainty associated with a measurement result y there-
fore becomes:

uy = sR 2 + u2
ref (17)
A further simplification of this expression is possible when
u2ref is negligible in comparison to the reproducibility variance.
O’Donell and Hibbert, using simulated data, have compared
several methods to take account of the bias in the measure-
ment uncertainty. They conclude that correcting for the bias is
consistently the best throughout the range of biases evaluated
[34].
When the method has been validated in an interlaboratory
method-performance study, the reproducibility standard devia-
tion can be applied to assess the uncertainty due to (lack of)
precision. The laboratory that applies the method is then sup-
posed to show that it is sufficiently proficient: reproducibility
takes into account intermediate precision (i.e. repeatability and
in-house sources of variation due to e.g. time or different oper-
ators) and laboratory biases and it has to be shown that the
laboratory’s repeatability is comparable to the one obtained in
the interlaboratory study.
Reproducibility is supposed to include sources of variance
such as different times and operators. Therefore the laboratory
should show that its intermediate precision is not larger than
the reproducibility standard deviation. As said higher, repro-
ducibility takes laboratory biases into account and therefore the
laboratory must have evidence that its lab bias is lower than the
reproducibility standard deviation. It may derive this evidence
from analysing relevant reference materials or in other ways,
such as by participating in proficiency studies.
Some applications of interlaboratory data in the estima-
tion of the uncertainty of chromatographic methods have been
described [35,36].
Notice however that, dependent on the situation, other pre-
cision measures than the reproducibility standard deviation can
be included in the uncertainty statement. Operational definitions
of uncertainty have been proposed that take into account differ-
ent sources of uncertainty according to the situation considered
[37]. Barwick and Ellison propose a guide for the evaluation of
uncertainty from in-house validation data [38].
6. Conclusion
Interlaboratory studies serve several needs and aspects of the
quality management of chemical measurements. They allow the
validation of analytical methods, assessing the proficiency of
individual laboratories, estimating measurement uncertainty and
certifying reference materials in a wide range of application
fields (ranging from clinical chemistry and microbiology over
environmental science to food analysis).
Several guidelines, intended very often, for application in dif-
ferent areas, are therefore very general. They have to be adapted
for specific situations so that they can easily be integrated to fit
particular customer’s needs.
Acknowledgement
This article is partly based on a draft text on the same sub-
ject, prepared in collaboration with Professor D.L. Massart who
unfortunately passed away on 26 December 2005.
References
[1] Accuracy (trueness and precision) of measurement methods and
results—parts 1–6 (ISO 5725), International Organisation for Standard-
ization (ISO), Geneva, 1994.
[2] IUPAC, The International Harmonized Protocol for the proficiency test-
ing of (chemical) analytical laboratories, Pure Appl. Chem. 65 (1993)
2123–2144.
[3] IUPAC, The International Harmonized Protocol for the proficiency testing
of analytical laboratories, Pure Appl. Chem. 78 (2006) 145–196.
[4] General requirements for the competence of reference material producers
(ISO 34), International Organisation for Standardization (ISO), Geneva,
2000.
[5] Guide to the expression of uncertainty in measurements, International
Organisation for Standardization (ISO), Geneva, 1995.
[6] IUPAC, Protocol for the design, conduct and interpretation of method-
performance studies, Pure Appl. Chem. 67 (1995) 331–343.
[7] IUPAC, Protocol for the design, conduct and interpretation of collaborative
studies, Pure Appl. Chem. 60 (1988) 855–864.
[8] AOAC, Guidelines for Collaborative Study, Procedures to validate char-
acteristics of a method of analysis, J. AOAC Int. 78 (1995) 143A–
160A.
[9] Guidelines for CIPAC collaborative study procedures for assessment
of performance of analytical methods, Collaborative International Pes-
 Y. Vander Heyden, J. Smeyers-Verbeke / J. Chromatogr. A 1158 (2007) 158–167
ticides Analytical Council (CIPAC), UK (1989) available online:
http://www.cipac.org/howprepa.htm.
[10] Precision of test methods—determination of repeatability and reproducibil-
ity for a standard test method by inter-laboratory tests, (ISO 5725),
International Organisation for Standardization (ISO), Geneva, 1986.
[11] J.O. De Beer, B.M.J. De Spiegeleer, J. Hoogmartens, I. Samson, D.L.
Massart, M. Moors, Analyst 117 (1992) 933.
[12] Certification of reference materials—general and statistical principles
(ISO 35), International Organisation for Standardization (ISO), Geneva,
1989.
[13] Y. Vander Heyden, J. Saevels, E. Roets, J. Hoogmartens, D. Decolin, M.G.
Quaglia, W. Van den Bossche, R. Leemans, O. Smeets, F. Van de Vaart, B.
Mason, G.C. Taylor, W. Underberg, A. Bult, P. Chiap, J. Crommen, J. De
Beer, S.H. Hansen, D.L. Massart, J. Chromatogr. A 830 (1999) 3.
[14] P.C. Kelly, J. Assoc. Off. Anal. Chem. 71 (1988) 161.
[15] W. Horwitz, L.R. Kamps, K.W. Boyer, J. Assoc. Off. Anal. Chem. 63 (1980)
1344.
[16] K.W. Boyer, W. Horwitz, R. Albert, Anal. Chem. 57 (1985) 454.
[17] T.P.J. Linsinger, R.D. Josephs, Trends Anal. Chem. 25 (2006) 1125.
[18] M. Thompson, P.J. Lowthian, J. AOAC Int. 80 (1997) 676.
[19] M. Thompson, Analyst 125 (2000) 385.
[20] C. Hartmann, J. Smeyers-Verbeke, W. Penninckx, Y. Vander Heyden, P.
Vankeerberghen, D.L. Massart, Anal. Chem. 67 (1995) 4491.
[21] C. Hartmann, J. Smeyers-Verbeke, D.L. Massart, J. Pharm. Biomed. Anal.
17 (1998) 193.
[22] Proficiency testing by interlaboratory comparisons—part 1: development
and operation of proficiency testing schemes (ISO 43), International Organ-
isation for Standardization (ISO), Geneva, 1994.
[23] Statistical methods for use in proficiency testing by interlaboratory compar-
isons (ISO 13528), International Organisation for Standardization (ISO),
Geneva, 2005.
[24] Selection, use and interpretation of proficiency testing schemes,
EURACHEM Nederland and Laboratory of the Government Chemist
(LGC), UK (2000) available online: http://www.eurachem.ul.pt/guides/
ptguide2000.pdf.
167
[25] Guidelines for the requirements for the competence of providers of
proficiency testing schemes, ILAC Guide G13, International Lab-
oratory Accreditation Cooperation (ILAC) (2000) available online:
http://www.ilac.org/publicationslist.html.
[26] D.L. Massart, B.G.M. Vandeginste, L.M.C. Buydens, S. De Jong, P.J. Lewi,
J. Smeyers-Verbeke, Handbook of Chemometrics and Qualimetrics: Part
A, Elsevier, Amsterdam, 1997.
[27] W.J. Youden, Int. Qual. Control 15 (1959) 24.
[28] J. Mandel, T.W. Lashof, J. Qual. Technol. 6 (1974) 22.
[29] H. Beernaert, Ringtest clenbuterol in urine, IHE Report, Instituut voor
Hygiene en Epidemiologie (IHE), Brussels, September 1992.
[30] M. Thompson, L. Owen, K. Wilkinson, R. Wood, A. Damant, Analyst 127
(2002) 1666.
[31] M. Thompson, L. Owen, K. Wilkinson, R. Wood, A. Damant, Meat Sci. 68
(2004) 631.
[32] AMC (Analytical Method Committee), Analyst 120 (1995) 2303–2308.
[33] Ellison, Accred. Qual. Assur. 3 (1998) 95–100.
[34] G.E. O’Donnell, D.B. Hibbert, The Analyst 130 (2005) 721.
[35] P. Dehouck, Y. Vander Heyden, J. Smeyers-Verbeke, D.L. Massart, J. Crom-
men, Ph. Hubert, R.D. Marini, O.S.N.M. Smeets, G. Decristoforo, W. Van
de Wauw, J. De Beer, M.G. Quaglia, C. Stella, J.-L. Veuthey, O. Estevenon,
A. Van Schepdael, E. Roets, J. Hoogmartens, Anal. Chim. Acta 481 (2003)
261–272.
[36] P. Dehouck, Y. Vander Heyden, J. Smeyers-Verbeke, D.L. Massart, R.D.
Marini, Ph. Hubert, P. Chiap, J. Crommen, W. Van de Wauw, J. De Beer, R.
Cox, G. Mathieu, J.C. Reepmeyer, B. Voigt, O. Estevenon, A. Nicolas, A.
Van Schepdael, E. Adams, J. Hoogmartens, J. Chromatogr. A 1010 (2003)
63.
[37] E. Hund, D.L. Massart, J. Smeyers-Verbeke, Trends Anal. Chem. 20 (2001)
394.
[38] V.J. Barwick, S.L.R. Ellison, Protocol for uncertainty evaluation from
validation data, VAM Technical Report No. LGC/VAM/1998/088,
Valid Analytical Measurement (VAM) Programme, Laboratory of
the Government Chemist (LGC), Teddinton, 2000; available at:
http://www.vam.org.uk/publications/publicationdocs/315.pdf.