See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/221954344

Limitations of the application of the Horwitz

equation

Article in TrAC Trends in Analytical Chemistry · December 2006

DOI: 10.1016/j.trac.2006.11.002

CITATIONS READS

22 1,846

2 authors:

Thomas P J Linsinger Ralf D Josephs

European Commission Bureau International des Poids et Mesures
121 PUBLICATIONS 2,346 CITATIONS 72 PUBLICATIONS 895 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

FP7 NanoLyse “Nanoparticles in Food: Analytical methods for detection and characterization”, Jan
2010 – Sep 2013, grant agreement no. 245162 View project

Peptide/Protein Purity View project

All content following this page was uploaded by Thomas P J Linsinger on 09 October 2017.

The user has requested enhancement of the downloaded file.

Trends in Analytical Chemistry, Vol. 25, No. 11, 2006 Trends

Limitations of the application

of the Horwitz equation
Thomas P.J. Linsinger, Ralf D. Josephs

We revisit the basic assumptions of the Horwitz equation using the example precision predicted by the Horwitz equa-
of mycotoxin assays. Prediction intervals from the Horwitz equation often tion for a measuring method at that
span one order of magnitude. Including this variation in calculation of Horrat particular level of analyte and is calcu-
values would lead to a range of values exceeding 2, which is often used as a lated as:
criterion to assess interlaboratory comparisons, so we question the suitability
of Horrat value for this purpose. In addition, available analytical data show Horrat ¼ RSDR; measured =RSDR; Horwitz predicted
significant improvement in reliability over time, which casts serious doubts ð2Þ
on the applicability of the Horwitz equation for current analytical methods.
A Horrat value of 1 indicates satisfactory
We discuss the use of the Horwitz equation in the analytical laboratory,
interlaboratory precision, whereas a value
and conclude that it is not suitable for estimating uncertainties, as required
of 2 indicates unsatisfactory precision (i.e.
by ISO 17025.
one that is too variable for most analytical
The Horwitz equation can be a valuable summary of historical data of
purposes or where the variation obtained
analytical performance. However, it should not be used as a performance
is greater than that expected for the type
criterion due to:
of method employed according to
 shortcomings of the basic model;
Horwitz).
 uncertainty in the values determined using it; and,
Furthermore, the values predicted from
 its incompatibility with accepted methods for the determination of
the Horwitz equation have been used in
measurement uncertainty, as required by ISO 17025.
legislation to set acceptance limits for
We recommend that, instead of using the Horwitz equation, there should
analytical methods (e.g. [3]). It has even
be a proper identification of all components of uncertainty of measurement
been suggested recently to use the results
and reasonable estimation, as stipulated by the ‘‘Guide to the Expression of
from the Horwitz equation as an estima-
Uncertainty in Measurements’’ (GUM).
tion of measurement uncertainties [4].
ª 2006 Elsevier Ltd. All rights reserved.
Subsequent analysis of more datasets by
Keywords: Horrat value; Horwitz equation; Interlaboratory comparison; ISO 17025; Horwitz himself and others [5–7] fre-
Measurement uncertainty; Quality assurance quently showed significant deviations
from the values predicted by the original
1. Introduction Horwitz function.
Thomas P.J. Linsinger*
In this article, we will discuss some basic
EC-JRC, Institute for Reference
Materials and Measurements In 1980, Horwitz et al. published an assumptions of the Horwitz equation and
(IRMM), Retieseweg 111, evaluation of 1000 interlaboratory com- will compare results from mycotoxin
B-2440 Geel, Belgium parisons that led them to conclude that interlaboratory studies spread over more
there is a fixed relationship between ana- than 30 years. Furthermore, we will point
Ralf D. Josephs
lyte level and reproducibility standard to the error made by using regression
Bureau International des Poids
et Mesures (BIPM), Pavillon de deviation (RSDR) [1,2]. According to this parameters in the prediction of future
Breteuil, F-92312 Sèvres Cedex, analysis, the relationship between RSDR events. Finally, we will show why use of
France and the analyte level c is: the Horwitz equation as a performance
criterion contradicts modern practice in
RSDR ½% ¼ 2ð10:5log10 cÞ ð1Þ
quality management and is not compliant
irrespective of the kind of analyte, matrix with guides to the estimation of measure-
or method. Equation (1) has been widely ment uncertainties, such as the Eurachem
*
used to assess the quality of interlabo- Guide ‘‘Quantifying Uncertainty in Ana-
Tel.: +32 14 571 956;
Fax: +32 14 571 548;
ratory comparisons using the Horrat lytical Measurement’’ [8] and the ‘‘Guide
E-mail: thomas.linsinger@ value, which gives a comparison of the to the Expression of Uncertainty in
ec.europa.eu actual precision measured with the Measurements’’ (GUM) [9].

0165-9936/$ - see front matter ª 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.trac.2006.11.002 1125
Trends
Mycotoxin data have been used because the original
RSDR values are available in the literature and more
subsequent data are available for this group of analytes.
However, this does not limit the generality of our con-
clusions, as data for other analyte groups show similar
scatter.
2. Mathematical limitations of the Horwitz
equation
In his original paper and in some of the follow-up papers,
Horwitz demonstrated that the concentration level is the
most important variable in explaining the reproducibility
of interlaboratory-comparison studies. However, he
never stated that all variation can be explained by this
parameter, which is a crucial prerequisite for using the
equation as a performance criterion.
Database 1 from Horwitzs paper on mycotoxin assays
[5] was transferred to Statistica 7.0 in order to demon-
strate this difference. A regression line of RSDR versus
2ð10:5log10 cÞ was calculated (Fig. 1). The regression
parameters were calculated and are shown in Table 1.
Statistical regression reveals that data are homosce-
dastic and confirms the former statement of Horwitz
et al. [5] that the mycotoxin assays in the period 1968–
1992 show a higher variability than predicted from the
Horwitz equation. The slope of the regression line RSDR
vs. 2ð10:5log10 cÞ is significantly different from 1 on a 95%-
confidence level (two-sided t-test), while the intercept is
RSDR for mycotoxin assays
300
250
200
150
RSDR
100
50
-50
0 10 20
1 mg/kg 100 µg/kg
Figure 1. Mycotoxin data from [5]. Solid line: regression line for mycotoxins assays; inner dotted line: 95%-confidence interval of the regression
line; outer dotted line: 95%-prediction interval for new observations; dashed line: regression line of the traditional Horwitz equation.
1126 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
Table 1. Regression parameters of the mycotoxin data from [5]
Parameter Value
Slope ± s 1.291 ± 0.087
Intercept ± s 4.66 ± 3.01
Coefficient of determination (r2) 0.218
Standard error of the estimation (sy x) 26.80
not significantly different from zero. This difference is
also visible in Fig. 1, where the line from the classical
Horwitz equation is completely outside the 95%-confi-
dence band of the regression of the mycotoxin assays.
However, more revealing is the coefficient of determi-
nation of 0.22. This indicates that only 22% of the
variance of the RSDR can be explained by the Horwitz
equation. Although concentration certainly influences
the reproducibility, nearly 80% of the variance remains
independent of 2ð10:5log10 cÞ and is not explained by the
Horwitz equation. This poor fit is also reflected in the
very broad prediction bands in Fig. 1.
This does not matter as long as the evaluation stopped
at the regression (i.e. if the conclusion had been that the
parameter ‘‘concentration’’ significantly influences
comparability of results). However, the Horwitz equation
has been abused to predict RSDR values and to set per-
formance criteria. To illustrate the fallacy of setting
regression equal to prediction, RSDR values for three
different concentrations were calculated based on the
regression parameters shown in Table 1. Furthermore,
30 40 50 60 70 80 90
(1-0.5logc)
2
10 µg/kg 1 µg/kg 100 ng/kg
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006 Trends

the half-width of the 95%-prediction intervals (PIs) for predicting an RSDR of 87 ± 52% (0.1 lg/kg) can be
the respective RSDR values were calculated according to doubted.
Equation (3): While the prediction error makes the use of Horrat
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi values to judge the quality of intercomparisons doubtful,
2
nþ1 ðx  xÞ it invalidates its use for setting performance criteria, as
PI ¼ t0:95;n2  syx  þP 2 ð3Þ
n ðxi  xÞ the predictive uncertainty should be taken into consid-
eration. Two examples demonstrate this:
syx standard error of the estimation
 rather than stating ‘‘at a mass fraction of 10 lg/kg,
n number of data
RSDR should be 46% or better’’, the correct statement
x x-value for which the prediction interval is esti-
should be ‘‘RSDR should be 46 ± 53%’’; and, simi-
mated
larly,
x average x+value of the regression line
 for a concentration of 200 lg/kg, the correct state-
xi individual x-values of the regression line
ment should be ‘‘RSDR should be 31 ± 52%’’.
The limitations of this approach are obvious from the
These prediction limits are the limits between which an
prediction bands, as shown in these examples. It
RSDR for a given concentration can be expected. Pre-
furthermore suggests that ‘‘anything goes’’, a view that
dicted RSDR values and their respective prediction
is unlikely to be taken by any legislator. If this is why the
intervals for three different concentrations are shown in
prediction intervals have not been used so far, one
Table 2.
should not forget that not mentioning the uncertainty of
By definition, PIs cannot be higher than their corre-
the prediction does not eliminate it – it is just invisible.
sponding RSDR values because this would result in
The Horwitz equation therefore does not allow method
negative RSDR values. The occurrence of such cases
performance to be predicted with any reasonable
(e.g., Table 2 for 10 lg/kg) clearly demonstrates the
certainty.
mathematical limitations of the Horwitz approach.
In general, severe deviations from the predicted RSDR
value can occur, as shown in Table 2. For a concen-
3. Methodological limitations of the Horwitz
tration of 15 lg/kg, more than 1 out of 20 intercom-
equation
parisons will have an RSDR of more than twice the
predicted value. Transformed into the Horrat value, 1
In addition to the mathematical limitations, the Horwitz
out of 20 intercomparisons will have a Horrat value >2,
approach disregards other crucial prerequisites.
which is generally regarded as unsatisfactory. The situ-
Analytes, matrices, methods and time are considered as
ation is even worse for higher concentrations because
irrelevant for method reproducibility [10]. The approach
more than 12% of intercomparisons can be expected
is derived from regression analysis of many interlabo-
to have Horrat values >2 at concentrations above
ratory comparison studies, but is counter-intuitive and is
200 lg/kg.
contradicted by everyday experience in the laboratory.
The situation is apparently better at lower concen-
trations (e.g. 0.1 lg/kg), where the upper 95%-predic-
tion limit is ‘‘only’’ a factor of 1.6 above the predicted
3.1. Influence of analytes
value. However, 0.1 lg/kg corresponds to an x-value of
It is a widespread observation that the kind of analyte
64 in Fig. 1 and deviations from the predicted curve start
affects repeatability and reproducibility of a measure-
to become particularly severe at this concentration level.
ment procedure. This is even the case for closely related
Only at concentrations below 5.4 lg/kg is the 95%-
analytes, such as trace metals. Evaluation of 13 profi-
prediction interval smaller than twice the RSDR pre-
ciency tests of trace metals in water showed that the
dicted. This means that only below this concentration is
reproducibility of As is worse than that of Cd, even if the
it statistically valid to state that Horrat values >2 are
concentration levels of Cd are on average one order of
unsatisfactory. Horrat values >2 can be expected for all
magnitude below those of As [11]. Horwitz himself dis-
higher concentrations. However, the usefulness of
cussed the influence of the analyte [12], but this finding
was afterwards neglected. Interestingly enough, Horwitz
concluded in his paper on mycotoxins [5] that myco-
Table 2. Calculated RSDR and their confidence limits for toxin assays are less reproducible than other assays.
mycotoxins Apart from the intrinsic problems connected with
some analytes, there can be another influence of the
Concentration RSDR, predicted ± PI (%)
analyte; namely, that the concentration levels differ from
10
1 Æ 10 (0.1 lg/kg) 87 ± 53 analyte to analyte. Proficiency tests focus on analytes in
1 Æ 109 (1 lg/kg) 63 ± 52
typical concentrations. For the mycotoxins used in [5],
1 Æ 108 (10 lg/kg) 46 ± 52
this means that concentration levels for aflatoxins were

http://www.elsevier.com/locate/trac 1127
Trends
significantly below those of other mycotoxins, such as
deoxynivalenol or zearalenone. Worse precision in the
determination of aflatoxins can therefore be an intrinsic
problem of aflatoxin determination rather than a
concentration effect. This potential correlation between
analyte type and analyte concentration casts further
doubt on the applicability of the model.
3.2. Influence of methods
As was explicitly pointed out in [5], there is a difference
in reproducibility between the various methods. Not
surprisingly, measurements based on liquid chromato-
graphy (LC) tended to have a better reproducibility than
those based on thin-layer chromatography (TLC), which
in turn were more reproducible than enzyme-linked
immunosorbent assay (ELISA) measurements. This
shows that, as expected, the method clearly influences
the precision of results, which is in strong contradiction
with the basic assumption of the model.
In general, ELISA and TLC methods are used as
screening tests for mycotoxins because of poorer perfor-
mance in comparison with LC or GC methods [13]. A
notorious problem of ELISA-based methods is cross
reactivity of analog compounds resulting in large scatter
and significant overestimation. For example, it is well-
known that ELISA methods used for the determination of
the mycotoxin deoxynivalenol (DON) principally cannot
distinguish between the naturally-occurring mycotoxins
DON, 3-acetyl-DON, 15-acetyl-DON, and 3,15-diacetyl-
DON, because the mycotoxin antibodies are designed
against 3,7,15-triacetyl-DON. These findings were
underpinned by interlaboratory studies showing signifi-
cantly higher results for ELISA methods, when compared
with LC or GC results [14].
3.3. Influence of time
The passage of time should not in itself change the
accuracy of measurements but analytical equipment
changes with time. The combined effect of all these
influences can be summarized in the variable ‘‘time’’.
This influence will typically be more pronounced in new
fields of analysis, be it new analytes or new concentra-
tions. Improvement in trace analysis can therefore be
expected to be larger than for long-established bulk
analytical techniques.
The mycotoxin assays based on LC, TLC and ELISA
evaluated in [5] span the period 1968–1992, i.e. nearly
25 years. It would be disappointing if the performance of
laboratories had not improved over this time. Time can
influence reproducibility of assays in many ways. One
influence is certainly the availability of reliable reference
materials, as shown in [15]. Lack and quality of
calibrator and matrix reference materials was certainly a
problem in the early days of mycotoxin analysis.
Agreement between various laboratories might therefore
have been expected to improve over time.
1128 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
A second factor is the development of analytical
equipment and methodologies. Looking at the database
from 1993 [5], a vast number of interlaboratory studies
was based on TLC and ELISA, which are today consid-
ered as semi-quantitative assays and screening tests at
best. It might be expected that the wider use of LC and
GC, the development of columns of better quality and
more reliable standards would result in an improvement
of the reproducibility over time, as also shown by
Whitaker et al. [16]. This assumption is confirmed by
statistical data. While in 1993, Horwitz et al. [5]
concluded that mycotoxin assays tended to show
reproducibilities above those predicted by the Horwitz
equation, Thompson et al. showed that interlaboratory
comparisons after 1997 have significantly better repro-
ducibility than expected from the Horwitz equation, thus
refuting an earlier statement that reproducibility does
not improve over time [17].
A third factor is the improvement in analytical equip-
ment itself. This improvement results frequently in higher
signals that can be determined more accurately. Examples
for this are modern diode-array detectors that are signif-
icantly more sensitive than older models. For example,
this increased sensitivity of instrumentation can be
exploited by narrowing the wavelength range that is
measured, thus decreasing the influence of interferences.
A major improvement in mycotoxin analysis was
achieved by the technical advances in the field of LC-mass
spectrometry (MS). In recent years very accurate, sensi-
tive and robust LC-tandem mass spectrometry (MS2)
methods with electrospray ionization (ESI) or atmospheric
pressure chemical ionization (APCI) have been developed
for a variety of mycotoxins. These methods do not require
any derivatization of the analytes, as GC methods and
matrix effects are considerably reduced using the multiple
reaction monitoring (MRM) mode [18].
In addition, the development of immuno-affinity
columns or MycoSep columns for several mycotoxins
improved sample clean-up [13,19]; for example, a recent
interlaboratory study on aflatoxin B1, B2, G1, G2 myco-
toxins in hazelnut paste by immuno-affinity column
clean-up with LC – fluorescence detection (FLD) using
post-column bromination demonstrated substantially
improved RSDRs in the range 6.1–7.0% for total
aflatoxins and 7.3–7.8% for aflatoxin B1 at mass fraction
levels of 4.0–11.8 lg/kg total aflatoxins [20] (predicted
about 45%). Another example of significant improve-
ment in method performance in the field of mycotoxin
analysis was demonstrated in a project for the produc-
tion of certified reference materials (CRMs) for aflatoxin
M1 (AfM1) in milk powders [21]. The characterization
study on AfM1 in milk powders by different immuno-
affinity column clean-ups with LC-FLD methods resulted
in even better RSDRs in the range 5.9–4.8% [21] for
AfM1 at mass fraction levels of 0.11–0.44 lg/kg
(predicted >70%), respectively.
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
4. Measurement uncertainty and the Horwitz
equation
As described above, the predicted RSDR value from the
Horwitz equation has been prescribed as a criterion for
method performance [3]. This means using it as an
estimate for measurement uncertainty, as explicitly
suggested by Massart et al. [4]. However, this practice is
not in line with the GUM [9] and infringes more than
one current quality-management practice.
Especially, the contradiction with the estimation of
uncertainties of measurements as demanded in section
5.4.6.2 of ISO 17025 [22] is of utmost importance. The
basic idea of any uncertainty estimation is to gather
information on a particular measurement. In an ideal
case, all uncertainty contributions for the particular
measurement are evaluated. If measurement uncer-
tainty is evaluated from method-validation data (e.g., as
suggested by the Eurachem Guide [8]), the actual mea-
surement must be linked to the average method perfor-
mance using the normal quality-assurance tools such as
quality-control charts. Using the Horwitz equation as an
estimate for RSDR means that there is no link whatso-
ever between the actual measurement and the uncer-
tainty estimation. This means that, even if one accepts
RSDR values as estimates for uncertainties, the values
from the Horwitz equation are most likely not to be
accurate estimates for the measurements in question.
The ideal solution would be identification of all
components of uncertainty of measurement and rea-
sonable estimation, as stipulated by the aforementioned
standards and guides [8,9,22]. In the field of myco-
toxin analysis, considerable improvement in method
performances was demonstrated in the project for the
production of CRMs for AfM1 in milk powders [21], as
already mentioned above. Within the frame of this
project, expanded measurement uncertainties for the
determination of AfM1 were also assessed for each
laboratory by summing the combined standard
uncertainties of sample mass, common calibrant,
external calibration, precision and recovery at AfM1
mass-fraction levels of 0.1 lg/kg and 0.4 lg/kg. Rela-
tive expanded uncertainties in the range 7.7–23.7%
(n = 7) and 6.8–19.7% (n = 8) for the lower and
higher mass-fraction levels were calculated. The rela-
tive expanded uncertainties are significantly better
than anticipated from the Horwitz equation and, above
all, the approach complies with the requirements of the
GUM [9].
It might be argued that using the values from the
Horwitz equation constitutes an ‘‘expert judgment’’, as
explicitly foreseen in the GUM. However, the GUM
clearly states that uncertainty evaluation requires
‘‘detailed knowledge of the measurand and the mea-
surement’’ (3.4.8). Using data from other methods for
Trends
other matrices and analytes is not in line with this
requirement. Furthermore, it is difficult to argue that, if
expected RSDR values are somewhere between 11% and
115%, assuming a value of 63% is equivalent to an
expert judgment.
Another incongruity in using the Horwitz equation as
an estimate for measurement uncertainty concerns the
demand of ISO 17025 that a laboratory must make
efforts to determine the goal of the analysis of its cus-
tomer. The result of a measurement always includes the
measurement uncertainty, be it explicit or implicit. It is
therefore assumed that the customer knows how accu-
rate the results need to be. Using an average perfor-
mance, such as the RSDR from the Horwitz equation,
corresponds to what is usual, but not to what is neces-
sary and therefore complies with neither ISO 17025 nor
other current approaches for judging laboratory perfor-
mance. Thompson et al. [23] decidedly dismissed the
practice of normalizing z-scores to the average perfor-
mance in the recent IUPAC protocol for proficiency
testing, as it does not take into consideration customer
needs. This affects even more RSDR values from the
Horwitz equation, which correspond to outdated average
laboratory performance, as shown above, and which are
unrelated to current customer needs so they are no
longer valid.
5. Conclusion
We have demonstrated that the fit of the individual data
from the Horwitz equation is rather poor. The correla-
tion is not good enough to use the Horwitz equation as a
predictive model and confidence levels can exceed the
expected values by more than a factor of 2.
In addition, there are technical reservations about the
Horwitz equation (i.e. its assumptions that RSDR values
do not depend on analyte, matrix, method and/or time
are mistaken, as demonstrated by recent interlaboratory
studies). The Horwitz equation does not make
allowances for the improvement of analytical methods
and techniques, so using the Horwitz equation is
benchmarking against outdated standards and leads to
complacency with results that are not currently state-
of-the-art.
Nevertheless, the Horwitz equation is a useful tool to
summarize historical measurement performance but the
unreliability of its results for a specific problem in ques-
tion, the disagreement with modern quality manage-
ment and the need of uncertainty estimation make it
unsuitable as a criterion for method performance. In-
stead of using the Horwitz equation, measurement
uncertainties should be identified and estimated
according to the GUM approach [9], which is consistent
with ISO 17025 [22].
http://www.elsevier.com/locate/trac 1129
Trends
In view of the movement away from specifying par-
ticular analytical methods towards specifying method
performance criteria, the Codex Committee on Methods
of Analysis and Sampling of the Codex Alimentarius
Commission is discussing a fitness-for-purpose approach
to evaluating methods of analysis [24,25]. This
approach would be based on an uncertainty function
constructed from precision data inter alia, and is to be
judged against the Horwitz equation. This kind of
uncertainty function should be reconsidered, because
the Horwitz equation does not provide an accurate,
traceable and state-of-the-art judgment, as we have
shown above.
Acknowledgement
The authors were very saddened to hear about the death
of William Horwitz while the present paper was being
submitted. An eminent chemist and administrator at the
Food and Drug Administration, he was the recipient of
many prestigious awards for his work in analytical
chemistry and was for many years Executive Director of
the Association of Official Analytical Chemists (now
AOAC International). We wish to acknowledge his
important contribution to analytical food chemistry and
his excellent work in the field of food standards.
We also thank Susanna Linsinger for transferring the
data from Horwitzs publication into a spreadsheet.
References
[1] W. Horwitz, L.R. Kamps, K.W. Boyer, J. Assoc. Off. Anal. Chem.
63 (1980) 1344.
[2] W. Horwitz, Anal. Chem. 54 (1982) 67A.
[3] European Commission, Commission Decision 2002/657/EC imple-
menting Council Directive 96/23/EC, Off. J. Eur. Commun. L221
(2002) 8.
[4] D.L. Massart, J. Smeyers-Verbeke, Y. Van der Heyden, LC-GC Eur.
10 (2005) 528.
1130 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
[5] W. Horwitz, R. Albert, S. Nesheim, J. AOAC Int. 76 (1993) 461.
[6] W. Horwitz, R. Albert, J. AOAC Int. 79 (1996) 589.
[7] M. Thompson, Analyst (Cambridge, UK) 125 (2000) 385.
[8] S.L.R. Ellison, M. Rosslein, A. Williams, (Editors), EURACHEM/
CITAC Guide Quantifying Uncertainty in Analytical Measurement,
2nd Edition, Eurachem, 2000 (http://www.eurachem.ul.pt/
guides/QUAM2000-1.pdf).
[9] International Organization for Standardization (ISO), ISO Guide to
the Expression of Uncertainty in Measurements, ISO, Geneva,
Switzerland, 1995.
[10] W. Horwitz, R. Albert, J. AOAC Int. 89 (1996) 1095.
[11] W. Kandler, Aufbau und Betrieb eines Kontrollprobensystems zur
Qualitätssicherung in der Wasseranalytik, PhD Thesis, Vienna
University of Technology, Austria, 1999.
[12] W. Horwitz, J. AOAC Int. 84 (2001) 919.
[13] R. Krska, S. Baumgartner, R.D. Josephs, Fresenius J. Anal. Chem.
371 (2001) 285.
[14] R.D. Josephs, R. Schuhmacher, R. Krska, Food Addit. Contam. 18
(2004) 417.
[15] R. Schuhmacher, R. Krska, J. Weingaertner, M. Grasserbauer,
Fresenius J. Anal. Chem. 359 (1997) 510.
[16] T. Whitaker, W. Horwitz, R. Albert, S. Nesheim, J. AOAC Int. 79
(1996) 476.
[17] M. Thompson, J.L. Philip, J. AOAC Int. 80 (1997) 676.
[18] F. Berthiller, R. Schuhmacher, G. Buttinger, R. Krska, J. Chroma-
togr. A 1062 (2005) 209.
[19] R.D. Josephs, R. Krska, Fresenius J. Anal. Chem. 369 (2001) 469.
[20] H.Z. Senyuva, J. Gilbert, J. AOAC Int. 88 (2005) 526.
[21] R.D. Josephs, R. Koeber, T.P.J. Linsinger, A. Bernreuther, F.
Ulberth, H. Schimmel, Anal. Bioanal. Chem. 378 (2004)
1190.
[22] International Organization for Standardization (ISO), ISO 17025,
General requirements for the competence of testing and calibra-
tion laboratories, ISO, Geneva, Switzerland, 2005.
[23] M. Thompson, S. Ellison, R. Wood, Pure Appl. Chem. 78 (2006)
145.
[24] ALINORM 05/28/23, Report of the 26th session of the Codex
Committee on Methods of Analysis and Sampling 2005, 28th
session of the Codex Alimentarius Commission, Joint FAO/WHO
Food Standards Programme, Rome, Italy, 2005. (www.codexal-
imentarius.net/download/report/636/al28_23e.pdf).
[25] CX/MAS 05/26/4, Proposed draft recommendations on the
fitness-for-purpose approach to evaluating methods of analysis,
26th session of the Codex Committee on Methods of Analysis and
Sampling, Codex Alimentarius Commission, Joint FAO/WHO Food
Standards Programme, Budapest, Hungary, 2005. (ftp://
ftp.fao.org/codex/ccmas26/ma26_04e.pdf).
Trends in Analytical Chemistry, Vol. 26, No. 7, 2007
Correspondence
Limitations of the application of
the Horwitz Equation: A rebuttal
Michael Thompson
1. Introduction
I must take issue with the paper ‘‘Limitations of the
application of the Horwitz Equation’’ [1].
The actual limitations of the Equation have been
known to the analytical community for at least 10 years.
It is still regarded as a useful tool in appropriate cir-
cumstances (i.e. within its known limitations).
However, the critique is founded on an unfortunate
use of regression and a misreading of the literature, so it
has the potential to cause confusion for legislators,
enforcement agencies, accreditation agencies, profi-
ciency-test providers and practicing analytical scientists.
I believe that the ÔproblemsÕ discussed in the paper are
insubstantial and can be seen to be so if we moderate
with a little common sense the particular approach to
the estimation of uncertainty that seems to underlie the
paper. It would be a great loss if the Horwitz function
were incorrectly thought to be discredited. Chemical
measurement is a difficult enough task and, to conduct it
efficiently, we need access to all of the relevant tools that
are available.
For the benefit of readers unfamiliar with the topic, the
Horwitz Equation generalizes statistics derived from col-
laborative trials (interlaboratory method-performance
studies) in the food-analysis sector. It originated from the
empirical observation that reproducibility (interlabora-
tory) relative standard deviations (RSDR % ¼ 100^ rR =c)
tended to be around 4% when the analyte-mass fraction
was c = 0.01 (that is, at a concentration of 1% by mass).
Moreover, RSDR% tended to double approximately for
every reduction in analyte concentration by a factor of
100. This relationship can be expressed mathematically as
RSDR % ¼ 2ð10:5log10 cÞ , although, in many ways it is pref-
School of Biological and Chemical Sciences, Birkbeck College (University of
London), Malet Street, London WC1E 7HX, UK
E-mail: M.Thompson@bbk.ac.uk
0165-9936/$ - see front matter ª 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.trac.2007.05.008
Correspondence
erable to transform the Equation to show the reproduc-
^H ¼ 0:02c0:8495 .
ibility standard deviation itself, r
2. The statistics
In the statistical section of the paper, we see simple
regression applied to unsuitable data. First, the depen-
dent variable (the percentage-reproducibility relative
standard deviation, RSDR % ¼ 100^ rR =c) is constrained
non-negative on the low side and contains many out-
liers on the high side. Second, the independent variable
is also a function of the concentration c (i.e. we are
considering the regression of 100^ rR =c against
2ð10:5log10 cÞ ), which would automatically give rise to a
meaningless correlation even if the r ^R data were com-
pletely unrelated to c.
A further problem is that the dataset itself tends to
prejudice the outcome: much of the data fall in a con-
centration range where the Horwitz function is already
known to be inapplicable and would not be used. No
reliable information about the Horwitz function could
therefore be derived from this exercise, and, indeed, the
outcome cannot be reconciled with the data (e.g., the
calculated prediction limits for RSDR% manifestly do not
represent the data: the lower limit is below zero for about
half of its range while none of the data is below zero.
RSDR% is non-negative by definition.) However, the pa-
per treats the resulting problems as shortcomings of the
Horwitz Equation, rather than of the statistical model.
3. Properties of the Horwitz function
The paper condemns the Horwitz function on the
incorrect grounds that it is used unthinkingly by ana-
lytical chemists to predict uncertainties.
659
Correspondence
To clear up the resulting confusion, we need to be abso-
lutely clear about the previously published knowledge of the
Horwitz function, which can be summarized as follows.
1. In the food analysis sector (to which its use is largely
restricted), the Horwitz function is a remarkably
good predictor of the trend of reproducibility stan-
dard deviations in the concentration range of
roughly 108–101 mass fraction (i.e. from 10 ppb
to 10% by mass (the ‘‘Horwitz region’’)).
2. Individual observations within the Horwitz region
show considerable deviations from the trend. About
half of this variation is the result of random chance
brought about by the small number (in statisticiansÕ
terms) of laboratories participating in collaborative
trials. The remainder is due to systematic effects, aris-
ing from the use of specific measurement technolo-
gies, and, to some extent, where methods have been
used at concentrations near their limits of detection
(LODs) where RSDR% values are high by definition.
3. At concentrations below about 108, the trend of
results deviates from the Horwitz function and fol-
lows a relative standard deviation of 0.20–0.25.
(In this range the Horwitz function would predict
standard deviations that, if realized, would render
analysis futile: the results would be mostly below
the LOD (e.g., at a concentration of 109, the
function predicts an RSDR% of 45%).
4. At concentrations above 101, the trend of the re-
sults is again for smaller RSDR% than predicted by
the function.
5. Some specific measurement technologies can buck
the trend in the Horwitz region by showing system-
atically better or worse standard deviations than pre-
dicted.
6. When laboratories are required to achieve a perfor-
mance better than predicted by the Horwitz func-
tion, they can do it.
Given all of the above, only an very naı¨ve analytical
chemist would use the Horwitz function as the paper
perceives it to be used, namely:
(a) to define fitness for purpose without reference to the
customerÕs requirements;
(b) to estimate uncertainties;
(c) to predict relative standard deviations in specific in-
stances; and,
(d) to use the function outside the Horwitz region.
4. The Horwitz function and uncertainty
In contrast with using the Horwitz function for predic-
tion, using it to prescribe standard deviations and
uncertainties is not only useful, it is justified rationally.
There is a cogent argument that shows that, within
appropriate application sectors (e.g., food analysis), the
function represents an evolution of analytical methods
660 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 26, No. 7, 2007
towards fitness for purpose. Methods that are unneces-
sarily accurate, and therefore unduly expensive, tend to
be discarded in favor of cheaper ones. Those that are not
accurate enough, and therefore too frequently lead to
financial penalties due to incorrect decisions, are also
discarded and replaced by more accurate methods. The
outcome is an uncontrived general drift towards fitness
for purpose, which is manifested in the food sector (and
in the appropriate concentration range) as. . .the Horwitz
function.
Even without this rationale, it is perfectly justifiable to
use the Horwitz function (or, indeed, any function
agreed among the stakeholders) as a performance crite-
rion in proficiency testing. It is used there to define in
advance an appropriate standard uncertainty for a par-
ticipant in the proficiency test, not to predict the
uncertainty that a participant delivers. This is exclu-
sively the concern of the proficiency-test provider, its
advisory committee and its customers, and is not a
metrological principle. Naturally, the provider will use a
criterion that is relevant to the application sector.
In the same way, the Horwitz function could be a valid
way of expressing fitness for purpose in routine analysis,
if it were found to be relevant after expert consideration
and consultation in that sector. Again, this is not a
metrological principle: it is a way of prescribing, not
describing, the uncertainty of data.
A similar consideration applies to the use of ‘‘Horrat’’
values used to assess the performance of analytical
methods in the food sector (the Horrat being the ratio of
the observed RSD% to that predicted by the Horwitz
function). An average Horrat of 2 from a collaborative
trial strongly suggests that the method will not provide
results that are fit for purpose. A single value that large
could occasionally arise by random chance from a sat-
isfactory method. However, collaborative trials demand
a minimum of five test materials, and the probability of
the mean of five or more results showing such a high
Horrat by chance is very small.
Reiterating a previous statement, only the naı¨ve
would use the Horwitz function directly to estimate
uncertainty, even in the Horwitz region and certainly
not outside it. However, the function does have an
important and critical bearing on uncertainty estimates.
Metrologists rightly object to analytical scientists using
reproducibility standard deviation r ^R from a collabora-
tive trial as an unqualified estimate of standard uncer-
tainty. Different laboratories will vary in their precision
performance, some better and some worse than the trend
indicated by r ^R . More importantly, r ^R does not include
possible error contributions from method bias. This is
saying that r ^R itself underestimates uncertainty, at least
in principle. However, in practice, r ^R interpolated di-
rectly from relevant collaborative trials more often ex-
ceeds uncertainty estimates based on single laboratory
validation. This is because cause-and-effect (‘‘bottom-
Trends in Analytical Chemistry, Vol. 26, No. 7, 2007
up’’) uncertainty models are notoriously unable to
incorporate the variations of unknown origin that give
rise to the ubiquitous between-laboratory effect. If the
cause cannot be identified, or is not even known to exist,
the effect cannot be estimated a priori. In short, most
laboratories using the ‘‘bottom-up’’ approach underes-
timate their uncertainty.
As a consequence, an uncertainty estimate less than
r
^R must be suspect prima facie, unless the laboratory can
demonstrate that it has taken exceptional measures to
reduce error, as would happen, for example, in a na-
tional reference laboratory, so it is always a useful cross-
check to compare uncertainty estimates with an inter-
polated value of r ^R derived from relevant collaborative
Correspondence
trials. However, occasionally, there are no relevant col-
laborative trial statistics available from which to infer a
value for r ^R . In the food-analysis sector, at least, it is
worth comparing the estimated standard uncertainty
with the known trend of r ^R , which, in the Horwitz re-
gion, is r ^H . If the uncertainty estimate is smaller than
r
^H , it is possibly incorrect and should at least be further
investigated. In such an investigation, z-scores from a
proficiency test might help.
Reference
[1] T.P.J. Linsinger, R.D. Josephs, Trends Anal. Chem. 25 (2006) 1125.
http://www.elsevier.com/locate/trac 661
Correspondence
Correspondence
Reply to Professor Michael
ThompsonÕs rebuttal
Thomas P.J. Linsinger1,*, Ralf D. Josephs2
1. Introduction
We greatly appreciate Professor Thompson giving an
overview about the application of the Horwitz Equa-
tion today. His contribution is most welcome because
it is the appropriate time for discussion of uncertainty
estimations and the role and limitations of the Horwitz
Equation, especially in the field of food analysis
regarding method-performance criteria. In his reaction
to our paper [1], Prof. Thompson brings two main
arguments:
(1) our statistical approach is wrong, which invali-
dates our conclusions; and,
(2) the Horwitz Equation can be used as a fitness-for-
purpose criterion, provided it is not used naı̈vely.
2. Our statistical evaluation
We would like to point out that we used the same
data that have been previously evaluated by Horwitz
et al. [2]. As we explain in our publication, we used
mycotoxin data, as the individual RSDR values of the
intercomparisons are available in the original publi-
cation by Horwitz et al., and as more subsequent data
are available for this group of analytes. Shortcomings
of the dataset itself therefore also affect the original
evaluation. Published evaluations for other analytes
(e.g., polychlorinated biphenyls (PCBs)) give only
graphical representations that show similar scatter.
This indicates that our results for mycotoxins are more
generally valid. Regarding our evaluation, we believe
1
EC-JRC, Institute for Reference Materials and Measurements (IRMM),
Retieseweg 111, B-2440 Geel, Belgium
2
Bureau International des Poids et Mesures (BIPM), Pavillon de Breteuil, F-
92312 Sèvres Cedex, France
*
Corresponding author. Tel.: +32 14 571 956; Fax: +32 14 571 548;
E-mail: thomas.linsinger@ec.europa.eu
662 0165-9936/$ - see front matter ª 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.trac.2007.05.009
Trends in Analytical Chemistry, Vol. 26, No. 7, 2007
that Prof. ThompsonÕs comments strengthen, not
weaken, our reservations.
(a) The data used (including the outliers) have been re-
garded as ‘‘technically valid’’ in the publication by
Horwitz et al. [2], and other published collections
of intercomparisons also show outliers. Removal
of outliers does not significantly alter the result
of our statistical evaluation. In fact, presence of out-
liers is to be expected, as mere screening methods
(e.g., ELISA) have also been included in the dataset.
(b) Relative standard deviations (RSDs) are not only
non-negative by definition, but are also expected
to be below 50%. The calculation for RSD assumes
normally-distributed data. For RSDs above 50%,
there would be a significant probability of obtain-
ing results below 0, as about 95% of results are ex-
pected to be in a range of xi  2s, so data from
intercomparisons that show RSDs above 50% can-
not follow normal distributions, and the RSD val-
ues for these intercomparisons are therefore
meaningless.
However, such data were used in the original publi-
cations, despite the fact that there are these reasons to
question the validity of several individual data points. As
we have used the same dataset and evaluation principles
as the original publication, reservations against our
evaluation apply equally to the original evaluation by
Horwitz et al.
In his criticism of our statistical approach, Prof.
Thompson overlooks that we use statistics only to
quantify the obviously large scatter around the
regression line and no statistical treatment can de-
crease this scatter. We are therefore convinced that
our statistical evaluation is fit for its purpose, namely
to give an impression of the variance around the
regression line and thus the low reliability of the
estimate from the Horwitz Equation for an individual
method.
Trends in Analytical Chemistry, Vol. 26, No. 7, 2007
3. Use of the Horwitz function as fitness-for-
purpose criterion and for uncertainty evaluation
Prof. Thompson proposes that the Horwitz Equation is a
good fitness-for-purpose criterion. Meaningful evaluation
of the fitness for purpose of a method requires a priori
definition of performance criteria, which will depend on
the purpose of analysis (e.g., protection of human health,
or product pricing). In contrast, the Horwitz Equation
was developed as an empirical description of RSDR
values and was only afterwards adopted as a fitness-
for-purpose criterion. The observation that accepted
analytical methods tend to fulfill the Horwitz criterion is
therefore a circular argument.
Prof. Thompson argues that the Horwitz function is
an appropriate benchmark for uncertainty estimations.
It is of course true that even the most rigorous single-
laboratory validation (which, by the way, can also
follow a ‘‘top-down’’ approach) needs confirmation of
the result, especially confirmation of absence of any
effects unaccounted for. The proper way to do this is
participation in intercomparisons (as also suggested by
Prof. Thompson) or measurement of certified reference
materials. If the laboratory result agrees with the tar-
get value within the respective uncertainties, the lab-
oratoryÕs uncertainty estimation was correct, regardless
of whether this uncertainty was lower or higher
than the reproducibility standard deviation. There is
therefore no reason to treat uncertainties lower than
the reproducibility standard deviation with special
suspicion.
4. Inappropriate use of the Horwitz Equation in
uncertainty estimations
We do not share Prof. ThompsonÕs optimism that the
relationship between the Horwitz Equation and mea-
surement uncertainty estimation will be readily under-
stood. It is clear that uncertainty is the statistical
parameter related to a laboratoryÕs analytical measure-
ment results. r^R and r
^H , are statistical parameters and
functions related to interlaboratory studies. There is a
relationship between the two but it is not as straight-
forward as suggested by the Horwitz Equation.
Our own experience in giving training courses on the
subject, dealing with other laboratories as well as the
scarcity of uncertainty data in the analytical literature,
clearly shows that the estimation of measurement
View publication stats
Correspondence
uncertainty is not yet very well established and there is
great incentive to make inappropriate use of seemingly
easy tools for this problem. This is possibly the reason
why the estimate from the Horwitz Equation is regularly
put forward as the estimate for a reproducibility standard
deviation to be in turn used as estimate of measurement
uncertainties of results obtained by this method. In
addition, one has to bear in mind that the guidelines for
uncertainty estimations (e.g., the ISO Guide to the
Expression of Uncertainty in Measurement [3]) strongly
emphasize that knowledge of oneÕs own measurement is
a prerequisite for uncertainty estimation. In our opinion,
using reproducibility data from other laboratories,
maybe even from other methods, does not fulfill this
requirement. Furthermore, we do not see any reason
why a laboratory that finds – after having validated a
method and having shown the applicability of validation
data to real samples – that its uncertainty estimate
agrees with published reproducibility data would want to
replace its uncertainty estimation with this published
reproducibility. Careful expert consideration will there-
fore eliminate the need to use the Horwitz Equation
rather than support its use in measurement uncertainty
estimations.
We agree with Prof. Thompson that chemical mea-
surements are not only difficult but they are also
expensive and occur in large quantities, especially in the
food sector, so experts should avoid prescribing gen-
eralized fitness-for-purpose approaches based on unsuit-
able models and historical data, and should rather rely
on thoroughly-validated methods with corresponding
uncertainties cross-checked against application-specific
criteria.
In our view, Prof. Thompson has not refuted our ori-
ginal conclusions – that the RSDs predicted from the
Horwitz Equation cannot be used as an accurate
benchmark for measurement uncertainties for a partic-
ular method. Furthermore, current practice of quality
management and uncertainty estimation is at odds with
any use of the Horwitz Equation other than as a sum-
mary of historical method performance.
References
[1] T.P.J. Linsinger, R.D. Josephs, Trends Anal. Chem. 25 (2006) 1125.
[2] W. Horwitz, R. Albert, S. Neshelm, J. AOAC. Int. 76 (1993) 461.
[3] International Organization for Standardization (ISO), ISO Guide to
Expression of Uncertainty in Measurements, ISO, Geneva, Switzer-
land, 1995.
http://www.elsevier.com/locate/trac 663