T. Ns et al., J. Near Infrared Spectrosc.

1, 111 (1993) 1

T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993)

Artificial Neural Networks in Multivariate Calibration

Artificial neural networks in multivariate

calibration
Tormod Ns, Knut Kvaal, Tomas Isaksson and Charles Millera
MATFORSKNorwegian Food Research Institute, Osloveien 1, N1430 s, Norway.
a
Present address: Du Pont Polymers, Industrial Polymers Research, PO Box 1089, Orange, TX 77631-1089, USA.

This paper is about the use of artificial neural networks for multivariate calibration. We discuss network
architecture and estimation as well as the relationship between neural networks and related linear and non-linear
techniques. A feed-forward network is tested on two applications of near infrared spectroscopy, both of which
have been treated previously and which have indicated non-linear features. In both cases, the network gives more
precise prediction results than the linear calibration method of PCR.
Keywords: Artificial neural network, back-propagation, feed-forward network, PCR, PLS, near infrared spectroscopy,
multivariate calibration.

Introduction Previous comparisons between the ANN method

This paper addresses the use of artificial neural
networks (ANN) in multivariate calibration, with
special emphasis on their performance in near infra-
red (NIR) spectroscopy. In particular, we will focus
on so-called feed-forward networks that use
back-propagation (BP) for estimation. This topic
has been addressed by other researchers,1,2,3 but
there are still many unanswered questions about, for
instance, prediction properties in practical NIR ap-
plications.
The first half of the paper is devoted to a discus-
sion of the choice of network and learning (estima-
tion) rule, and a comparison between neural
networks and other statistical methods such as par-
tial least squares (PLS), principal component re-
gression (PCR) and non-linear PLS. This discussion
will also serve as a background and explanation for
the problems considered in the second part, which
is an empirical investigation of the performance of
ANNs in NIR spectroscopy.
and other techniques have given quite different re-
sults with respect to prediction ability in NIR analy-
sis. For example, in Long et al.1 the ANN method
was compared to PCR and the latter gave the best
prediction results, while in Borggaard and Thod-
berg 2 a special application of the ANN method gave
much better results than both PCR and PLS. The
main goal of this paper is to provide more informa-
tion about the prediction ability of the ANN method
in practice, and also to compare it with the perform-
ance of a standard linear technique, namely PCR. In
this way we want to contribute to the discussion
about the performance of non-linear calibration
techniques in NIR spectroscopy. We will also inves-
tigate how ANNs should be used in order to give as
good prediction results as possible. For more theo-
retical discussions regarding approximation proper-
ties of ANN network models and statistical
convergence properties of estimation algorithms,
we refer to References 4 and 5. For a general back-
ground discussion of the ANN method, we refer to
References 6, 7 and 8.

NIR Publications 1993, ISSN 0967-0335

2 Artificial Neural Networks in Multivariate Calibration
Figure 1. An illustration of a simple feed-forward network
with one hidden layer and one output node.
Choice of network
Feed-forward networks
The field of neural networks covers a wide range
of different network methods which are developed
for and applied to very different situations. In par-
ticular, the feed-forward network structure is suit-
able for handling non-linear relationships between
input and output variables, 6 when the focus is
prediction. Reference 6 also discusses the differ-
ences between feed-forward and other networks. In
this work we will confine ourselves to discuss some
of the main features of the feed-forward network
and how it can be fitted to data.
A feed-forward network is a function in which
the information from the input data goes via inter-
mediate variables to the output data. The input data
(X) is frequently called the input layer and the
output data (Y) is referred to as the output layer.
Between these two layers are the hidden variables
which are collected in one or more hidden layers.
The elements or nodes in the hidden layers can be
thought of as sets of intermediate variables analo-
gous to the latent variables in bilinear regression
(e.g. PLS and PCR). An illustration of a feed-
forward network with one output variable and one
hidden layer is given in Figure 1. The information
from all input variables goes to each of the nodes in
the hidden layer and all hidden nodes are connected
to the single variable in the output layer in each
case. The contributions from all nodes or elements
Figure 2. An illustration of the signal processing in a
sigmoid function.
are multiplied by constants and added before a pos-
sible transformation takes place within the node.
The transformation is in practice often a sigmoid
function, but can in principle be any function. The
sigmoid signal processing in a node is illustrated in
Figure 2.
The feed-forward neural network in Figure 1
gives a regression equation of the form
I J
i =1
y = f bi gi wij x j + a i1 + a 2 + e
(1)
j =1
where y is the output variable, the xs are the input
variables, e is a random error term, gi and f are
functions and b i , wij , ai1 and a2 are constants to be
determined. The constants w ij are the weights that
each input element must be multiplied by before
their contributions are added in node i in the hidden
layer. In this node, the sum over j of all elements
wijxj is used as input to the function gi. Then, each
function gi is multiplied by a constant b i before
summation over i. At last the sum over i is used as
input for the function f. More than one hidden layer
can also be used, resulting in a similar, but more
complicated function. Note that for both the hidden
and the output layer, there are constants, a i1 and a2,
respectively, that are added to the contribution from
the rest of the variables before the transformations
take place. These constants play the same role as the
intercept (constant) term in a linear regression
model.
As can be seen from Equation 1, an artificial
feed-forward neural network is simply a non-linear
parametric model for the relationship between y and
all the x-variables. There are functions gi and f that
have to be selected6 and parameters wij and bi that
T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993)
must be estimated from the data. The process by
which these parameters are determined is, in the
terminology of artificial neural computing, called
learning. The best choice for g i and f can in
practice be found by trial and error, in that several
options are tested and the functions that result in the
best prediction ability are selected. It has been
shown that the class of models in Equation 1 is
dense (for suitable choice of functions g i and f) in
the class of smooth continuous functions. This
means that any continuous function relating y and
x1, , xJ can be approximated arbitrarily well by a
function of this kind. 4
As with linear calibration methods, network
models must be constructed with consideration of
two important effects: underfitting and overfitting.9
If a model that is too simple or too rigid is selected,
underfitting is the result, and if a model that is too
complex is used, overfitting can be the conse-
quence. The optimal model complexity is usually
somewhere between these two extremes. Tech-
niques have been developed that simultaneously
estimate the complexity (which is related to the
number of hidden nodes) and the model parame-
ters,10 but this subject is not covered here.
The back-propagation learning rule
The learning rule that is most frequently used for
feed-forward networks is the back-propagation
technique.6 In back-propagation, the objects in the
calibration set are presented to the network one by
one in random order, and the weights w ij and bi
(regression coefficients) are updated each time in
order to make the current prediction error as small
as possible. This process continues until conver-
gence of the regression coefficients. Typically, at
least 10,000 runs or training cycles are necessary in
order to reach a reasonable convergence. In some
cases many more iterations are needed, but this
depends on the complexity of the underlying func-
tion and the choice of input training parameters.
More specifically, the back-propagation is based
on the following strategy: (1) Subtract the output of
the current observation from the output of the net-
work, (2) compute the squared residual (or another
function of it) and (3) let the weights for that layer
be updated in order to make the squared output error
3
as small as possible. The error from the ouput layer
is then back-propagated to the hidden layer and
the same procedure is repeated for the w ijs. The
differences between the new and the updated pa-
rameters are dependent on so-called learning con-
stants which can have a strong influence on the
speed of convergence of the network. Such learning
constants are usually dynamic, which means that
they decrease as the number of iterations increases
(see the examples below). The constants can be
optimised for each particular application. In some
cases, an extra term (momentum) is added to the
standard back-propagation formulae in order to im-
prove efficiency of the algorithm. We refer to Ref-
erence 8 for details.
Although this kind of back-propagation seeks to
minimise a least squares criterion, it is important to
note that this kind of learning is different from
standard least squares (LS) fitting where the sum of
squared residuals (yy)2 is minimised over all pa-
rameter values bi and wij. Theoretical studies of the
convergence properties of ANNs can be found in
Reference 10. Extensions of the back-propagation
technique described above have been developed in
order to make it more similar to the standard LS
fitting of all objects simultaneously.8
A modification of the standard network
A modification of the standard model described
in Equation 1 has been suggested in Reference 2.
The idea of this modification is to add direct linear
connections between the input and the output layer.
This leads to the network model
I J J
y = f bi gi wij x j + a i1 + a 2 + d j x j + a 3 + e (2 )
i =1 j =1 j =1
The ds here are the constants that are multiplied
by the input variables in the direct connections
between the input and output layer and the as are
constant terms. In this way, the network is com-
posed of a linear regression part and a standard
non-linear network part. As for the standard net-
work, back-propagation is used for learning. In Ref-
erence 2 this modified network performed well on
near infrared data.
4 Artificial Neural Networks in Multivariate Calibration
Relations to other methods
The artificial neural network models have strong
relations to other statistical methods that have been
used frequently in the past. In this section, however,
we will only focus on a limited set of relations to
methods that are frequently discussed and used in
the chemometric and NIR literature. For a discus-
sion of more statistically oriented aspects of such
relations, we refer to References 5 and 10.
Multiple linear regression (MLR)
MLR is based on the following model for the
relationship between y and x:
I
y = a+ bi xi + e (3)
i =1
where the a and b is are regression coefficients to be
estimated. We see that this model is identical to a
special version of model Equation 1 in which the
hidden layer is removed and the transfer function is
set equal to the identity function. It is also easy to
see that if we replace both f and gi in Equation 1 by
identity functions, we end up with a model which is
essentially the same as the MLR model. This is
easily seen by multiplying w ij and bi values and
renaming the products. This procedure is called
reparameterisation of the network function.
However, in the learning procedure, the MLR
equation is most frequently fitted by using standard
LS instead of applying, for example, the iterative
fitting procedure described above. Below we will
investigate differences between the empirical prop-
erties of these two estimation techniques.
Partial least squares regression and
principal component regression
The regression equation for both PLS and PCR
(with centred y and x) can be written as
I J I versions of ANN.
y= bi wij x j + e = bi ti + e ( 4)
i =1 j =1 i =1
where the wijs now correspond to the loading
weights 9 of factor i and the b i s correspond to the
regression coefficients of y on the latent variables
ti = wijxj (i.e. y-loadings). It can be seen that apart
from the non-linear g i and f in the feed-forward
network Equation 1, the two equations (1) and (4)
are identical. This means that the model equation
for PLS and PCR is a special case of an ANN
equation with linear g i and f.
However, for the learning procedure, PCR and
PLS are quite different from ANN. The idea behind
the PCR/PLS methods is that the weights w ij are
determined with a strong restriction on them in
order to give as relevant scores for prediction as
possible. 9 In PLS, the w ij s are the PLS loading
weights found by maximising the covariance be-
tween y and linear functions of x, and in PCR they
are the principal component loadings obtained by
maximising the variance of linear functions of x. In
other words, estimation of bi and wij is not done by
fitting Equation (4) as well as possible, but through
the use of quite a different rule. If both b i and wij in
Equation 4 were determined by LS directly, the
ordinary MLR method would be the result, since
Equation 4 is only a reparameterisation of the linear
regression equation. Since the ANN method con-
centrates on fitting without restrictions (on the pa-
rameters), there are reasons to believe that ANNs
are more sensitive to overfitting than are PLS and
PCR. As a compromise, it has been proposed to use
principal components as input for the network in-
stead of the original variables themselves. 2 This
procedure may, however, have some drawbacks,
since it restricts the network input to only a few
linear combinations of the original spectral values.
This problem is studied in the application section
below.
In PCR and PLS models, the scores ti and loading
weights w ij are orthogonal (for each j). This is not
the case for ANN models, in which the w ij s are
determined without this restriction. The orthogonal-
ity properties of PLS and PCR are strong reasons
for their usefulness in the interpretation of data, and
this same argument can be used against the present
Projection pursuit (PP) regression
The PP regression method 11 is based on the equa-
tion
T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993)
I J
y= si wij x j + e (5) more flexible model than the ANN model, but the
i =1 j =1
for the relation between y and the x-variables. In this
case, the si s are smooth functions of the linear
function ti = wijxj . Again, we see that the model
Equation 5 is similar to an ANN model with linear
f and gi replaced by s i. Notice also that since the sis
have no underlying functional assumption, PP re-
gression is based on an even more flexible function
than the ANN in Equation 1. Note also that since the
si s are totally general except that they are smooth,
we no longer need the constants a i1 and a2 .
Regarding the estimation of the weights w ij and
functions si , the PP regression is based on a least
squares strategy. First, the constants w ij and the
smooth functions s i are determined, then the effect
of this factor is subtracted and the same procedure
is applied to the residuals. This continues until the
residuals have no systematic effects left. For each
factor, the si and wij are found by an iterative pro-
cedure. The s functions are usually determined us-
ing moving averages or local linear fits to the data.
The PP regression method as described here is
more flexible than the class of ANN functions, and
the optimal functions si and constants w ij are deter-
mined simultaneously. This procedure is very dif-
ferent from the ANN procedure, in which different
functions g and f have to be tried independently.
However, one main drawback with PP regression is
that it gives no closed form solution for the predic-
tion equation, only smooth fits to samples in the
calibration set. Therefore, prediction of y values for
new samples must be based on linear or other inter-
polations between the calibration points.
The relationship between the PP and ANN meth-
ods is treated in more detail in Maechler et al.5
Non-linear PLS regression
In 1990, Frank 12 published a method called non-
linear PLS, which is a kind of hybrid of PP regres-
sion and PLS. The technique is essentially based on
the same model as PP regression, but the constants
wij are determined the same way as the PLS loading
weights are determined from residual matrices after
all previous factors have been subtracted. Thus, this
5
version of non-linear PLS is based on a similar, but
estimation of coefficients is carried out under a
strong restriction, namely the PLS criterion based
on maximisation of covariance between y and linear
functions of x variables.
The non-linear PLS method proposed in Wold13
is also essentially based on a model of the same kind
as the PP regression. In this case, however, the
smooth functions are approximated by splines, and
the estimation of parameters follows a quite differ-
ent rule than the procedure presented in Frank. 12
Examples
In this section we will test the feed-forward ANN
described by Equation 1 on two real situations in
which it is desired to relate NIR spectral data (X) to
a quality parameter (Y). Both datasets have been
used elsewhere and the wavelengths (X-data) are
strongly collinear.14 In this study, the main empha-
sis will be on the different aspects of prediction
ability of the ANN method, considered as a function
of the number of nodes in the hidden layer, the
number of input variables (or principal compo-
nents) and the number of iterations.
All prediction results below will be presented in
terms of root mean square error of prediction
(RMSEP), which is defined by
1
Ip 2
RMSEP = ( yi yi ) / Ip
2
(6)
i =1
where Ip is the number of samples used in the pre-
diction set. All reported results are obtained by
testing the networks on samples that were not used
for training of the network. All networks obtained
in this paper are computed using NeuralWorks Pro-
fessional II/plus (Neuralware Inc., PA, USA). Dur-
ing the network training, all input and output values
are scaled so that they are in the range 01 (for
input) and 0.20.8 (for output), respectively, (for
arguments see the manual of the program). After
convergence of the network, all prediction results
are transformed back to original scale. The learning
constants and momentum values that are used
throughout this work are given in Table 1. It is
6 Artificial Neural Networks in Multivariate Calibration

Table 1. The learning constants of the neural network: For both the output and hidden layer we used the same learning
constant and momentum value. The constants change as the number of iterations increases.

Number of iterations 02000 200020,000 20,00040,000 40,000100.000

Learning constant: 0.9 0.5 0.25 0.1

Momentum: 0.6 0.3 0.15 0.1

important to note that these constants are changed

as the number of training cycles increases. Such a
procedure is recommended (see the manual of the
program) and is natural because it enables a sort of
fine-tuning of the network as it approaches con-
vergence. All non-linear networks used are based on
a sigmoid transfer function.

Water predictions in meat

The data set used in this study contains the near
infrared transmission (NIT) spectra of 103 different
meat samples at 50 different wavelengths (X-vari-
ables) and the concentration of water in these sam-
ples as determined by oven-drying (Y-variable).
Seventy samples are used for calibration and the
remaining 33 samples are used for testing the per-
formance of the calibration model. The data set is
described in more detail in Reference 15. Regarding
the X-data, both original NIT spectra and NIT spec-
tra that are corrected for scattering effects (by the
MSC method)16 will be considered.
Figure 3. PCR prediction error as a function of the number
of components (meat data). The upper curve corresponds
to uncorrected data and the lower curve correspond to
scatter-corrected data.
Table 2. RMSEP results for a sigmoid ANN with the use
of the 50 individual x-values as input (meat data). Both
uncorrected and scatter-corrected data are used. All re-
sults are based on 50,000 iterations.

Prediction ability Number of nodes in hidden layer

The prediction results using the PCR method are
given in Figure 3. The best result is RMSEP = 0.77, 1 10 25 50
obtained by using eight principal components from Scatter-corrected 0.82 0.70 0.74 0.81
scatter-corrected data. The RMSEP obtained for the
PCR model that uses the maximum number of fac- Uncorrected 3.95 2.28 2.26 2.27
tors from the scatter-corrected data is equal to 1.65.
In other words, PCR gave much better results than
a full MLR based on all components. It can also be this data set, and all results reported in this section
seen that scatter-corrected data gave clearly better are therefore based on this number. As can be seen,
results than uncorrected data. the scatter-corrected data gave much better results
The prediction results obtained using non-linear than the uncorrected data, indicating the advantage
ANN models with different numbers of hidden of input data prelinearisation, even for the non-
nodes are given in Table 2 for both uncorrected and linear ANN technique. The effect of using different
scatter-corrected data. As will be discussed below, numbers of nodes in the hidden layer is weak. For
the use of 50,000 iterations was close to optimal for the scatter-corrected data the best results were
T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993) 7

Table 3. RMSEP results for sigmoid ANNs computed on sented in Figure 4. The table presents some results
scatter-corrected meat data, using different numbers of obtained using different numbers of principal com-
principal components as input and different numbers of
ponents as input, and different numbers of nodes in
nodes in the hidden layer. 50,000 iterations were used.
the hidden layer. Scatter-corrected (MSC) data gave
Number of nodes in hidden layer much better results than uncorrected data and there-
fore only the results using scatter-correction are
1 3 8 15 reported. It is clear that it is best to use an interme-
diate number of PCs as input, namely eight, and
3 0.84 0.79 0.74 0.92 also an intermediate number of nodes in the hidden
Number layer, namely three (RMSEP 0.64). In other words,
of input 8 0.72 0.64 0.68 0.66
there is an effect of underfitting and an effect of
variables 15 1.05 1.03 1.01 1.06 overfitting both with respect to the number of nodes
and the number of input components (variables). It
22 1.32 1.31 1.16 1.37 is also worth noting that the results obtained using
principal components as input are better than those
obtained using individual x-responses as input,
although the differences are small. The best ANN
result was about 15% better than the best PCR
result.
For the scatter-corrected data, a linear transfer
function network based on all input variables and
with only one hidden node, was also tested and the
RMSEP was equal to 0.89, which is somewhat
larger than the value obtained for the best sigmoid
network. For comparison, the linear ANN based on
all available principal components gave a RMSEP
of 1.66 after 50,000 iterations. (This result is ex-
pectedly similar to the linear PCR based on all
available components.) This difference in perform-
ance between the linear ANN method based on all
original variables and the linear ANN method based
on all principal components is somewhat surpris-
ing, since the two methods are based on essentially
the same model. The most plausible explanation of
Figure 4. A smooth illustration of the numbers in Table the different performances must therefore be that
3. As smoothing technique is used the default method of the iterative ANN learning rule, as opposed to
PROC G3GRID in SAS (SAS Institute, NC, USA). standard least squares fitting, can be dependent on
linear transformations of the data. Our interpreta-
tion of this result is that when original variables are
obtained using 10 and 25 nodes, and for the uncor- presented to the network, back-propagation (at least
rected data, the best results were obtained using 10, with the learning constants in Table 1) is not able to
indicating that an intermediate number of hidden detect the directions of minor variability and rele-
nodes is optimal. The best network gave about 9% vance in the X-space. When principal components
better results than PCR. are used, however, these directions are sorted out
Results obtained by using a sigmoid ANN and blown up by the scaling, and therefore are
method based on principal components instead of given the same possibility as the major variabilities
the original x-variables are presented in Table 3. A to influence the determination of regression coeffi-
smooth graphical illustration of the results is pre- cients.
8 Artificial Neural Networks in Multivariate Calibration
Figure 5. Three replicates of four different networks com-
puted on the meat data. For each network, the three repli-
cates are illustrated by the same symbol. For each
network, only a limited number of RMSEPs were re-
corded. In each case they are joined by linear interpola-
tion. Overlapping curves are represented by just one line.
Linear ANN models with three, eight and 15
principal components as input were also tested and
these gave exactly the same results as the linear
PCR solutions. This result is expected because an
ANN model with a linear transfer function network
in the principal components is essentially the same
model as the linear PCR method.
In Ns and Isaksson15 the same data were used
for evaluating the locally weighted regression
(LWR) method (in that paper, however, 100 wave-
lengths were used). The LWR method based on
three principal components (after scatter-correc-
tion), gave an RMSEP of 0.65, which is almost
equal to the prediction ability obtained from the
best ANN model. The ANN, however, needed more
principal components, namely eight, to obtain a
comparable result.
The influence of starting values on the RMSEP
To test the dependence of the fitted neural net-
works on the starting values of the network parame-
ters, we recomputed different networks using
different starting values. In Figure 5, the prediction
results (on the test data) of three replicated compu-
tations (from 5000 to 100,000 iterations) for four
quite different networks are given. It can be seen
that the differences between the replicates is quite
small and the relative differences between the four
networks is, in this case, independent of starting
values. However, the differences between replicates
are large enough to indicate that the smaller differ-
ences in Table 3 should not be interpreted too
strongly.
The rate of convergence
From testing of networks (on the prediction set)
for different number of iterations, it was found that
about 50,000 iterations gave the best results for this
data set. The results from four different networks
are presented in Figure 5. This figure also indicates
that already after 30,000 iterations, we are close to
the optimal values. From 50,000 to 100,000 itera-
tions there is a small increase in the RMSEP. This
is interpreted as a slight overfitting of the network
model. The results in Table 2 and Table 3 are pre-
sented for 50,000 iterations.
Polymer data
The data used in this experiment contains the
near infrared diffuse reflectance (NIR) spectra of 90
different polyurethane elastomers at 138 wave-
lengths (X), and the flex modulus (or stiffness)
value for these samples (Y). The flex modulus val-
ues of these samples ranged from 5.8 to 68.9 PSI. A
training set of 47 samples and a prediction set of 43
samples was chosen such that the Y-values of the
prediction samples were within the range of the
values for the training set samples. In this example
only the scatter-corrected data (by the MSC
method) are used. Details regarding this data set are
presented in Reference 17. It should be noted that
linear PLS analysis of these data revealed a strong
non-linear relationship between the X and Y data
when three PLS factors were used.
Prediction ability
The PCR prediction results are presented in Fig-
ure 6. The best prediction result was in this case
RMSEP = 2.84, obtained using 38 components
(note that 21 components gave only slightly less
precise predictions). The RMSEP was, however,
quite stable with respect to the number of compo-
nents, but increased as a result of overfitting as the
number of components became very large. The use
of the maximum number of components (46 = 471,
T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993) 9

Table 4. RMSEPs for polymer data in which all 138

wavelengths are used as input (scatter-corrected data).
100,000 iterations were used.

Number of nodes in hidden layer

1 25 138

2.58 1.96 2.13

Table 5. RMSEP results for sigmoid ANNs computed on

scatter-corrected polymer data, using different numbers
of principal components as input and different numbers of
nodes in hidden layer. 100,000 iterations were used except
for the computations in the last line, where only 1600
Figure 6. PCR prediction error as a function of the number
iterations were used (see text).
of components (polymer data).
Number of nodes in hidden layer
because of mean-centring) resulted in a RMSEP of
1 3 8 13
3.86.
Non-linear ANN prediction results for networks 3 3.05 2.98 3.61 3.22
that use each of the input wavelengths as input are
given in Table 4, for different numbers of nodes in 5 3.16 3.13 3.86 4.53
Number
the hidden layer. As in the previous data set, an
of input 13 2.65 2.64 3.00 3.11
intermediate number of nodes, namely 25, in the
variables
hidden layer gave the absolutely best result 16 2.76 2.63 2.85 2.96
(RMSEP = 1.96). As can be seen, the improvement
over the best PCR result is about 30%. It is impor- 21 2.57 2.51 2.64 2.80
tant to note that the use of all available wavelengths
46 3.11 3.27 3.38 3.79
in a traditional least squares approach would be
impossible in this case, because the number of sam-
ples is much smaller than the number of variables.
However, for the ANN method with back-propaga-
tion learning, this situation posed no problem.
The RMSEPs obtained by using principal com-
ponents as input variables for the ANN are given in
Table 5. As for the previous data set, there is an
effect of overfitting with respect to the number of
hidden nodes. It is also clear that the use of 13, 16
and 21 components as input resulted in similar and
optimal prediction results. Note also the overfitting
effect as we increase the number of components to
the maximum.

The influence of starting values on the RMSEP
As in the previous data set, prediction results of
three replicates of four different network models are
shown in Figure 7. There are some differences be-
tween the prediction errors obtained from the repli-
Figure 7. Three replicates of four different networks com-
puted on the polymer data. For each network, the three
replicates are illustrated by the same symbol. For each
network, only a limited number of RMSEPs were re-
corded. In each case they are joined by linear interpola-
tion. Overlapping curves are represented by just one line.
10
cate models, but these are not large enough to justify
a change in the main conclusions regarding the
differences among the four networks.
The rate of convergence
For this data set, 100,000 iterations resulted in
slightly better prediction results (on the test data)
than 50,000 iterations, but these differences are
very small. The only exception here was the net-
work that used 46 principal components, for which
about 1600 iterations resulted in the best model. In
other words, a strong effect of overfitting as the
number of iterations was increased was observed in
this case. All results in Tables 4 and 5 are given for
100,000 iterations, except those obtained from the
network with 46 principal components as input.
Discussion
There are a number of conclusions that can be
extracted from the applications above, which both
were selected from earlier publications in which
they had indicated non-linear features. First of all,
the sigmoid ANN method gave clear improvements
in prediction ability compared to the linear PCR
method (15% and 30%). In addition, scatter-
corrected data gave better results than the uncor-
rected NIR data both for PCR and for the non-linear
ANN. This result shows that even the very flexible
ANN functions are not always able to cope with
non-linearities from the strong non-linear scattering
effects in NIR spectra. Next, there is always, except
in one case, an effect of overfitting with respect to
the number of nodes used in the hidden layer. This
result means that the use of a function that is too
flexible results in less precise networks.
In both examples, the use of principal compo-
nents as input data gave good results, but the
RMSEP varied quite considerably with the number
of components used. In both examples, there was a
clear effect of overfitting with respect to the number
of principal components used, as was also the case
for the linear PCR. For one of the sets, the use of all
individual x-values as input gave better prediction
results than the optimal number of principal com-
ponents. In the other case the difference in predic-
tion ability between networks based on principal
components and networks based on the original
Artificial Neural Networks in Multivariate Calibration
spectral variables was very small. This indicates
that the overfitting effect is not necessarily so
strong as expected when the original variables are
used. When the principal components are used as
input, however, the overfitting effect was strong. To
understand this fully more research is needed.
Networks with sigmoid functions gave, in gen-
eral, better predictions than networks with linear
transfer functions.
The number of iterations that gave the best result
was different in the two cases. In the first example,
50,000 was close to the optimum and in the other
example 100,000 was close. The differences be-
tween the two choices, however, was small in both
cases. It should also be mentioned that the rate of
convergence is dependent on the learning parame-
ters. Parameters other than those used in the present
investigation could possibly give somewhat differ-
ent results, but this was not investigated. The differ-
ences between replicated computations of the same
network model (with different starting parameters),
was quite small in all the cases considered.
The main conclusion is that the ANN method, if
used in the right way, was a good method for cali-
bration in the two cases tested. The improvements
compared to the linear PCR were, however, much
more moderate than in the examples in Borggaard
and Thodberg.2 This shows that the performance of
ANN methods relative to linear methods is strongly
dependent on the situation considered. This conclu-
sion is supported if we also compare the above
results with the results in Long et al.1
A last point to mention is that even though arti-
ficial neural networks may in certain cases be better
than for example PCR from a prediction point of
view, they are more difficult to understand and the
results are much more difficult to visualise and
interpret. These aspects may, however, be improved
in future developments of ANN techniques.
References
1. R.L. Long, V.G. Gregoriou and P.J. Gemper-
line, Anal. Chem. 62, 1791 (1990).
2. C. Borggaard and H.H. Thodberg, Anal. Chem.
64, 545 (1992).
3 W.F. McClure, H. Maha and S. Junichi, in Mak-
ing Light Work: Advances in Near Infrared
T. Ns et al., J. Near Infrared Spectrosc. 1, 111 (1993)
Spectroscopy. (Ed. by I. Murray and I. Cowe).
VCH, Weinheim, pp. 200209 (1992).
4. A.R. Barron and R.L. Barron, in Computing
Science and Statistics, Proceedings of the 21st
Symposium on the Interface. American Statisti-
cal Association, Alexandria, VA, pp. 192203
(1988).
5. M. Maechler, D. Martin, J. Schimert, M. Csop-
penszky and J.N. Hwang, in Proceedings of the
2nd International Conference on Tools for AI
(1990).
6. Y.H. Pao, Adaptive Pattern Recognition and
Neural Networks. Addison-Wesley (1989).
7. J. Zupan and J. Gasteiger, Anal. Chim. Acta.
248, 1 (1991).
8. J. Hertz, A. Krogh and R. Palmer, Introduction
to the Theory of Neural Computation. Addison-
Wesley, USA (1991).
9. H. Martens and T. Ns, Multivariate Calibra-
tion. John Wiley & Sons, Chichester (1989).
11
10. A.R. Barron, in Nonparametric Functional Es-
timation (Ed. by G. Roussas). Kluwer Aca-
demic Publishers, Dordrecht, The Netherlands,
pp. 10341053 (1991).
11. J.H. Friedman and W. Stuetzle, J. Am. Stat.
Assoc. 76, 817 (1981).
12. I. Frank. Chemometrics and Intelligent Labora-
tory Systems 8, 109 (1990).
13. S. Wold, Chemometrics and Intelligent Labora-
tory Systems 14, 71 (1982).
14. S. Weisberg. Applied Linear Regression. John
Wiley & Sons, New York (1985).
15. T. Ns and T. Isaksson, Appl. Spectrosc. 46, 34
(1992).
16. P. Geladi, D. McDougal and H. Martens, Appl.
Spectrosc. 39, 491 (1985).
17. C.E. Miller and B.E. Eichinger, Appl. Pol. Sci.
42, 2169 (1991)
Paper received: 23 October 1992
Accepted: 5 February 1993
12 Artificial Neural Networks in Multivariate Calibration