Escolar Documentos
Profissional Documentos
Cultura Documentos
Orth, FH Ffm
Summer 2011
Data Mining
Contents
Data........................................................................................................................................2
Types and Scales of Data and the Role of Data in an Analysis ........................................2
Applications and Methods ................................................................................................3
Contingency Tables................................................................................................................4
Linear modeling.....................................................................................................................6
Example: one x variable, one y variable............................................................................7
Example: two x variables, one y variable..........................................................................7
General Solution of the Least Squares Problem in Linear Modeling................................8
Example (continued) two x variables one y variable.........................................................9
Changes in content or structure of the X matrix................................................................9
Linear Modeling using R (another example)...................................................................10
Understanding the output of R's lm( )-function...............................................................11
Interpreting the model in the light of the application......................................................13
Improving Interpretation by Centring..............................................................................14
Centering and Scaling......................................................................................................14
Adding a Square Term to a model...................................................................................16
Using stepwise regression to stabilize a model...............................................................18
Transforming the y variable before fitting (relative errors => log).................................19
Regression diagnostics and model improvement............................................................20
Blood Cell Count example (8 x-variable, one y-variable)...............................................23
Estimating Confidence intervals and prognosis intervals for linear models...................26
General linear modeling (GLM)...........................................................................................31
Fitting a piecewise constant function..............................................................................31
Fitting Splines..................................................................................................................33
Repeated Measures, Longitudinal Analysis, Mixed Effects Modeling................................35
Estimating parameters using Maximum-Likelihood (ML)..................................................39
Classification using logistic regression...........................................................................41
Data Analysis for Categorical x-data....................................................................................43
Example: two categorical x and one continuous y-variable............................................43
Analysis of variance........................................................................................................44
Using Linear Modeling to treat categorical x-data..........................................................46
Classification categorical y-variables................................................................................47
Iris-Example: one categorical y-variable (3 categories) and four variables....................47
A Global Method of Classification..................................................................................48
A Local Method of Classification....................................................................................49
Other global Classification Methods...............................................................................50
Cox-PH for reliability and life-time analysis.......................................................................52
Literature:.............................................................................................................................56
page 1 (57)
Data
Types and Scales of Data and the Role of Data in an Analysis
Data can come in different "shapes" and "sizes". Data can be text: text can be atomic or text
can be structured, text can allow an ordering or be simply there ... Data can be numbers:
Numbers can be discrete or continuous, continous numbers can live on an interval, on a semi-
axis or the whole real number-line. Depending on what the type of data is, different Data
Mining Methods will be used to deal with them to extract information.
semi-interval (ratio-scale)
unbounded
There is a hierarchy in these scales: a nominal scale is the lowest level, then comes ordinal
scale, the continuous scale the highest. This is because the nominal scale just reflects that
there is a difference between objects of observation (they belong to different classes) but it
does not reflect whether an object A is better of worse than object B, the ordinal scale does
allow this but in turn does not reflect the notion of distance, i.e. of how much better object A
may be than object B. This is then reflected by a continuous scale(s), where there is also a
notion of distance. There are different types of continuous scales, depending of whether they
are bounded or unbounded and whether they have a physical dimension, like length or weight,
or whether they are dimensionless, like rations, angles and so on.
Of course on the computer, continuous scales can only approximated by the available data
formats double or long double but in any case mathematical operations like +, - *, / are
possible. The highest scale would be one, that is dimensionless (like ratio-scales) because for
these all mathematical functions are available (e. g. log( ), tan( ), etc.) , i. e. they can be
transformed. In statistics usually one uses only monotonic functions to transform, because all
other functions would possibly destroy information. More often than not continuous data are
scaled before they are analyzed (an example of a monotonic transformation). This will be
discussed later.
page 2 (57)
Data are also differentiated by their purpose or role in an analysis. The most common way to
differentiate data is to divide into x-data and y-data. The idea is that x-data are easily
available but not so important and y-data are hard to get but important. So that it becomes
interesting to model (unknown) y-data using known x-data.
y-data important
BUT: expensive
AND: available too late to change the world (or the part of it, that is of
current concern)
In context of x- and y-data the terms exogenous and endogenous are sometimes used, exo- and
endo- being Greek meaning outside and inside respectively, thus indicating that x-data send
out something, that y-data catch. In mathematics x-data are usually called independent
variables, y-data are dependent variables, and ideally y can be expressed as a function of x,
i.e. y = f(x). In the context of data mining and data analysis the word function is avoided and
replaced by the finer-sounding word model.
The distinction into y-data and x-data is not always necessary, but it plays an important role in
many data-analysis or data-mining applications.
Methods used include descriptive statistics, distribution testing and fitting, simple statistical
tests, correlation and regression analysis, classification and clustering, analysis of variance
(ANOVA) and covariance, linear modeling (LM), generalized linear modeling (GLM) and
mixed effects modeling (LME), survival analysis, time series analysis, longitudinal analysis
and in general multivariate data analysis.
page 3 (57)
In this lecture emphasis will be placed on four groups of methods used to treat problems
according to the type or scale of the available data on the x- and on the y-side:
As was stated in the last paragraph, not all data analytical problems concern x- and y-data, so
obviously not all data analytical methods fall into one of the four above categories.
Nonetheless awareness of the four categories and methods that can be classified accordingly
is very useful.
Contingency Tables
The word contingency comes from the latin word contingere, which means touching together
or happening together inexpectedly (in time). The word also comes from the Greek
(endechmena) which means something that is possible. In philosophy it
describes a state of affairs that is neither necessarily true nor false, neither necessary nor
impossible. So contingency describes the finiteness of an existence that can be as it is, could
also be different or could maybe not exist at all.
In statistics contingency describes the connection between two nominal characteristics. In the
simplest case there are two such nominal characteristics, and each of these two characteristics
has 2 possible settings.
pass fail
regu 7 0
rare 1 5
The contingency table can be used to investigate if the two events regular class attendance and
exam success are touching together and happening unexpectedly together in time or
page 4 (57)
whether we are in a state that is neither necessarily true nor false, neither necessary nor
impossible.
X1 \ X2 0 1
0 n1,1 n1,2 n1,.
1 n2,1 n2,2 n2,.
n.,1 n.,2 n
Notice, that in the sum column on the right, there is a dot in the index on the right, and in the
row on the bottom, there is a dot in the index on the left. The dot replaces the index over
which the sum has been formed. These sums are called marginals or marginal sums.
n
n . ,1
n!n 1,1!n 1,2!n 2,1!n 2,2!
2, . =
8!5!7!6!
138 13!7!0!1!5!
123456 2 2
= = = which is very small
910111213 11313 429
So in this case we reject the Null Hypothesis of independence and accept the alternative
Hypothesis, namely that success in exams is highly dependent on regular attendance in the
exercise sessions (what a happy surprise!)
Obviously, given the marginal sums, n1,. and n.,1, n1,1 must be less than the smaller of these
two and at the same time greater or equal to 0, and the sum of all these probabilities must sum
up to 1. So in order to establish significance, one should not only calculate
P X 1,1=n 1,1n.,1 , n. ,2 , n 1, . , n 2, . for the n1,1 in question, but also for all numbers greater than
n1,1 (in the example this is not possible since n1,1 is already as big as n.,1). Then one establish
-levels and make statements about significance.
page 5 (57)
When numbers get larger, exact tests like the one above R has implemented Fisher's exact
test, which es very similar to what was done above become very time consuming even for
a computer and approximations are used. Like for many other tests the -distribution can be
used. The test statistics ca be calculated from the data using the formula:
2
nn 1,1n 2,2n 2,1n 1,2
=
n . ,1n .,2n 1, .n 2, .
This statistics works if n>60 and should be tested against -distribution for 1 degree of
freedom. It was originally suggested by Pearson
For sample size between 20 and 60 one should either use a correction suggested by Yates:
nn1,1n2,2n 2,1n 1,2 n/ 22
corr=
n.,1n. ,2n 1, .n2, .
which again can be tested against -distribution for 1 degree of freedom, or one can use
Fisher' s exact test directly. In R the call is Fisher.test( ).
Linear modeling
Linear Modeling is used for modeling x and y data when both x and y are f continuous scale,
in fact in principle x and y are in RM and RK respectively.
Now if f in the above discussion is too complex, it is hard to impossible to find, particularly if
there are many x- and y-variables (i.e. if x and y are vectors). So the idea is to restrict f in
such a way, that it belongs to certain classes of functions. The easiest type of functions are
linear functions, next easiest are polynomials, splines, step-functions and the like.
When talking about linearity it is always important to be aware of what exactly is meant by
linear, or precisely what depends linearly on what. A model can by such that y is linear in
x, i.e. y is proportional to x; if b be the proportionality constant, which can be interpreted
geometrically as a gradient, y = xb. Linear Modeling does more than modeling y that is linear
in x! This is very important! Linear Modeling is concerned with modeling dependencies of
different kinds, where models contain unknown coefficients (like the b in what we just
discussed), and y depends linearly upon these coefficients!
So the linear model y = xb, the affine model y = a + xb, the square model, y = a + xb + x2c,
as well as all polynomial models, step functions and regression splines belong to the class of
page 6 (57)
linear models, because they are all linear in the coefficients, the coefficients that are to be
estimated by data analysis.
In linear modeling there may be one or many y-variables and one or many x-variables and the
relation between them is supposed to be of the type: Y = X B + E
where Y is a (N*K) matrix of y-values, q variables (columns) and N observations (rows),
where X is a (N*M) matrix of x-values, p variables (columns) and N observations (rows),
where B is a (M*K) matrix of coefficients, linking the x and the y variables,
and E is a (N*K) matrix of errors in the measurements of the y-values.
The question of course is, how to get the coefficients? We'll see that later!
A second example is the case where there are two x variables and one y variable, so there's x1
and x2 and y (K=3 (why not 2?), M=1, and N=4).
x1 x2 y ^y res
-1 -1 3 2,5 0,5
1 -1 5 5,5 -0,5
-1 1 7 7,5 -0,5
1 1 11 10,5 0,5
page 7 (57)
The linear model (in this special case linear also in the x-variables!) has the form y = b0 +
b1x1 + b2 x2, + , ~ N(0,). Using x and y data as in the table above right, the coefficients
would be: b0 = 6.5, b1 = 1.5 and b2 = 2.5, and This in turn would yield the model-predictions
in the column at top extreme right. Coefficients can be interpreted as height and
gradients. So 6.5 is the mean height of the plane, 1.5 is the gradient (steepness) when going
from left to right, 2.5 is the gradient when going from front to behind. Using predictions, ^y,
one can again calculate residuals as differences to the measured y and then estimate (= 0.5,
depending on the denominator used) as the standard deviation of these residuals. This is part
of the model check as in (A) in the first example.
x1 x2 y ^y res
-1 -1 3 3 0
1 -1 5 5 0
-1 1 7 7 0
1 1 11 9 2
Which set of coefficients is better and why? Well, look at the residuals, defined as the
difference between y and ^y: In the case at the top, the residuals absolute values sum up to 2,
as in the case at the bottom! So there's no difference here. But in the case at top the residuals
are more homogeneous (i.e. more evenly distributed), their absolute values are all the same.
In fact due to this their sum of squares is smaller: yi y i 2 =1 for the first model and
i=1...4
So in the sense of least squares the first model is better. It is in actual fact the best as can
be tried out by checking out different sets of coefficients.
Now using the matirx calculus as introduced above, Y would be a 4*1 matrix, X would be a
4*3 matrix and the coefficients, B, would be represented by a 3*1 matrix, to get the equation
Y = XB.
3 1 1 1 b
0
Y = 5 , Y = X B= 1 1 1 2
b 1 So yi y i 2 =Y Y =Y Y T Y Y .
7 1 1 1 i=1...4
11 1 1 1 b2
To minimize this, minimize Y XBT Y XB=Y T BT X T Y XB by differentiating
w.r.t. B using the multiplication rule and setting this to 0:
X T Y XBY T BT X T X = 0 . Since the two summands are transposes of each
T
other, the sum can only be zero, if each summand is zero, so X Y XB=0 and hence:
page 8 (57)
X T Y XB=0 => X T XB= X T Y => B= X T X 1 X T Y .
Notice that only due to the fact, that the model is linear in B, the least squares functional is
quadratic in B, its derivative is again linear in B, and equating it to zero means solving a
linear set of equations, which can be done by mere matrix inversion. So linearity in B is
important. Otherwise finding the correct B would involve solving a non linear set of equations
which is a lot more difficult. Solving this would involve using classical interactive methods
like the Newton-Raphson algorithm or more modern computer based methods like Simulated
Annealing to reach a solution.
Note that in the case above the X'X-matrix is diagonal. This is because the columns of X are
orthogonal (i.e. the scalar products of different columns are 0). The inversion of X'X is just
done by inverting the diagonal elements.
If there are small perturbations of the X-matrix, X'X is no longer diagonal, (X'X)-1 is
somewhat more difficult to calculate, but if the perturbations are not too big, this poses no real
problem for the computer. However, if perturbations are too large, problems may arise with
the inversion of X'X.
Adding observations (rows in x) leaves the structure of X'X unchanged, but tends to make the
numbers in X'X larger, hence those in (X'X)-1 smaller. Adding coefficients to the model adds
columns to X makes the size of X'X larger. In this case care has to be taken, that there are
enough observations and that X'X does not degenerate but stays invertible.
page 9 (57)
Linear Modeling using R (another example)
To illustrate the possiblities of linear modeling we shall work with a small data set, in which
we try to predict body size of persons from their shoe size.
(Imagine you're late for your basketball training session, your friends are all on the field in
the gymnasium and already dribbling. You enter the changing room shoes everywhere
and you want to know if the people there are taller or smaller than you, because basketball is
much more fun when everyone else is smaller than you ....)
shoesize bodysize
190
1 38 153
2 38 161
185
3 39 167
180
4 39 169
175
5 40 173 krpergre
6 40 176 170
7 41 182
165
8 41 181
160
9 42 188
10 42 189
155
We want to find the coefficients b0 and bshs, because this will allow us to predict body size
from shoe size in the future. Ideally we also want an estimate for in order to judge the
quality of the predictions made the bigger is, the less we will have confidence in our
predictions! (This is part of model diagnostics).
shoesize<-c(38,38,39,39,40,40,41,41,42,42)
bodysize<-c(153,161,167,169,173,176,182,181,188,189)
bspl<-data.frame(shoesize,bodysize)
fm<-lm(bodysize ~ shoesize, data=bspl) # fits the model
summary(fm) # outputs the model:
Call:
lm(formula = bodysize ~ shoesize, data = bspl)
Residuals:
Min 1Q Median 3Q Max
-5.6000 -0.8125 0.1250 1.7625 2.7500
Coefficients:
page 10 (57)
Estimate Std. Error t value Pr(>|t|)
(Intercept) -132.1000 22.9163 -5.764 0.000422 ***
shoesize 7.6500 0.5725 13.361 9.42e-07 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residuals: at each x, theres a yobs (the dot in the graph) and a ypred (the position on the line
at the x-value in question), the difference, yobs ypred, is the so-called residual.
Min, 1Q, Median, 3Q and Max are respectively Min, first quartile, median, third quartile and
max of the residuals.
Coefficients: there are two coefficients, b0 also called the intercept and bx the
gradient to describe the regression line, given by ypred = b0 + bxx. The function call to
lm(formula = y ~ x, data = dummy)
uses the method ordinary least squares to fit the model (linear function) to the data, i.e.
the expression
SSres = (yi (b0 + bxxi))2 is minimized (SS means Sum of Squares).
It is possible in R to utilize so called weighted least squares, meaning that different
observations are give different weight. To do this call
lm(formula = y ~ x, data = dummy, weight=w^2)
so that
SWSres = wi2 (yi (b0 + bxxi))2 is minimized
To be able to use weighted least squares the weights have to be known in advance.
Typically wi = 1/si where si is an estimate of the standard deviation of the measurement(s)
at the point i. In the shoesize/bodysize example above, no reasonable estimate for si can be
given. Hence no weights are used in that example.
Residual Standard error (RSE): The residuals have a mean value, thats zero (because
otherwise a model with a better SSres could easily be constructed by just adding this mean
value to the intercept-coefficient). But the standard deviation of residuals is neither zero
nor one. The residual standard error is almost the standard deviation of the residuals,
sresidual, but not quite!
sresiuals := {1/N (yi (b0 + bxxi))2 }1/2 = (SSres / N)
whereas
RSE := {1/DFres (yi (b0 + bxxi))2 }1/2 = (SSres / DFres)
A sharp eye is needed to spot the difference: it's in the denominator!
page 11 (57)
sresidual2 is sometimes called the ML-estimator (maximum-likelihood-estimator) of variance,
whereas RSE2 is the REML-estimator (restricted-ML-estimator) of variance.
(Maximum-likelihood estimation will be treated in a later chapter).
Degrees of Freedom (DFres): Each observation has the freedom to be what iti is, hence
each such value adds a degree of freedom to the system, and each coefficient in the
model takes one such degree of freedom away from the residuals, so
DFres = #observed values #coefficients = 10 2 = 8 (in our shoesize/bodysize
example).
The idea is: if you fit a model with two coeffients to two values, there are no residuals,
because the line just satisfies the system of linear equations: yi = b0 + bxxi for i = 1, 2.
Hence SSres = (yi (b0 + bxxi))2 = 0.
The sum will become nonzero if a third observation is added (and the system is no longer
solvable) and it grow with every additional observation to be fitted. So in fact its the
number of these additional observation values that should count toward DFresid, which
should in turn be used to standardize the sum of squares.
F-statistic: This is the tool for testing if the regression coefficients are statistically
significant. An F-value is always the quotiont of two variances (i.e. two MS-values), in
this case MSregression / MSres, where MSregression is the explained variance:
MSregression = SSregression / DFregression = ((b0 + bxxi) ymean )2 / 1.
Why divided by 1 and why write it down? Again this concerns degrees of freedom, but
this tim not residual degrees of freedom, DFresiduals, but DFregression. In this case only the
coefficient bx contributes to the variation of the regression model, i.e. the prediction values
ypred,i = b0 + bxxi. R can be made to calculate MSregression using the command:
sum((fm$fitted.values - mean(fm$fitted.values))^2),
where fm$fitted.values can be substituted simply by fm$f.
In the case above shoesize example , the value of the F-statistic is 178.5 on 1 and 8
degrees of freedom, because DFregression is 1 and DFresidual is 8. Using R this can be verified
using the formula:
sum((fm$f-mean(fm$f))^2)/(sum(fm$re^2)/8)
page 12 (57)
When checking the theoretical F-distribution (after R.A. Fisher) on 1 and 8 degrees of
freedom, the corresponding p-value would be 9.416e-07. So a value of 178.5 would
have a probability of ~ 10-6, if the x and y had no connection and varied completely
randomly. This in turn can be checked using:
pf(sum((fm$f-mean(fm$f))^2)/(sum(fm$re^2)/8),1,8,lower.tail=FALSE)
This means that it is very, very, very improbable to have an F-statistic of 178.5 on 1
and 8 DF assuming the null-hypothesis that the y-data are completely random So this
null-hypothesis must be rejected at 90%, at 95%, at 99% even at 99.9%.
Estimate, Std. Error, t value Pr(>|t|): Coefficients can be tested to be significant. This is
done by comparing their value (estimate) to their Std. Error.
The formula for the estimate is b = (XTX)-1 XTy (in the unweighted case), where b is the
coefficient vector consisting of b0 and bx, y is the vector of y-values, and X is a matrix
with 2 columns, one of which is filled with 1s, and the other is the vector of x-values. XT
is the transpose of X (i.e. rows and columns exchanged, so XT has two rows (the first of
which is just 1s) consisting of 10 values.
The formula for the Std.Errors is calculated from the sqare root of the diagonal of (XTX)-1
(which is a 2 by 2 matrix) multiplied by RSE (see above). Since RSE is 2.561 and since
the two Std. Errors are 22.9163 and 0.5725, the diagonal elements must have been
8.948 bzw. 0.2235 (their values are not displayed by R).
Judging th quality and the limits of a model, i.e. understanding the statistical output in the last
section is only one part of statistical analysis. The other part, particularly when doing Data
Mining is understanding the model. To do this it is necessary to understand the meaning of the
coefficients. One part is understanding what is the physical unit of the coefficient, given
these units for x and y, the other part is understanding the value of the coefficient. Both is not
always easy. In fact coefficients are often difficult to interpret directly in the context of the
application.
In the example above, one coefficient is -132.1, the other is 7.65. What do these mean and
what are the associated physical units? It doesn't make much sense to say: body-size 132.1
cm! But this is the would-be size of a (obviously non-existing) person with shoe size 0.
Imagine extrapolating the line on the right toward 0 on the x axis; that's why it's called
"intercept".
The other coefficient is just the gradient of the line. So it's physical unit is not cm, but cm per
unit of shoesize, cm/sz-unit, and it's value can be read off the graph of the fitted function:
increasing shoe size by 1 unit increases body size by 7.65 cm, this means going from shoesize
page 13 (57)
38 to shoesize 42 increases bodysize by 4 sz-unit times 7.65 cm/sz-unit = 30.6 cm (from
roughly 158 to 188 cm).
190
185
Centring means, that the interval for x is shifted
in such a way, that it is centred around 0. In the
180
example the x-values vary from 38 to 42, hence
175
krpergre
xcentre = 40, and x - xcentre varies from -2 to 2, an
170
interval with centre at 0.
165
The data in the example have been designed i.e.
160
chosen at the will of the investigator. In such a 155
In R centring can be performed by changing the x-variables after the ~ [please use I( ) for
technical reasons].
> fm_s=lm(bodysize ~ I(shoesize-40), data=bspl)
> fm_s
Call:
lm(formula = bodysize ~ I(shoesize - 40), data = bspl)
Coefficients:
(Intercept) I(shoesize - 40)
173.90 7.65
Centring has improved interpretability of the result. Why? Because now the intercept, here
173.9 is where the regression line crosses x - xcentre = 0, and this can be interpreted as a value
for the y values at the centre of the investigated x-region not somewhere far away at a non-
existant shoesize 0! So 173.9 cm corresponds to the size of a person with a shoe size of 40
(because for x = 40) the centred x-variable is 0.
Scaling means that x-variables are dilated (stretched or shrunk) after centring depending on
the size of the interval through which x is varied in order to make coefficients if different
model terms comparable. After scaling x-variables and fitting the model for these scaled (and
page 14 (57)
centred) variables all coefficients will be of the same physical unit as the y variable, and
hence all be comparable, even if the origiinal x-variables had different units.
There are two ways commonly used to scale data after centering:
(A) min-max-scaling (sometimes called orthogonal scaling)
(B) unit-variance-scaling.
x x centre
In min-max-scaling the formula is: x sc=
x max x centre
On first encounter scaling does not seem to have improved the situation at all. The value of
the intercept has not changed compared to just centering. Only the value of the coefficient of
shoesize term has changed. It is now twice as big. So what?
With just centering (or with no preparation at all) the coefficient could be interpreted as
measuring the average change in size, when shoesize changed by 1. This change was 7.65 cm.
Now the coefficient measures the change that occurs, when you change from the centre-value
of 40 to the maximum value of 42, respectively to the minimum value of 38. This has two
advantages that may often be useful in application:
It is easy to read off the lowest possible and the highest possible prediction values from the
coefficients: Highest is coeffintercept + coeffshoesize, and lowest is coeffintercept coeffshoesize, hence in
the example the lowest predicted value would be 158.6 = 173.9 15.3, the highest predicted
value would be 189.2 = 173.9 + 15.3;
It would be easier to compare coefficients (see remark above): If we had had a model that
considered other influencing factors (x-variables), that may have an effect on body size, like
possibly sex, age, thumbsize and so on, and all such factors were centred and scaled they
would all have become dimensionless by scaling and centring and all coefficients would have
taken on the dimension of the response, in our case of body size, which is cm. (Before scaling
the unit would have been cm/unit of shoesize, or cm/per year of age, or cm/mm of thumbsize.)
page 15 (57)
x x mean
In UV-scaling, the formula is: x UV = where xmean and s are mean and standard
s
deviation of the data. The scaling is called Unit-variance because after scaling has been done,
the variance of the variable will just be 1, as of course will be the standard deviation which is
just the square root of variance.
In the case for the bodysize/shoesize example a model that is linear in the x-variable shoesize
seemed quite adequate. Looking at the plot of the regression function and the observed values,
there was good correlation (measured by the dimensionless R-measure) and small deviation
(measured by RSE in the unit cm). In many real life Data Mining problems linear models are
not adequte. There are different ways in which models can be improved.
One idea, that might improve the model, is using a model of the form:
ybs = b0 + bshsxshs + bshs*shs xshs + , where ~ NID(0,)
This model will allow for some curvature instead of just fitting a straight line:
Using R this can be done by:
fmsq<-lm(bodysize ~ shoesize + I(shoesize^2), data=bspl)
The I(..) around the shoesize^2-term is necessary to avoid that R evaluates the squaring
immediately but rather just inside the lm( )-function, when needed to do the fitting.
The result is the following:
Note that there is a bad surprise in the result we get (see below)!!!
> summary(fmsq)
Call:
lm(formula = bodysize ~ shoesize + I(shoesize^2), data = bspl)
Residuals:
Min 1Q Median 3Q Max
-4.52857 -0.84643 0.06429 0.98929 3.47143
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -988.1714 761.0281 -1.298 0.235
shoesize 50.5071 38.0865 1.326 0.226
I(shoesize^2) -0.5357
0.4760 -1.125 0.298
160 165 170 175 180 185
freedom
Multiple R-Squared: 0.9637,
Adjusted R-squared: 0.9533
F-statistic: 92.87 on 2 and
7 DF, p-value: 9.13e-06
schuhgre
page 16 (57)
both signs that the model has a better fit than the linear model above. That's not a bad
surprise.
>plot(shoesize,fitted (fmsq))
>points(shoesize, bodysize)
>points(shoesize, fitted(fmsq),col="red")
>lines(formula=fitted (fmsq)~shoesize, col="red")
>abline(coef(fm), col="green")
This situation can further be improved by centering and scaling! This will be illustrated using
the min-max scaling technique already discussed above:
> fm_sc_sq=lm(bodysize ~ I((shoesize-40)/(42-40)) + I((shoesize-40)^2/4),
data=bspl)
> fm_sc_sq
Call:
lm(formula = bodysize ~ I((shoesize - 40)/(42 - 40)) + I((shoesize -
40)^2/4), data = bspl)
Coefficients:
(Intercept) I((shoesize - 40)/(42 - 40))
174.971 15.300
I((shoesize - 40)^2/4)
-2.143
> summary(fm_sc_sq)
Call:
lm(formula = bodysize ~ I((shoesize - 40)/(42 - 40)) + I((shoesize -
40)^2/(42-40)^2, data = bspl)
Residuals:
Min 1Q Median 3Q Max
-4.52857 -0.84643 0.06429 0.98929 3.47143
page 17 (57)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 174.971 1.241 140.955 2.39e-13 ***
I((shoesize - 40)/(42 - 40)) 15.300 1.126 13.582 2.76e-06 ***
I((shoesize - 40)^2/4) -2.143 1.904 -1.125 0.298
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
When checking significance of the coefficients, we see that now the intercept and the linear
coefficients are both statistically significant. Only the squared coefficient is not significant.
Thi is a considerable improvement, considering that without centring and scaling none of the
three coefficients are significant, and in particular the coefficients cannot be interpreted /
understood.
With centring and scaling interpretation is easy. Intercept and linear coefficient can be
interpreted as in the linear model (see above), and the interpretation of the square term is just
as easy: Due to centring and scaling the extreme values for x are 1 and 1, so x is maximally
+1 and if the coefficient of the square term is bxx, it means that the maximal correction of the
linear model due to the taking up the square term in the model. In this example this correction
is very small, namely 2.143 cm, that statistical analysis (t-test) classifies it as insignificant.
Just check RSE, which is still 2.519.
Stepwise regression is a procedure, that chooses model coefficients out of those given in the
formula-specification in the lm()-function call, in order to make a parsimonious model (i.e.
with as few coefficients as possible).
The function step( ) uses the so-called Akaike Information Criterion (AIC) to decide which
terms to eliminate from the model. AIC is somewhat difficult to understand, it can be
calculate using AIC = 2loglik + 2p, where loglik = i=1,,N log (Pr true estimate (yi)). loglik (log-
likelihood which will be explained in the section on Maximum Likelihood) is a negative
number; its absolute value gets smaller when the fit gets better. Since adding a model term
would add 2 to AIC (makes AIC worse) loglik has to improve by at least 1 to compensate this
increase.
Below is the result of applying step() to fmsq (see section above). The output should be
interpreted as follows (see below):
Start: AIC= 20.91: AIC was originally 20.91, since p=3, loglik is 7.455=(20.916)/2,
- I(schuhgre^2): Taking away the square term gives an AIC of 20.573, since p is now 2,
loglik is 8.286=(20.5734)/2,
<none>: leaving the model as it is, i.e. taking away nothing, obviously doesn't change the AIC
and doesn't change loglik,
page 18 (57)
- schuhgre: taking away the linear term increases AIC 21.151, so since again p=2, loglik
is 8.575=(21.1514)/2.
So the model krpergre ~ schuhgre is best as it has the lowest AIC, and hence the
highest loglik of all models with p=2 and this model investigated further. Can leaving out
more terms improve the model again?
Step: AIC= 20.57: This is the AIC we continue with
<none>: leaving the model as it is, i.e. taking away nothing, obviously doesn't change the AIC
and doesn't change loglik,
- schuhgre: taking away the linear term increases AIC to 50.06, and since now p=1 (just
the intercept) loglik is now 24.03=(50.062)/2. For this model both AIC and loglik are very,
ver8.575=(21.1514)/2y bad, so the linear model is best. For this model, coefficients are
given. They correspond to the coefficients in the sections above.
> step(fmsq)
Start: AIC= 20.91
krpergre ~ schuhgre + I(schuhgre^2)
Call:
lm(formula = krpergre ~ schuhgre, data = bspl)
Coefficients:
(Intercept) schuhgre
-132.10 7.65
In the example just discussed, the x-variables were not scaled and centred. Scaling and
centring does not change values of loglik or AIC, so for choosing the best model using the
step( )-function it does not matter, whether scaling and centring has been done. It does
matter when interpreting coefficients, though as has been discussed in sections above.
page 19 (57)
To do this in R first transform y using log(), and then fit log(y) using lm( ). For prediction the
log has to be inverted. logarithm is used when errors are relative, but any other invertible
transformation can be used, like y1/2, neglog(y)=-log(1-y), logit(y)=log(y)-log(1-y) etc. can be
used.
Please note that R does not follow the usual naming-convention for logarithms normally
ln( ) denotes the inverse of of exp( ), and log( ) is log10( ), i.e. the inverse of 10x in R
however log() is inverse of exp( ) and log10( ) should be used to get the usual log10().
> points(shoesize,bodysize)
> points(shoesize,
180
exp(fitted(fmlog)), col="red")
> lines(formula=
175
exp(fitted(fmlog)) ~ shoesize,
krpergre
col="red")
170
> abline(coef(fm),col="green")
165
After fitting a model and afer investigating general model properties, like coefficients, R-
value, RSE (residual standard error), it is absolutely necessary to check residuals to get an
idea, how the model can possibly still be improved.
page 20 (57)
There are several characteristic plots, which give precise information on how to judge and
improve a model. These techniques work best for design data but can also be used albeit
with care for ad-hoc data.
Diagnostic plots
Name What to look for? What to do?
Res vs variable curvature take up square term
Obs vs Fit strong outlier lonely outlier make sure the obs value was correct
Res vs Fitted trumpet-form transform y variable
N-Plot / qq-plot outlier top right / bottom left don't trust model here
3,00+07
20
Observed
2,00+07
2
6
18
1,00+07 11 14
169
15 13 12
0,00+00 3814 17
57 19
0,00+00 1,00+07 2,00+07 3,00+07 4,00+07
Predicted
pred
Normal probability plot 20 400
quantile/quantile plot:
One further diagnostic tool is the so called normal probability plot, also called qq-plot
(quantile-quantile-plot), depending on which way the axes are chosen. The idea in this plot is
to plot the theoretical position of the i-th value of n normally (or otherwise) distributed values
against the true positions of the measured values in this case the residuals.
page 21 (57)
The idea is, one wants to test if same data are normally distributed it works the same for
other distributions (e.g. Weibull):
R uses the qq-plot for regression diagnostics in which x- and y-axes are exchanged, i.e.
theoretical values are on the x-axis and outliers are on the left very far down or on the right
very high up in the plot.
Residuals Rankit Plot
Theoretical Quantiles
page 22 (57)
Blood Cell Count example (8 x-variable, one y-variable)
To exemplify model diagnostics and to illustrate some of the possibilities we shall investigate
the following example:
The data consists of cell-count data for different leukocyte-subtypes of healthy children. The
frist column is age in months. The counts are given in number of cells per l of blood. They
have been measured using flow-cytometry, an analytic method in which cells are marked and
coloured according cluster molecules that they present on their surface. Once marked they can
be counted using powerful microscopes and software.
Leukocyte is another name for white blood cells. Subtypes are Lymphocytes, Monocytes
and Granulocytes (the latter have been excluded from this study). The Abbreviation CD
means Cluster Designation and is a state of the art way to classify cells in the immune system.
In fact CD14+ are monocytes. Lymphocytes have three subtypes: CD3+ are T-cells (cells that
have been trainied in the thymus); CD19+ are B-cells (grow up in the bone marrow and are
not trained outside); CD56+ are natural killer cells. T-cells are again differentiated into
CD3+CD4+, so called T-helper cells and
CD3+CD8+, so called toxic cells.
6000
cells.
3000
> attach(ss08_0423)
> plot(age,CD3p)
1000
page 23 (57)
Fitting the model and plotting the line using
> mod<-lm(CD3p ~ age, data=ss08_0423)
> abline(mod$coef)
2000
gives the line as above.
1000
mod$resid
Residuals i.e. the difference between measured
0
values and fitted.values can be plotted using
> plot(age,mod$resid)
-1000
> abline(0,0)
-2000
This gives the plot at right. Obviously the residuals are
not homogeneous. This means the model has to be 0 50 100 150 200
done in the lm( )- function, requires homogeneous error variance, and inhomogeneous
residuals are a very strong indication that this assumption is violated!
2000
> plot(mod$f,mod$res) 1000
mod$f
2 - transform the marker values using a contracting transformation (like log(...) or root(...))
before modelling them. Taking logs is a good way to treat an error structure that is
multiplicative and not additive (i.e. relative errors are homogeneous).
3.2
alter
page 24 (57)
transforms age into "immunological age". This should be done with data from outside of this
data set, in order not to "sacrifice informaion".
In the following considerations we transformed age by taking square roots. This is not quite
satisfying, because above say 20 years of age the immunological status does not change much
any more.
3.8
relation looks like this:
3.6
e.g. for CD3p:
> mCD3p<-lm(CD3p ~ age,data=ss08_0423)
3.4
CD3p
> abline(mCD3p$coef)
3.2
quality is to plot the histogram of the residuals:
3.0
> hist(mCD3p$residuals)
values.
15
0.2
10
0.1
5
0.0
mCD3p$res
0
-0.1
mCD3p$residuals
-0.2
The commands
-0.3
> plot(fitted(mCD3p),resid(mCD3p))
-0.4
> plot(mCD3p$f,mCD3p$res)
3.1 3.2 3.3 3.4 3.5 3.6
are identical.
mCD3p$f
Put the line in using
> abline(0,0)
There is a slight problem with the three values at the bottom right. So it seems rather to be a
problem with the three values than with the model. But one is never 100% sure!
page 25 (57)
Estimating Confidence intervals and prognosis intervals for linear
models
Prognosis error for a new observation at the prediction point cannot degenerate to a single
point just by repeating the design: every new observation has its intrinsic right to err, and
the expected size of this error cannot converge to zero by merely repeating experiments. The
situation is well known from quality control: It is one thing to ask how does a mean value
deviate the answer being +/ s/ n and another to ask how does a new measurement
deviate (given the information from the design) the answer here being +/ s(1+1/n).
This is standard error, in order to get the corresponding confidence- and prognosis intervals
just multiply by the corresponding t-value, i.e. ~2 for 95% and ~3 for 99,5% confidence level.
Simple Bootstrap for estimating standard deviation and percentile intervals for
characteristics like mean value, median, standard deviation, correlation coefficient
The famous Baron von Mnchhausen succeeded in ... escaping from a swamp by pulling
himself up by his own hair (or bootstraps, depending on who tells the story). Well in German
versions of the story it was ..by his own hair, in most English versions it's by his bootstrap.
In any case Bradley Efron, in 1979 inventor of the Bootstrap-method for estimating statistical
error and for determining confidence intervals, Baron v. M. was inspiration at least for the
name of the method ((see [ET], p. 5)!
The bootstrap consists of taking samples from the sample in order to induce variation.
Imagine you had a sample of n observed values. Then calculate your favourite characteristic,
e.g. a median or a correlation coefficient. In order do calculate its standard deviation or its
confidence interval, construct an virtual n faced die, where you have marked the n faces
with the n observed values (or pairs of values). Now use the die get R = 1000 samples of size
n (you need to throw n*1000 times). For each such sample calculate the characteristic of
interest. Sort the 1000 characteristic values and pick out the percentiles of interest. For 95%
percentiles pick the 25th and 975th values.
page 26 (57)
for(i in 1:n){
z_star <- sample(z,size=length(z), replace=T) #bootstrap samples
charv[i] <- mean(z_star) #bootstrap charact.
}
#Quantile, Quartile, Histogram
sort(charv)[c(975,500,25)]
sort(charv)[c(750,500,250)]
hist(sort(charv))
page 27 (57)
the validity of of the model and the distribution of the experimental errors play a less
important role than in the classical method and the Monte Carlo method. As in the Monte
Carlo method random sampling, i.e. throwing a die, is done, albeit not from a statistical
distribution, whose shape (usually normal) must be assumed, but directly from the residuals,
to be precise: from deleted residuals out of cross validation. Residual, ei = yi y^i and deleted
residual ri = ei / (1 hi).
Now draw samples of size n ( = run number in the design) with replacement, call them r*r,
and do this say R = 1000 times, so r = 1, , R. (Please note, there are many, many more such
samples that can possibly be drawn, namely (2n- 1)!/(n!n!) of them.) In order to simulate
estimation error, determine R = 1000 new models from the right sides y*r = y^+ r*r, i. e.
calculate R = 1000 sets of coefficients, b*r, and at a new prediction point, x+, R = 1000 new
model estimates, y+*r. Sort these and like in the Monte-Carlo method and pick out entry no aR
and (1-a)R, in order to get Percentile intervals for the model estimators.
To get prognosis intervals for a single new prediction sample a second time from the deleted
residuals, M = 1 or 2 times for each of the R = 1000 runs above would be sufficient let
these results be called e*rm, r = 1, R, m = 1, , M. Now simulate prognosis error using
d*rm = y+*r (y^+ +e*rm ),
and simulate the new observation response values using,
y^*rm = y^+ d*rm = 2 y^+ y+*r +e*rm.
Sort these and pick out the entries aRM and (1a)RM.
page 28 (57)
#The 2 level bootstrap draws randomly from stud. Residuen
for(i in 1:n){ #outer level for coefficients
z_star <- p + sample(r,replace=T) #Residuen wrfeln
b_star <- I%*%t(X)%*%z_star #bootstrap-coefficient
p_star <- t(x_plus)%*%b_star #pred. with bootstrap-model
epsil <- sample(r,size=m,replace=T) #inner level fr pred. error
delta[i,] <- p_star-(p_plus+epsil) #matrix of simulated error
}
#Quantile, Quartile, Histogram
p_plus-sort(delta[,])[c(m*975,m*500,m*25)]
p_plus-sort(delta[,])[c(m*750,m*500,m*250)]
simulated_new_measurements<-p_plus-sort(delta[,])
hist(simulated_new_measurements)
#cleaning up
detach(boot_ex)
For a symmetric distribution this confidence interval is symmetric about the emp. value.
For a left-skew distribution obviously the confidence interval of a position parameter will be
right skew (and vice versa) as can be seen above.
page 29 (57)
But using a transformation to symmetrize the distribution
forward
transform
backward
transform
. the distribution for the transformed values are symmetric, hence the confidence interval
for the transformed entity is symmetric and the backward-transformed interval will be
left-skew!
page 30 (57)
General linear modeling (GLM)
As we have when the words linear "linear modeling" are used, what is meant is not
necessarily y = b0 + b1x1. Of course this is a linear model. It is linear in the coefficients b and
in the variables x. "Linear Modeling" refers to models that must be linear in the b's but not
necessarily in the x's. In fact it is the strength of linear modeling that it allows non-linear
terms in x!
y may be a vector of responses, g( ) is a so called link function (like log() in the example of
the last lecture) that must be invertible, x is the vector of x-values, x1, xm, hm(x) are
functions of this x, bi are the coefficients that are to be estimated from the collected data.
Models of the types (1) and (2) have been considered above, at least for x in one dimension
(but using 2 dimension does not introduce any new concepts yet )
Coming back to the example of shoe-size: We fit a piecewise constant function using R:
Define indicator functions for the intervals for the intervals I1 = I(37.5, 39.5], I2 = I(39.5, 41.5] and
I3 = I(41.5, 43.5].
> I1<-function(x) {if(x>37.5 &&x <=39.5) 1 else 0}
> I2<-function(x) {if(x>39.5 &&x <=41.5) 1 else 0}
> I3<-function(x) {if(x>41.5 &&x <=43.5) 1 else 0}
page 31 (57)
> bspl<-data.frame(schuhgre,krpergre)
> bspl
schuhgre krpergre
1 38 153
2 38 161
3 39 167
4 39 169
5 40 173
6 40 176
7 41 182
8 41 181
9 42 188
10 42 189
Trying to fit a linear model directly using Ij(x) will not work in R,because the functions Ij have
only been defined for numbers and not vectors.
So
> fm_pc<-lm(krpergre ~ 0 + I(I1(schuhgre)) + I(I2(schuhgre)) +
I(I3(schuhgre)), data=bspl)
yields the following error:
Fehler in model.frame(formula, rownames, variables, varnames, extras,
extranames, : Variablenlngen sind unterschiedlich (gefunden fr
'I(I1(schuhgre))')
One way of fixing this, is to construct columns for the three Ij(x) using the apply( )-function in
R to construct the vectors I1(x), I2(x) and I3(x):
> apply(as.matrix(bspl$schuhgre),1,I1) # ,1, means "column"
[1] 1 1 1 1 0 0 0 0 0 0
> apply(as.matrix(bspl$schuhgre),1,I2)
[1] 0 0 0 0 1 1 1 1 0 0
> apply(as.matrix(bspl$schuhgre),1,I3)
[1] 0 0 0 0 0 0 0 0 1 1
Call:
lm(formula = krpergre ~ 0 + I(apply(as.matrix(schuhgre), 1, I1)) +
I(apply(as.matrix(schuhgre), 1, I2)) + I(apply(as.matrix(schuhgre),
1, I3)), data = bspl)
Coefficients:
I(apply(as.matrix(schuhgre), 1, I1)) 162.5
I(apply(as.matrix(schuhgre), 1, I2)) 178.0
I(apply(as.matrix(schuhgre), 1, I3)) 188.5
> summary(fm_pc)
Call:
lm(formula = krpergre ~ 0 + I(apply(as.matrix(schuhgre),
1, I1)) + I(apply(as.matrix(schuhgre), 1, I2)) +
I(apply(as.matrix(schuhgre),
1, I3)), data = bspl)
page 32 (57)
Residuals:
Min 1Q Median 3Q Max
-9.500e+00 -1.875e+00 -6.939e-16 3.750e+00 6.500e+00
Coefficients:
Estimate Std. Error t value Pr(>|t|)
I(apply(as.matrix(schuhgre), 1, I1)) 162.500 2.735 59.41 1.01e-10
***
I(apply(as.matrix(schuhgre), 1, I2)) 178.000 2.735 65.07 5.32e-11
***
I(apply(as.matrix(schuhgre), 1, I3))
0
9
1
188.500 3.868 48.73 4.01e-10 ***
---
5
8
1
Signif. codes: 0 '***' 0.001 '**' 0.01
'*' 0.05 '.' 0.1 ' ' 1
0
8
1
Residual standard error: 5.471 on 7
5
7
1
degrees of freedom
Multiple R-Squared: 0.9993,
0
7
1
Adjusted R-squared: 0.999
e
p
k
r
g
r
re
F-statistic: 3379 on 3 and 7 DF, p-
5
6
1
value: 2.008e-11
0
6
1
To get the graph, use the following commands:
>plot(schuhgre,krpergre )
5
5
1
>points(schuhgre,krpergre)
>points(schuhgre,
38 39 40 41 42
fitted(fm_pc),col="red")
>lines(formula=fitted(fm_pc) ~ schuhgre
schuhgre,col="red")
>abline(coef(fm),col="green")
Actually the red line should really be piecewise constant, but the graphics options are not so
easily found to do this.
At least the jump should be vertical at the points 39,5 and 41,5, because that's where the
indicator functions have their jumps. But fm_pc only knows the values of the function at the
points in the vector Schuhgre. That's why the jump is where it is.
Fitting Splines
Just as piecewise constant functions can be fit to the data, we can fit
(a) piecewise linear functions,
(b) piecewise linear functions that are continuous (i.e. linear splines),
(c) piecewise square functions,
(d) piecewise square functions that have continuous derivative (i.e. quadratic splines)
(e) piecewise cubic functions that have continuous second derivative (i.e. cubic splines).
This can be achieved by using an appropriate set of functions hm(x) in the general formula
above.
page 33 (57)
To illustrate the different cases, we continue the schuhgre/krpergre example, and
continue to use the points 37.5, 39.5 and 41.5 as knots, where the discontinuities in the
functions resp. its derivatives may be.
However, the data are too sparse to fit this model, since there is only one measurement point
in the third interval (41.5, 43.5], so that at most we could fit h5(x) here, but not h6(x), (which
in fact amounts to calculating the mean of the two values at schuhgre 42, i.e. 188.5).
In case (b)
h0(x) = 1, h1(x) = x, h2(x) = (x 39.5) I(39.5, inf] , h3(x) = (x 41.5) I(41.5, inf]
Since all of the functions are continuous (the jump in the indicator functions are zeroed out by
the (x xat the junp) factor in front), the fitted model will also be continuous.
Case (c) can in fact not be reasonably fitted with the data at hand, because between each of
two knots there are always only two observation points, so pieces of square functions cannot
be fitted. The model would have too many degrees of freedom, the functions hm(x) would be:
h1(x) = I(37.5, 39.5], h2(x) = xI(37.5, 39.5], h3(x) = xI(37.5, 39.5]
h4(x) = I(39.5, 41.5] , h5(x) = xI(39.5, 41.5], h6(x) = xI(39.5, 41.5]
h7(x) = I(41.5, 43.5] and h8(x) = xI(41.5, 43.5] h9(x) = xI(41.5, 41.5]
Case (d) however is possible, because the condition of continuity of the first derivatives
decreases the models degrees of freedom from 9 to 5, because the function and its derivative
must be 0 at the 2 knot points. The easiest way to satisfy the condition is to fit the following
set of hk(x):
h0(x) = 1, h1(x) = x, h2(x) = x, h3(x) = (x 39.5) I(39.5, inf] , h4(x) = (x 41.5) I(41.5, inf]
Since all of the functions are continuous and have continuous first derivative, the fitted model
will also be continuous and have a continuous first derivative.
As in case (c) in case (e) the data are too sparse to fit a cubic spline, but the basis of hk(x)'s to
be used would be:
h0(x) = 1, h1(x) = x, h2(x) = x, h3(x) = x, h4(x) = (x 39.5) I(39.5, inf] ,
h5(x) = (x 41.5) I(41.5, inf]
page 34 (57)
R-Software would not be very good, if one had to fit splines using the lm( )-function. In fact
there is a function to fit the very popular cubic splines:
smooth.spline(x, y = NULL, w = NULL, df, spar = NULL,
cv = FALSE, all.knots = FALSE, nknots = NULL,
keep.data = TRUE, df.offset = 0, penalty = 1,
control.spar = list())
Check the R-Help-system for details. The parameter list is much more complicated than the
user may have anticipated, because the functionality of smooth.spline() is much greater
than the possibilities within lm(). Details are beyond the scope of this introductory course.
The most common violation of IID are repeated measures. Not always, but usually, this means
that time turns up as a factor, and measurements on the same subject (or patient) are repeated
time after time. 26
26
1
2 1,28571429
T
T
26 3 4,71428571 T
26 4 13 T
In a fixed effects model (i.e. the models we've used until now), all of the gradients (i.e.
coefficients) are assumed to be deterministic; random errors are independent random errors
assumed to occur only in individual experiments, and these errors 1, 2, 3, 4
are independent!
In a mixed effects model, some of the gradients (i. e. coefficients) are assumed to be random
variables themselves (!) This makes things more complicated, non-independent but random errors
1, 2, 3, 4
requires more runs but it can handle correlated errors!
page 35 (57)
For example, we can assume the expectation value of the measurement for a patient at a given
time point is itself random (we usually assume expectations to be ).
To be precise: We assume i is the index for the patient and j is the index for the week, and
yij = 0i + 1iweekj + ij ij ~ NID(ind)
where
0i = 00 + 02 gi + u0i u0i = NID(pat)
1i = 10 + 12 gi
hence
yij = 00 + 02 gi + 10weekj + 12 gi*weekj + u0i + ij
The first term is the mean, the second considers the influence of the group to which the
patient belongs, the third that of the week, the forth is the interaction between the two. These
are the fixed effects. u0i is the random deviation of a patient from the mean, u1iweekj is the
random deviation of the influence of the week from the mean.
This can be modelled using the lme(...)-function in the nlme-library (see step 4 below).
Step 1: Look at the mean values for group "H" and group "T" at each tiw:
page 36 (57)
patients are at the same level, whereas at week 14 and thereafter group-H patients continue to
improve whereas group-T patients seem to stagnate.
Gradients at each tiw are just differences between group-H and group-T. These results are the
same as above.
t- and p-values are slightly different to t-test and linear modelling above. This is because
standard error is calculated from all residuals and not just from the residuals at the week
concerned.
This analysis does not yet use a mixed effects model, it just gives a list of linear models.
Before fitting the model the contrasts used should be set to sum to zero for each variable, so
that they are orthogonal to the intercept. This is most easily done using the options-command.
options(contrasts=c('contr.sum','contr.poly'))
htlme<-lme(log(CD3p+1)~factor(tiw)*factor(group), random=~1|
Pat.ID,data=ht[tiw>=1&tiw<=58,],na.action=na.exclude)
summary(htlme)$tTable
page 37 (57)
Value Std,Error DF t-value p-value
(Intercept) 4,7249 0,1996 756 23,6773 3,74E-093
factor(tiw)1 -3,2200 0,6095 756 -5,2826 1,67E-007
factor(tiw)2 -3,5075 0,1960 756 -17,8977 5,37E-060
factor(tiw)3 -2,8140 0,1736 756 -16,2087 6,18E-051
factor(tiw)4 -1,9226 0,1684 756 -11,4194 5,65E-028
factor(tiw)5 -1,0486 0,1655 756 -6,3351 4,07E-010
factor(tiw)6 -0,6234 0,1655 756 -3,7664 1,79E-004
factor(tiw)7 -0,4079 0,1642 756 -2,4839 0,0132
factor(tiw)8 -0,2553 0,1642 756 -1,5547 0,1204
factor(tiw)9 -0,1176 0,1642 756 -0,7160 0,4742
factor(tiw)10 -0,0593 0,1642 756 -0,3612 0,7181
factor(tiw)11 -0,1117 0,1642 756 -0,6799 0,4968
factor(tiw)12 -0,2108 0,1642 756 -1,2837 0,1996
factor(tiw)13 0,0014 0,1654 756 0,0084 0,9933
.
.
.
Random effects:
Formula: ~1 | Pat.ID
(Intercept) Residual
StdDev: 1.204801 1.011532
Interpretation:
Residual standard deviation is 1.01153, standard deviation of the patient mean is 1.204801.
Since patient standard deviation is relatively high, it is definitely not allowed to treat
measurement errors at different time points as independent.
No effort has been made to model time dependency numerically. Both group-T/H and tiw
were used as (qualitative) factors.
page 38 (57)
It may be interesting to model time dependence using tiw as a numerical variable, with
suitable scaling maybe the change in gradient of the group-effect will be significant. But this
is beyond the scope of this
page 39 (57)
Estimating parameters using Maximum-Likelihood (ML)
There are several ways to estimate distribution parameters in statistics when data are given
that presumably correspond to the distribution. One of the if not the most method is that of
maximizing likelihood. It is more general than least-squares regression, that is used in linear
modeling. In fact least-squares regression is a special case of maximum likelihood estimation
and we shall prove this later in this section.
This shall be illustrated using exponential time decay and a single measurement.
Assume a survival function is given by S t =et and is to be estimated from failure time
point at t = . What's the best ? The cumulative failure probability is given by
t
F t =1 S t =1 e . Of course the probability of a failure being exactly at t = is
exactly 0, because F t =1et is continuous. So we shall use probability density for a
failure, f t = e t . Now given the failure at t = , we turn the logic around and say:
likelihood of a parameter value , L , is the probability density at t = , assuming a
parameter value of . So L = e
In the plot at right, the function f t = e t has Exp-decay for different values of lambda
quickly.
0.6
=0.5 =
2 .
0.0
0 2 4 6 8 10
Indeed, the maximum of L = e can be found t
page 40 (57)
have been made, log-likelihood is easier to use. Of course, since logarithm is a monotonic
function log-likelihood will be maximal at the same values of the parameters as likelihood.
In all following calculations and examples, log-likelihood will be used, not likelihood itself.
The situation becomes a little more interesting, when there are several measurements.
Likelihood of a parameter is again defined as the probability resp. probability density of the
measured values. When measurements are independent the probability of several events
occurring is the product of the probabilities, the same ist true for probability densities. So
likelihoods become products and log-likelihoods become sums (which are a lot easier to
differentiate!).
As an example we use exponential decay again, and assume that there are two independent
failure time points t1 and t2. Then likelihood is
L = dP t failure at t 1 failure at t 2 | = e t e t
1 2
and log-likelihood is
l =log [ dP t failure at t 1 failure at t 2 | ]=log t 1 log t 2 ,
and hence
1 1 2
dl = t 1 t 2 , which is zero if =
t 1t 2 .
In problems concerning failure and survival, it is unusual to use the probability densities for
likelihood calculations, it's better and usual to use hazards, which are in fact conditional
failure probabilities on condition that no failure has occurred up to the considered time point.
This will be explained some more in the section on Cox proportional hazards model.
The maximum likelihood estimation method can also be used to estimate and of a normal
distribution as well as for estimating coefficients in a linear model of the type
ybs = b0 + b1x1 + b2x2 + , where ~ NID(0,) (NID = normally and independently distr.),
as discussed before.
Assuming there is one measurement x1, and x1 ~ NID(,). What would have the highest
2
x1
1
2 2
likelihood? Well L , = e ,
2
2
1 x 1
hence l , = 2 lo g 2 log 2 .
2
From this formula, it can readily be seen, that the maximum-likelihood estimator for must
be the same as the least squares estimator note the minus sign. Taking the derivative yields
x 1
dl , = , which is zero for ML = x1. A best estimate for can obviously not be
2
2
1 x 1
found from just one measurement: dl , = 3 , which cannot be zero, if the
page 41 (57)
numerator in the second fraction is zero, which it is since ML = x1, as was just shown.
Otherwise the best estimate for would be =x 1 .
When there are two different measurements, x1, and x2, both ~ NID(,), then
2 2
x1 x 2
1
2 21 2
L , = e e 2
2 2
x 1 2 x 2 2
l , =log 2 2 lo g 2 2
2 2
x 1 x 2 x 1 x 2
dl , = 2 , which is zero for ML = ,
2
2 2 2 2
2 x 1 x 2 2 x 1 x 2
dl , = , which is zero for ML= .
3
2
Having seen these calculations for one and for two measurements, it is easy to generalize to N
measurements, x1, ... xN. In fact the formulae for both parameters generalize easily:
1 N 1 N
ML = i=1 x i and 2ML= i =1 x i 2 . Here the two parameters have been estimated
N N
indpendently of each other. When as is common practice is estimated first, then in
estimating , the xi only have N-1 degrees of freedom. Hence an N-1-dimensional normal
distribution must be used. This would reduce numerator in the fraction N / to N-1, and
hence in solving the equation for , there would only be N-1 in the denominator. This variant
of ML-esimation is called Restricted Maximum Likelihood estimation, or REML for short.
At the beginning of the course, we divided variables according to there scale and according to
their use. As regards use there are two types, exogenic variables (i.e. those that send out
effects) and endogenous variables (i.e. those that take in these effects). We called the former
X-variables and the latter Y-variables. As regards scale we considered different several scale-
types for variables, namely continuous scales, ordered scales and discrete scales (where in fact
in the last case talking of a scale is maybe a slight misdenomer (Fehlbenennung), because
the word scale is inevitably associated with numbers and discrete scales really just
differentiate between categories or classes, which just exist side by side.)
In a data analysis situation, depending on availability of X and Y-data, depending on the scale
of these data, and of course depending on the questions to be answered, different statistical
methods or modeling techniques shall be applied. In particular, when inferences about Y-data
are to be drawn from available X-data, modeling techniques to be used depend on the scales
of the data involved.
page 42 (57)
Until now we have discussed the following methods
Well, in fact we discussed contingency tables for 2 discrete settings in each of the X and Y
side. anova, we discussed a little in first semester and GLM generalized linear modeling
contains simple techniques of transforming discrete data into continuous data, so one can use
linear modeling to do the analysis.
In fact, when there are two categories1, just use x=-1 for the first and x=1 for the other; when
there are more than 2 classes, DON'T just use one x variable, because this would introduce an
ordering(!), use what's called dummy-variables, that define contrasts between categories, or
use indicator variables x=0/1,that indicate whether an object is not/is in a given category.
Until now, in the Regression-setting, i.e. when we did linear modeling, the Y's were
continuous. When the Y-variable is discrete, we talk of a classification problem.
The simplest classification situation is that of classifying into two classes, namely class 0 and
class 1, for example female and male, based on some measurement, for example bodysize
(this may be interesting in a discotheque, where they want to admit approximately equal
numbers of guys and girls, and they don't want to employ a guard, but use automated
measurements of size to do he classification and count).
So to do the modeling a model assumption must be made. In what's called logistic regression
the model assumption is, that the ratio of the probabilities of belonging to one or the other
class is given by e x , where x is the measurement (it may be a vector) and is a coefficient
(or vector of coefficients). This ration is also called the odds of belonging to one class as
opposed to the other (the word is often used in horse-betting etc.). This means that the log of
the odds, also called the logit, for short, is a linear function of the x:
P Y =1 P Y =1 x
logit =lo g odds=lo g = x , equivalently = e . Since
P Y =0 P Y = 0
P Y =0 P Y =1 =1 it must follow, that:
x
e 1
P Y =1 | X = x = P Y =0 | X = x= =1 p x , .
x =: p x , and
1e 1 e x
1 Notice we use the word categories for known settings of a discrete X variable and the word classes for the
unknown settings of a discrete Y-variable, that are to be predicted, in the hope of reducing confusion.
page 43 (57)
Now given measurements xi and class memberships yi of a total of N individuals, i, =1, , N,
the log-likelihood of a parameter vector given independence of the classifications will
x x
be given by l = x i log 1e log 1e . To maximize this, take
i i
y i =1 yi =0
derivatives and set them to zero:
xi e x xi ex N i i
l = x i x x = x i y i p x i , = 0 !
y =1 1e y =0 1 e i=1
i
i
i
i
Since this is a nonlinear function, it is advisable to look for the solution of the equation by
using a Newton-Raphson iteration:
new = old H old 1 l old , where H old is the derivative, l , of l T
evaluated at old , i.e. the Hessian matrix (second derivative) of l , which in turn is given
N
T
by: H = x i x i p x i , 1 p x i , , because
i=1
xi T xi
xi e xi x ei 1e x x i xTi e x e x
i i i
e x 1 i
x
= xi 2
= x i x Ti x 2
= x i x Ti p xi , 1 p x i , .
1e i
1e 1e i
W=
old
p i 1 p i
0
old 1
0
p i 1 p i is a N*N diagonal matrix of weights. Setting
new T 1 T old
z = X W y p the iteration becomes: = X W X X W z , which can be
solved by using the lm(...) function in R.
Categorical x-data (and continuous y-data) are best analys either using ANOVA (analysis-of-
variance) or GLM (generalized linear modeling). Both techniques will be illustrated in his
section, using a new data set, that is within the R example pool.
page 44 (57)
003 1 3 900
004 1 4 1070
005 1 5 930
006 1 6 850
007 1 7 950
008 1 8 980
009 1 9 980
010 1 10 880
011 1 11 1000
012 1 12 980
013 1 13 930
014 1 14 650
015 1 15 760
016 1 16 810
017 1 17 1000
018 1 18 1000
019 1 19 960
020 1 20 960
021 2 1 960
022 2 2 940
023 2 3 960
024 2 4 940
025 2 5 880
.
.
.
2. Convert the columns Expt and Runs to factors in order to be able to use Analysis of
Variance Speed of Light Data
> mm$Expt <- factor(mm$Expt)
> mm$Run <- factor(mm$Run)
1000
3. Box-Plot:
900
The vertical lines (the "whiskers") extend to at most 1.5 times the box width (the interquartile
range) from either or both ends of the box. They must end at an observed value, thus
connecting all the values outside the box that are not more than 1.5 times the box width away
from the box.
page 45 (57)
Three times the box width marks the boundary between "mild" and "extreme" outliers. In a
typical boxplot, "mild" and "extreme" outliers are differentiated by open dots and stars
respectively.
There are variants of the boxplot. Which variant is used depends on the software.
Analysis of variance
>fm <- aov(Speed ~ Run + Expt, data=mm)
>summary(fm)
Df Sum Sq Mean Sq F value Pr(>F)
Run 19 113344 5965 1.1053 0.363209
Expt 4 94514 23629 4.3781 0.003071 **
Residuals 76 410166 5397
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
x3,10 x3,mea
n
xmean,1 xmean
0
SSRun is the sum of squared deviations of the xmean,j from xmean. In the above date this sum is
113344. Since there are 20 such numbers, there are 19 degrees of freedom, which are used to
calculate the mean of squared deviations, which is then 5965.
SSExpt is the sum of squared deviations of the xi,mean from xmean. In the above data this sum is
94514. Since there are 5 such numbers, there are 4 degrees of freedom, which are used to
compute the mean of squares, which is then 23629.
If there were no additional variation in the data, one should be able to calculate
xi,j = xmean + expt-correction + run-correction
= xmean + (xi,mean xmean) + (xmean,j xmean) = xi,mean + xmean,j xmean)
page 46 (57)
But there is additional variation! So SSResiduals is the sum of the squares of these deviations, 100
deviations in all, but not all independent, there are 76 residual degrees of freedom (DFresidual =
100 1 19 4 = 76), so MSresidual = 5397.
Time to calculate the F-statistic by comparing the variance induced by the two factors with
the random variance:
FExpt = MSExpt / MSResiduals = 4,3781, corresponding p-value = 0,307%
FRuns = MSRuns / MSResiduals = 1,1053, corresponding p-value = 36,32%
A quotient of 4,3781 (or above) is improbable if there are only random effects,
whereas a quotient of 1,1053 (or above) is highly probable.
So the influence of the factor experiment on the data is significant at 0,5% but not at 0,1%,
hence **. The factor Runs is not significant.
Call:
lm(formula = Speed ~ Run + Expt, data = mm)
Residuals:
Min 1Q Median 3Q Max
-206.60 -37.35 4.90 44.28 132.90
Coefficients:
Estimate Std. Error t- value
(Intercept) 9.506e+02 3.599e+01 26.413 2e-16 ***
Run2 -5.200e+01 4.646e+01 -1.119 0.266588
Run3 -2.800e+01 4.646e+01 -0.603 0.548545
Run4 6.000e+00 4.646e+01 0.129 0.897591
Run5 -7.600e+01 4.646e+01 -1.636 0.106032
Run6 -1.040e+02 4.646e+01 -2.238 0.028125 *
Run7 -1.000e+02 4.646e+01 -2.152 0.034551 *
Run8 -4.000e+01 4.646e+01 -0.861 0.391996
Run9 -1.000e+01 4.646e+01 -0.215 0.830167
Run10 -3.800e+01 4.646e+01 -0.818 0.415992
Run11 4.000e+00 4.646e+01 0.086 0.931621
Run12 -1.738e-13 4.646e+01 -3.74e-15 1.000000
Run13 -3.600e+01 4.646e+01 -0.775 0.440851
Run14 -9.400e+01 4.646e+01 -2.023 0.046576 *
Run15 -6.000e+01 4.646e+01 -1.291 0.200492
Run16 -6.600e+01 4.646e+01 -1.420 0.159552
Run17 -6.000e+00 4.646e+01 -0.129 0.897591
Run18 -3.800e+01 4.646e+01 -0.818 0.415992
Run19 -5.000e+01 4.646e+01 -1.076 0.285271
Run20 -4.400e+01 4.646e+01 -0.947 0.346641
Expt2 -5.300e+01 2.323e+01 -2.281 0.025325 *
page 47 (57)
Expt3 -6.400e+01 2.323e+01 -2.755 0.007343 **
Expt4 -8.850e+01 2.323e+01 -3.810 0.000281 ***
Expt5 -7.750e+01 2.323e+01 -3.336 0.001317 **
What happened to Run 1? Well, since all "corrections" must sum up to 0 (otherwise the grand
mean would not be the grand mean), Runs 1's correction must be minus the sum of all the
others.
measured:
Sepal.Length,
2
Sepal.Width,
1
Petal.Length,
4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
Petal.Width
Sepal.Length
The idea is to use these variables to decide, whether a flower is one of the three species:
setosa, virginca or versicolor.
page 48 (57)
In the plot the little dots are setosa, the triangles are the versicolor and the crosses (plusses)
are virginica.
attach(iris)
plot(Sepal.Length,Petal.Length,
7
pch=as.numeric(Species))
6
There are two different paradigms:
(1) classification by supervised learning
5
(2) clustering by unsupervised learning
Petal.Length
4
The picture last page corresponds to is
3
supervised learning. Different classes are
indicated by the symbols. The picture at right
2
is unsupervised learning. Here all symbols are
the same.
1
4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
The easiest global method is Linear Regression of an indicator matrix. Here the idea is to
code the class membership by 0 and 1. So in case of 3 classes, we'd have 3 y-variables, each
of which is 1 if the observation (in our case iris) is in the corresponding class. These three y-
variables are called the indicator matrix.
Sepal.Length Sepal.Width Petal.Length Petal.Width Species S(set) S(ver) S(vir)
1 5,1 3,5 1,4 0,2 setosa 1 0 0
2 4,9 3 1,4 0,2 setosa 1 0 0
3 4,7 3,2 1,3 0,2 setosa 1 0 0
4 4,6 3,1 1,5 0,2 setosa 1 0 0
51 7 3,2 4,7 1,4 versicolor 0 1 0
52 6,4 3,2 4,5 1,5 versicolor 0 1 0
53 6,9 3,1 4,9 1,5 versicolor 0 1 0
54 5,5 2,3 4 1,3 versicolor 0 1 0
101 6,3 3,3 6 2,5 virginica 0 0 1
102 5,8 2,7 5,1 1,9 virginica 0 0 1
103 7,1 3 5,9 2,1 virginica 0 0 1
104 6,3 2,9 5,6 1,8 virginica 0 0 1
Now make models for these y-variables. Once the models are available from fitting to a
training set of data, new observations can be classified by calculating predictions for all of the
indicator y-variables, and then by classifying the new observation into that class, i, which has
the highest yi,pred value.
> lmset<-lm(S.set. ~ Sepal.Length + Sepal.Width + Petal.Length +
Petal.Width, data=iris)
> lmver<-lm(S.ver. ~ Pet
> lmvir<-lm(S.vir. ~ Pet
page 49 (57)
Using the above models classification is done by rate total 0,85
setosa 1
chosing the iris type that has the highest pred-value. The versicolor 0,68
correct-classification rates are at right, some of the virginica 0,86
predictions are below:
It is not too hard to prove that if y1 + y2 + y3 = 1, then after fitting a linear model, the same is
true for the predictions, i.e. y1,pred + y2,pred + y3,pred = 1. (This can be verified in the column
Summe above and be proved by using yi,pred = X(XTX)-1XTyi and that this is a linear operation).
> plot(lmset$f,S.set.)
> plot(lmver$f,S.ver.)
1 .0
1 .0
0 .8
0 .8
0 .6
0 .6
S .v e r.
S .s e t.
0 .4
0 .4
0 .2
0 .2
0 .0
0 .0
0.0 0.5 1.0 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8
lmiris$f lmiris$f
One problem with the Linear Regression of an indicator matrix is that observations that are a
long way away from the separating (or decision) boundary, yi,pred = (vertical line in the plots
above), have a strong leverage (Hebelwirkung) on the model and hence the position of the
separating (or decision) line.
A second problem with Linear Regression of an indicator matrix is what's called masking.
This is in fact what seems to have happened in the plot above right, where versicolor was
modelled and predictions would be difficult, because setosa are on one side, virginica on the
other. The correct-classification rate for versicolor in any case is the lowest of the three. It is
only 0,68 compared to 0,86 for virginica and an impeccable 1 for setosa.
The easiest local method is k-nearest-neighbours. For each vector x in the input space define
a neighbourhood Nk(x), defined by the k closest points in the training set (observation (row)
page 50 (57)
vectors in the X-matrix). Then calculate ypred(x) = 1/k x(i) in N(k,x) yi. Depending on the number k
used, nearest neighbours classification can yield very complex and very particular separation
lines. In particular, if k=1, then all observations in the training set will be classified correctly.
data(iris3)
train <- rbind(iris3[1:25,,1], iris3[1:25,,2], iris3[1:25,,3])
test <- rbind(iris3[26:50,,1], iris3[26:50,,2], iris3[26:50,,3])
cl <- factor(c(rep("s",25), rep("c",25), rep("v",25)))
knn(train, test, cl, k = 4, prob=TRUE)
attributes(.Last.value)
Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth edition.
Springer.
Now nearest neighbours and linear regression on indicators are extreme methods. Good in
between methods exist:
LDA (linear discriminant analysis) and related to this QDA (quadratic discriminant analysis)
Also there is Logistic regression, which was treated in the section on Maximum Likelihood.
More modern methods are so called Kernel methods, that are beyond the scope of this lecture
at this stage. One of the methods related to this class of methods is called support vector
machines, SVM)
Let x be a p-vector of measurements (in the iris example p = 4, and the 4 measurements are
Sepal.Length, Sepal.Width, Petal.Length, Petal.Width),
let there be k classes (in the iris example k = 3 and the classes are setosa, versicolor and
virginica)
let pk be the prior probability (i.e. with no information from any measurement) of being in
class k, and
let fk(x), the probability function within class k, be a p-dimensional Gauss-function with mean
k and covariance matrix . need not be a diagonal matrix, but it is assumed that is the
same for all classes (if k is allowed to depend on the class k, then the same ideas can be used
and lead to QDA, quadratic discriminant analysis. This is advantageous from the point of
view of flexibility, but involves estimating many more parameters).
page 51 (57)
k(x) = xT-1k kT-1k +log(pk)
For a measurement x, the class is assigned, that has maximal k(x). This can be deduced from
the formula for the p-dimensional Gauss-function with covariance . It means that the
decision boundaries between two classes k and l are along lines of the form k(x) = l(x).
These are linear.
Of course in application the true parameters pk, k, and are not known. so they are estimated
from the data:
pk: Nk/N, where Nk is the number of class k observations, N the number of all observations,
k: 1/Nk class of xi is k xi,
: 1/(N-K) k=1 bis Kclass of xi is k (xi-k) (xi-k)T.
CART (classification and regression trees), respectively RPART (recursive partitioning and
regression trees)
> plot(rpart(Species ~ .,data=iris),margin=0.1,branch=0.5)
> text(rpart(Species ~ .,data=iris))
Petal.Length< 2.45
|
2.5
2.0
Petal.Width< 1.75
Petal.Width
1.5
setosa
1.0
0.5
versicolor virginica
1 2 3 4 5 6 7
Petal.Length
page 52 (57)
Cox-PH for reliability and life-time analysis
0,3
models (at right) or Weibull-models. 0,2
0,1
So h(t) = - S'(t)/S(t).
1
0,9
probabiblity of failure,
0,8
0,7
F (t ) 1 e t
F(t) = 1 - S(t) ,
In Exponential Decay h(t) = ist
0,6
0,5
h(t) = / (t/)-1.
0,2
0,1
The interesting thing about this model is that h0(t), so-called base line hazard and the
coefficients b can be estimated independently of each other! So it is usual to estimate
coefficients first and base line hazard after.
His idea was given a set of data consisting of subjects Sj and failure times tj to maximize
the probability P(S1=i1, ..., Sn=in), where (ij ) is the index sequence of the time-ordered failure
events as they actually occurred, amongst the probabilities of all possible permutations by
varying b.
page 53 (57)
P(Sj=i | given all data before j) = hi(tj) / k hk(tj),
where the sum goes over all subjects that are still at risk, meaning, that they have not had a
failure or have not been taken out of the test. Since numerator and denominator both contain
base line hazard h0(tj) as a factor, it can just be cancelled out (this is the trick that makes
coefficient estimation independent of the base line hazard) and leaves one with the quotient
P(Sj=i | all data before j) = exp( bxi) / k exp( bxk),
where again the sum goes over all subjects that are still at risk at time tj.
Using the chain rule for conditional probabilities, P(A and B) = P(A | B)*P(B) applied at all
failure time points for all n subjects one has:
P(S1=i1, ..., Sn=in) = j=1,n [exp( bxj) / k at risk at time j exp( bxk)] (Cox Partial
Likelihood). This partial likelihood can be maximized by varying the coefficients b.
As an example, we take 4 subjects, S1, S2, S3, S4, that fail at time points 2, 3, 4 and 5.
A covariate x may take values 1 and 1, for simplicity we assume 1 occurs twice and +1
occurs twice.
--++
20
-+-+
-++-
+--+
+-+-
15
++--
10
0
page 54 (57)
-2,5 -2 -1,5 -1 -0,5 0 0,5 1 1,5 2 2,5
The best coeff's in each case are - for ++; -0,47 for + + ; -0,2 for + +
for ++ ; 0,47 for + + ; 0,2 for + + .
The infinity cases are a degenerate case, which occurs, when the correlation between early
failure and high covariate values is very high. This is usually an indication that hazard in the
presence of the covariates is not proportional to a baseline hazard, i.e. CoxpH should not be
used.
For interpretation purposes it is important to notice, that coefficients depend only on the
ordering of the failures and the observed values of the covariates, not at all on the time points
of failure themselves! These will determine the base line hazard function, which can be
obtained by setting all x's to zero (i.e. centre) and fitting to the time points (for example a
Weibull-fit or some other non-parametric method offered in R).
In R Cox proportional hazard regression can be found in the survival-package. There is a so-
called penalized version which is better for collinear x-data in the penalize-package. But we
shall learn about this later (third semester on MVDA multivariate data analysis).
Input to CoxpH is a survival-object, that should be cast from a data-frame into a survival-
object using the function Surv( ). [Careful: capital S at the beginning.]
> coxtest<-read.delim("clipboard",dec=",")
> coxtest
day event x
1 2 1 -1
2 4 1 -1
3 3 1 1
4 5 1 1
> cox.mod<-coxph(Surv(day,event) ~ x, data = coxtest)
> summary(cox.mod)
Call:
coxph(formula = Surv(day, event) ~ x, data = coxtest)
n= 4
> basehaz(cox.mod)
hazard time
1 0.2246892 2
4.0
2 0.5755534 3
time
3 1.0249318 4
3.0
4 2.6254170 5
> plot(basehaz(cox.mod))
2.0
> plot(survfit(cox.mod))
0.5 1.0 1.5 2.0 2.5
hazard
page 55 (57)
The hazard function shown is baseline hazard, note that it is a discrete distribution, that
when integrated gives a piecewise continuous function.
1.0
for x = 0) with 95% confidence-intervals.
0.8
0.6
To get the survival function for different values of the
0.4
covariates, construct a new data frame and use the command:
0.2
0.0
> plot(survfit(cox.mod,newdata=data.frame(x=c(-
1,1)))) 0 1 2 3 4 5
1.0
0.8
0.6
0.4
0.2
0.0
0 1 2 3 4 5
page 56 (57)
Literature:
[DH] Davison, A.C.; Hinkley, D.V.: Bootstrap Methods and their Applications. Cambridge
University Press (1997)
page 57 (57)