Matrix Analysis in Situ

CAAM 335, Fall 2013

Steven J Cox

Preface
Our intent is to place the reader in the situation of the practicing engineer, scientist or mathematician needing matrix tools to model and analyze multivariable systems and/or synthesize
multivariable designs.
Beginning with modeling of structures in static equilibrium we focus on the linear nature of the
relationship between relevant state variables and express these relationships as simple matrixvector
products. For example, the voltage drops across the resistors in a network are linear combinations
of the potentials at each end of each resistor. Similarly, the current through each resistor is assumed to be a linear function of the voltage drop across it. And, nally, at equilibrium, a linear
combination (in minus out) of the currents must vanish at every node in the network. In short, the
vector of currents is a linear transformation of the vector of voltage drops which is itself a linear
transformation of the vector of potentials. A linear transformation of n numbers into m numbers is
accomplished by multiplying the vector of n numbers by an m-by-n matrix. Once we have learned
to spot the ubiquitous matrixvector product we move on to the analysis of the resulting linear
systems of equations. We accomplish this by stretching your knowledge of threedimensional space.
That is, we ask what does it mean that the mbyn matrix B transforms Rn (real ndimensional
space) into Rm ? We shall visualize this transformation by splitting both Rn and Rm each into two
smaller spaces between which the given B behaves in very manageable ways. An understanding
of this splitting of the ambient spaces into the so called four fundamental subspaces of B permits
one to answer virtually every question that may arise in not just the study of structures in static
equilibrium but the study of over and underdetermined systems as well.
In the second half of the notes we introduce the Laplace transform and argue that matrix methods
are equally eective in the modeling and analysis of linear dynamical systems, such as x (t) = Bx(t).
Although our modeling methodology adapts easily to dynamical problems we shall see, with respect
to analysis, the Laplace Transform directs our attention to the resolvent (zI B)1 for complex z. In
this case we nd that rather than splitting the ambient spaces we shall be better served by splitting
B itself. The process is analogous to decomposing a complicated signal, e.g., light, into a weighted
sum of simple components, e.g., colors. For we shall see that (most) matrices may be written as
weighted sums of matrices of very special type. The weights are the eigenvalues of the matrix
while the component matrices are projections composed from simple selfproducts of eigenvectors.
We follow Kato in our eigendecomposition of B is via study of the resolvent. As z is a complex
variable we pause to develop the essentials of Complex Calculus before using these tools to establish
the partial fraction expansion of the resolvent. From this expansion we then derive the spectral
respresentation and subsequent Jordan Form before establishing the inverse Laplace transform and
the subsequent representation of the matrix exponential as the inverse Laplace Transform of the
resolvent. We develop, and illustrate the ecacy of, each of these concepts in the context of time
varying stimulus of both electrical and mechanical systems.
Steve Cox

Contents
1 Vectors, Matrices, and Proofs

1.1 Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Matrix Methods for Electrical Networks

10

2.1 Neurons and the Strang Quartet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Resistor Nets with Current Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Resistor Nets with Current Sources and Batteries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
2.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3 Matrix Methods for Mechanical Networks

17

3.1 Elastic Fibers and the Strang Quartet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2 Gaussian Elimination and LU Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 A Small Planar Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 A Large Planar Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5 Positive Denite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4 The Column and Null Spaces

33

4.1 The Column Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.2 The Null Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.3 A Blend of Theory and Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.4 The Structure of a Nilpotent Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5 The Fundamental Theorem of Linear Algebra

45

5.1 The Row Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.2 The Left Null Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47
5.3 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.4 Vector Spaces and Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

ii

6 Least Squares

53

6.1 The Normal Equations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

6.2 Probabilistic and Statistical Interpretations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.3 Applying Least Squares to the Biaxial Test Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.4 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.5 The QR Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.6 Detecting Integer Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.7 Autoregressive Models and Levinsons Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7 Matrix Methods for Metabolic Networks

73

7.1 Flux Balance and Optimal Yield. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73

7.2 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.3 The Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
7.5 The Geometric Point of View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

8 Matrix Methods for Dynamical Systems

79

8.1 Neurons and the Dynamic Strang Quartet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

8.2 The Laplace Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
8.3 The BackwardEuler Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.4 Dynamics of Mechanical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
8.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

9 Complex Numbers, Functions and Derivatives

90

9.1 Complex Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

9.2 Complex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.3 Fourier Series and Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.4 Complex Dierentiation and the First Residue Theorem . . . . . . . . . . . . . . . . . . . . . . . . 99
9.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

10 Complex Integration

105

10.1 Cauchys Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

10.2 The Second Residue Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
10.3 The Inverse Laplace Transform and Return to Dynamics. . . . . . . . . . . . . . . . . . . . . .110
10.4 The Inverse Fourier Transform and the Causal Wiener Filter . . . . . . . . . . . . . . . . . 112
iii

10.5 Further Applications of the Second Residue Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 114

10.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

11 The Eigenvalue Problem

119

11.1 The Resolvent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

11.2 The Spectral Representation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
11.3 Diagonalization of a Semisimple Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
11.4 The Schur Form and the QR Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
11.5 The Jordan Canonical Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
11.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

12 The Symmetric Eigenvalue Problem

138

12.1 The Spectral Representation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

12.2 Orthonormal Diagonalization of a Symmetric Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 140
12.3 Perturbation Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
12.4 Rayleighs Principle and the Power Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
12.5 Return to Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

13 The Singular Value Decomposition

155

13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

13.2 The SVD in Image Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
13.3 Low Rank Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
13.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

14 Reduced Dynamical Systems

162

14.1 The Full Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

14.2 The Reduced Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
14.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

iv

1. Vectors, Matrices, and Proofs

This book is meant to help the reader reason about matrices. To reason is to understand the ideas
that allow intelligent application to hard problems in science and engineering. As the breadth of
application knows no bound the demanding reader will not be content with a compendium of textbook examples but rather needs knowledge of the foundations and analytical and computational
tools for tying to and building onto this foundation.
We begin with numbers and presume that the reader is familiar with the elementary algebra and
analysis of real numbers. We build vectors and matrices from numbers and then we study numerical
functions of these multi-number objects.
1.1. Vectors
A vector is a column of real numbers, and is written, e.g.,

2

.
4
x=
1

(1.1)

The vector has 3 elements and so lies the class of all 3element vectors, denoted, R3 . We denote
is an element of by the symbol . So, e.g., x R3 . We denote the rst element of x by x1 , its
second element by x2 and so on. We will typically use the postive integer n to denote the ambient
dimension of our problem, and so will be working in Rn . The sum of two vectors, x and y, in Rn is
dened elementwise by
z = x + y,

where zj = xj + yj ,

j = 1, . . . , n,

as is the scalar multiplication of a R by the vector x Rn ,

z = ax,
For example,

where zj = axj ,

     
2
1
3
+
=
5
3
2

j = 1, . . . , n.

   
4
24
and 6
=
.
2
12

The most common product of two vectors, x and y, in Rn is the inner product,

y1
n

y2 


T

xj yj .
x y x1 x2 xn .. =
.
yn

(1.2)

j=1

As xj yj = yj xj for each j it follows that xT y = y T x. For example,


8
10 1 3 2 = 10 8 + 1 2 + 3 (4) = 70.
4

(1.3)

So, the inner product of two vectors is a number. The superscript T on the x on the far left of
(1.2) stands for transpose and, when applied to a column yields a row. Columns are vertical and
1

rows are horizontal and so we see, in (1.2), that xT is x laid on its side. We follow Euclid and
measure the magnitude, or more commonly the norm, of a vector by the square root of the sum of
the squares of its elements. In symbols,



 n 2
T
x x x = 
xj .
(1.4)
j=1

For example, the norm of the vector in (1.1) is 21. As (1.4) is a direct generalization of the
Euclidean distance of high school planar geometry we may expect that Rn has much the same
look. To be precise, let us consider the siutation of Figure 1.1.
x

2.5

1.5

y
1

0.5

0.5

1.5

2.5

3.5

Figure 1.1. A guide to interpreting the inner product.

We have x and y in R2 and

   
1
x1
=
x=
3
x2

   
y
4
and y = 1 =
1
y2

and we recognize that both x and y dene right triangles with hypotenuses x and y respectively.
We have denoted by the angle that x makes with y. If x and y denotes the angles that x and y
respectively make with the positive horizontal axis then = x y and the Pythagorean Theorem
permits us to note that
x1 = x cos(x ),

x2 = x sin(x ),

and y1 = y cos(y ),

y2 = y sin(y ),

and these in turn permit us to express the inner product of x and y as

xT y = x1 y1 + x2 y2
= xy(cos(x ) cos(y ) + sin(x ) sin(y ))
= xy cos(x y )
= xy cos().

(1.5)

We interpret this by saying that the inner product of two vectors is proportional to the cosine of the
angle between them. Now given two vectors in say R8 we dont panic, rather we orient ourselves by
observing that they together lie in a particular plane and that this plane, and the angle they make
with one another is in no way dierent from the situation illustrated in Figure 1.1. And for this
reason we say that x and y are perpendicular, or orthogonal, to one another whenever xT y = 0.
2

In addition to the geometric interpretation of the inner product it is often important to be able
to estimate it in terms of the products of the norms. Here is an argument that works for x and y
in Rn . Once we know where to start, we simply expand via FOIL
(y T y)x (xT y)y2 = ((y T y)x (xT y)y)T ((y T y)x (xT y)y)
= y4x2 2y2(xT y)2 + (xT y)2y2

(1.6)

= y (x y (x y) )

2

and then note that as the initial expression is nonnegative, the nal expression requires (after taking
square roots) that
|xT y| xy.
(1.7)
This is known as the CauchySchwarz inequality.
1.2. Matrices
As a vector is simply a column of numbers, a matrix is simply a row of columns, or a column of
rows. This necessarily requires two numbers, the row and column indicies, to specify each matrix
element. For example


 
a11 a12 a13
5 0 1
(1.8)
A=
=
2 3 4
a21 a22 a23
is a 2-by-3 matrix. The rst dimension is the number of rows and the second is the number of
columns and this ordering is also used to address individual elements. For example, the element in
row 1 column 3 is a13 = 1.
The addition of two matrices (of the same size) and the multiplication of a matrix by a scalar
proceed exactly as in the vector case. In particular,

 
 

5 0 1
2 4 6
7 4 7
+
=
,
(A + B)ij = aij + bij , e.g.,
2 3 4
1 3 4
3 0 8
and
(cA)ij = caij ,


 

5 0 1
15 0 3
e.g., 3
=
.
2 3 4
6 9 12

The product of two commensurate matrices proceeds through a long sequence of vectors products.
In particular if C = AB then the ij element of C is the product of the ith row of A and the jth
column of B. Hence, for two A and B to be commensurate it follows that each row of A must have
the same number of elements as each column of B. In other words, the number of columns of A
must match the number of rows of B. Hence, if A is m-by-n and B is n-by-p then the ij element
of their product C = AB is
n

aik bkj .
(1.9)
cij =
k=1

Here i is pinned to A


 2
5 0 1
6
2 3 4
3

and j is pinned to B and so the product C will be m-by-p. For example,


 

4
5

2
+
0

6
+
1

(3)
5

4
+
0

1
+
1

4
7
24
1 =
=
.
2 2 + 3 6 + 4 (3) 2 4 + 3 1 + 4 (4)
10 5
4
3

In this case, the product BA is not even dened. If A is m-by-n and B is n-by-m then both AB
and BA are dened, but unless m = n they are of distinct dimensions and so not comparable. If
m = n so A and B are square then we may ask if AB = BA ? and learn that the answer is typically
no. For example,

 

 
 


18 12
5 0
2 4
10 20
2 4
5 0
.
(1.10)
=
=
=
32 3
2 3
6 1
22 11
6 1
2 3
We will often abbreviate the awkward phrase A is m-by-n with the declaration A Rmn . The
matrix algebra of multiplication, though tedious, is easy enough to follow. It stemmed from a rowcentric point of view. It will help to consider the columns. We use Matlab notation to denote the
jth column of A by A(:, j). If A Rmn and x Rn we recognize the product

x1
x2

(1.11)
Ax = [A(:, 1) A(:, 2) A(:, n)]
... = x1 A(:, 1) + x2 A(:, 2) + + xn A(:, n),
xn
as a weighted sum of the columns of A. For example
   
 
 

13
3
2
2
2 3
.
=
+3
=2
14
4
1
3
1 4

(1.12)

We illustrate this in Figure 1.2.

2A(:,1)+3A(:,2)

14

3A(:,2)

12

10

2A(:,1)

10

12

14

Figure 1.2. An illustration of the matrix vector multiplication conducted in (1.12). Both A(:, 1)
and A(:, 2) are plotted heavy for emphasis. We see that their multiples, by 2 and 3, simply extend
them, while their weighted sum simply completes the natural parallelogram.
A common goal of matrix analysis is to describe m-by-n matrices by many fewer than mn
numbers. The simplest such descriptor is simply the sum of the matrices diagonal elements. We
call this the trace and abbreviate it by
tr(A)

aii .

(1.13)

i=1

Looking for matrices to trace you scan (1.10) and note that 10 + 11 = 18 + 3 and you ask, knowing
that AB = BA, whether
tr(AB) = tr(BA)
(1.14)
4

might possibly be true in general. For arbitrary A and B in Rnn we therefore construct tr(AB)
(AB)ii =
and tr(BA)
(BA)ii =

aik bki

so tr(AB) =

n 
n


k=1

i=1 k=1

n 
n


bik aki

so tr(BA) =

aik bki ,

bik aki .

i=1 k=1

k=1

These sums indeed coincide, for both are simply the sum of the product of each element of A and
the reected (interchange i and k) element of B.
In general, if A is m-by-n then the matrix that results on exchanging its rows for its columns is
called the transpose of A, denoted AT . It follows that AT is n-by-m and
(AT )ij = aji .
For example,

T

5 2
5 0 1
= 0 3 .
2 3 4
1 4

We will have frequent need to transpose a product, so let us contrast

((AB)T )ij =

ajk bki

k=1

with
T

(B A )ij =

ajk bki

(1.15)

k=1

and so conclude that

(AB)T = B T AT ,

(1.16)

i.e., that the transpose of a product is the product of the transposes in reverse order.
Regarding the norm of a matrix it seems natural, on recalling our denition of the norm of
a vector, to simply dene it as the square root of the sum of the squares of each element. This
denition, where A Rmn is viewed as a collection of vectors, is associated with the name Frobenius
and hence the subscript in the denition of the Frobenius norm of A,
 m n
1/2

a2ij
.
(1.17)
AF
i=1 j=1

As scientic progress and mathematical insight most often come from seeing things from multiple
angles we pause to note (1.17) may be seen as the trace of a product. In particular, with B = AT
and j = i in the general formula (1.15) we arrive immediately at
(AAT )ii =

n

k=1

a2ik .

As the sum over i is precisely the trace of AAT we have established the equivalent denition
AF = (tr(AAT ))1/2 .

For example, the Frobenius norm of the A in (1.8) is 55.

(1.18)

1.3. Proofs
Regarding the proofs in the text, and more importantly in the exercises and exams, many will
be of the type that brought us (1.14) and (1.16). These are what one might call conrmations.
They require a clear head and may require a bit of rearrangement but as they follow directly
from denitions they do not require magic, clairvoyance or even ingenuity. As further examples of
conrmations let us prove (conrm) that
tr(A) = tr(AT ).

(1.19)

It would be acceptable to say that As AT is the reection of A across its diagonal both A and
AT agree on the diagonal. As the trace of matrix is simply the sum of its diagonal terms we have
conrmed (1.19). It would also be acceptable to proceed in symbols and say from (AT )ii = aii for
each i it follows that
n


(AT )ii =
aii = tr(A).
tr(AT ) =
i=1

i=1

It would not be acceptable to conrm (1.19) on a particular numerical matrix, nor even on a class
of matrices of a particular size.
As a second example, lets conrm that
if x = 0 then x = 0.

(1.20)

It would be acceptable to say that As the sum of the squares of each element of x is zero then in
fact each element of x must vanish. Or, in symbols, as
n


x2i = 0

i=1

we conclude that each xi = 0.

Our third example is a slight variation on the second.
if x Rn

and xT y = 0 for all y Rn

then x = 0.

(1.21)

This says that the only vector that is orthogonal to every vector in the space is the zero vector.
The most straightforward proof is probably the one that reduces this to the previous Proposition,
(1.20). Namely, since xT y = 0 for each y we can simply use y = x and discern that xT x = 0 and
conclude from (1.20) that indeed x = 0. As this section is meant to be an introduction to proving
let us apply instead a dierent strategy, one that replaces a proposition with its equivalent contra
positive. More precisely, if your proposition reads if c then d then its contrapositive reads if not
d then not c. Do you see that a proposition is true if and only its contrapositive is true? Why
bother? Sometimes the contrapositive is easier to prove, sometimes it throws new light on the
original proposition, and it always expands our understanding of the landscape. So let us construct
6

the contrapositive of (1.21). As clause d is simply x = 0, not d is simply x = 0. Clause c is a bit

more dicult, for it includes the clause for all, that is often called the universal quantier and
abbreviated by . So clause c states xT y = 0 y. The negation of some thing happens for every
y is that there exists a y for which that thing does not happen. This there exists is called the
existential quantier and is often abbreviated . Hence, the contrapositive of (1.21) is
if x Rn

and x = 0 then y Rn

such that xT y = 0.

(1.22)

It is a matter of taste, guided by experience, that causes one to favor (or not) the contrapositive
over the original. At rst sight the student new to proofs and unsure of where to start may feel
that the two are equally opaque. Mathematics however is that eld that is, on rst sight, opaque to
everyone, but that on second (or third) thought begins to clarify, suggest pathways, and oer insight
and rewards. The key for the beginner is not to despair but rather to generate as many starting
paths as possible, in the hope that one of them will indeed lead to a fruitful second step, and on to a
deeper understanding of what you are attempting to prove. So, investigating the contrapositive ts
into our bigger strategies of generating multiple starting points. Back to the problem at hand (note
that building the contrapositive is, if even a dead-end, a great piece of guiltfree procrastination)
Id like to point out two avenues suggested by (1.22). The rst is the old avenue take y = x
for then x = 0 surely implies that xT x = 0. The second I feel is more concrete, more pedestrian,
less clever and therefore hopefully contradicts the belief that one either gets the proof or not. The
concreteness I speak of is generated by the for it says we only have to nd one and I typically
nd that easier to do than nding many or all. To be precise, if x = 0 then a particular element
xi = 0. From here we can custom build a y, namely choose y to be 0 at each element except for the
ith in which you set yi = 1. Now xT y = xi which, by not c, is presumed nonzero.
As a nal example lets prove that
if A Rnn

and Ax = 0 x Rn

then A = 0.

(1.23)

In fact, lets oer three proofs.

The rst is a row proof. We denote row j of A by A(j, :) and notes that Ax = 0 implies that the
inner product A(j, :)x = 0 for every x. By our proof of (1.21) it follows that the jth row vanishes,
i.e., A(j, :) = 0. As this holds for each j it follows that the entire matrix is 0.
Our second is a column proof. We intrepret Ax = 0, x, in light of (1.11), to say that every
weighted sum of the columns of A must vanish. So lets get concrete and choose an x that is zero in
every element except the jth, for which we set xj = 1. Now (1.11) and the if clause in (1.23) reveal
that A(:, j) = 0, i.e., the jth column vanishes. As j was arbitrary it follows that every column
vanishes ans so the entire matrix is zero.
Our third proof will address the contrapositive,
if A = 0 Rnn

then x Rn

such that Ax = 0.

(1.24)

We now move concretely and infer from A = 0 that for some particular i and j that aij = 0. We
then construct (yet again) an x of zeros except we set xj = 1. It follows (from either the row or
column interpretation of Ax) that the ith element of Ax is aij . As this is not zero we have proven
that Ax = 0.
We next move on to a class of propositions that involve innity in a substantial way. If there are
in fact an innite number of claims we may use the Principle of Mathematical Induction, if rather
it is a claim about innitie sets then we may use the method of reciprocal inclusion.
7

The Principle of Mathematical Induction states that the truth of the innite sequence of
statements {P (n) : n = 1, 2, . . .} follows from establishing that
(PMI1) P (1) is true.
(PMI2) if P (n) is true then P (n + 1) is true, for arbitrary n.
For example, let us prove by induction that

n 

1 1
1 n
=
0 1
0 1

n = 1, 2, . . . .

(1.25)

We rst check the base case, here (1.25) holds by inspection when n = 1. We now suppose it holds
for some n then deduce its validity for n + 1. Namley

n 
 


n+1 

1 n+1
1 1
1 n
1 1
1 1
1 1
.
=
=
=
0
1
0 1
0 1
0 1
0 1
0 1
Regarding innite sets, the Principle of Reciprocal Inclusion states that two sets coincide of
each is a subset of the other. For example, given an x Rn lets consider the outer product matrix
xxT Rnn and let us prove that the two sets
N1 {y : xT y = 0} and N2 {z : xxT z = 0}
coincide. If x = 0 both sides are simply Rn . So lets assume x = 0 and check the reciprocal
inclusions, N1 N2 and N2 N1 . The former here looks to be the easy direction. For if xT y = 0
then surely xxT y = 0. Next, if xxT z = 0 then xT xxT z = 0, i.e., x2 xT z = 0 which, as x = 0
implies that xT z = 0.
This complete our list of examples. Of course these are only representative and questions and
perhaps anxiety remain. For example, as youve seen me use previous propositions in the support
of proving new ones, you may fairly ask What can I presume known in presenting a proof of a
new proposition? My reply is that, regarding open book exercises, e.g., Exer. 3.2.4, you may use
any propositions established (by you or this text) prior to that exercise. It is a dierent matter
regarding closed book exams. For those I will remind you that you may only rely on propositions
that you have already been tested on.
1.4. Exercises


1. Consider the matrix


0 1
.
1 0

A=

(1.26)

Evaluate the product Ax for several choices of x. Sketch both x and Ax in the plane for several
carefully marked x and explain why A is called a rotation. Argue, on strictly geometric
grounds, why A5 = A.


2. Consider the matrix

A=


0 1
.
1 0

(1.27)

Evaluate the product Ax for several choices of x. Sketch both x and Ax in the plane for several
carefully marked x and explain why A is called a reection. Argue, on strictly geometric
grounds, why A3 = A.
8

3. We will make frequent use of the identity matrix, I Rnn , comprised of zeros o the diagonal
and ones on the diagonal. In symbols, Iij = 0 if i = j, while Iii = 1. Prove the two propositions,
if A Rnn then AI = IA = A.
4. Prove that matrix multiplication is associative, i.e., that (AB)C = A(BC).
5. Prove that if x and y lie in Rn and A Rnn then
xT Ay = y T AT x.
Hint: The left side is a number. Now argue as we did in achieving (1.16).
6. Prove that tr(A + B) = tr(A) + tr(B).
7. Construct a nonzero A R22 for which A2 = 0.
8. A matrix that equals its transpose is called symmetric. Suppose S = AT GA where A Rmn
and G Rmm . Prove that if G = GT then S = S T .
9. Establish the triangle inequality
x + y x + y

x, y Rn .

(1.28)

First draw this for two concrete planar x and y and disuss the aptness of the name. Then, for
the general case use FOIL to evaluate x + y2 , invoke the CauchySchwarz inequality, (1.7),
and nish with a square root.
10. The other natural vector product is the outer product. Note that if x Rn then the outer
product of x with itself, xxT , lies in Rnn . Please prove that xxT F = x2 .
11. Via experimentation with small n arrive (show your

n
1 1 0
1
0 1 1 = 0
0
0 0 1

work) at a formula for fn in

n fn
1 n
0 1

and prove, via induction, that your formula holds true for all n.

2. Matrix Methods for Electrical Networks

We wish to conrm, by example, the prefatory claim that matrix algebra is a useful means of
organizing (stating and solving) multivariable problems. In our rst such example we investigate
the response of a neuron to a constant local current stimulus. We derive the system of equilibrium
equations by placing Ohms and Kirchhos laws in matrix form.
2.1. Neurons and the Strang Quartet
A neuron is typically idealized as simply a cylinder of radius a and length  that conducts
electricity both along its length and across its lateral membrane. Though we shall, in subsequent
chapters, delve more deeply into the biophysics, here, in our rst outing, we stick to its purely
resistive properties. These are expressed via two quantities: i , the resistivity, in cm, of the
cytoplasm that lls the cell, and m , the resistivity in cm2 of the cells lateral membrane.

Ri
Ri

Rm
Rm

Ri
Rm

Figure 2.1. A 3 compartment model of a neuron.

Although current surely varies from point to point along the neuron it is hoped that these variations
are regular enough to be captured by a multicompartment model. By that we mean that we choose
a number N and divide the neuron into N segments each of length /N. Denoting a segments axial
resistance by
Ri =

i /N
a2

and membrane resistance by

Rm =

m
2a/N

we arrive at the lumped circuit model of Figure 2.1. For a neuron in culture we may assume a
constant extracellular potential, e.g., zero. We accomplish this by connecting and grounding the
extracellular nodes, see Figure 2.2.
10

Ri

Ri

i0

Ri

Rm

Rm

Rm

Figure 2.2. A rudimentary neuronal circuit model.

This gure also incorporates the exogenous disturbance, a current stimulus between ground and the
left end of the neuron. Our immediate goal is to compute the resulting currents through each resistor
and the potential at each of the nodes. Our longrange goal is to provide a modeling methodology
that can be used across the engineering and science disciplines. As an aid to computing the desired
quantities we give them names. With respect to Figure 2.3 we label the vector of potentials

y1

x1
y2

x2
y3
and vector of currents y =
x=
.
x3
y4
y
x
4

y6
We have also (arbitrarily) assigned directions to the currents as a graphical aid in the consistent
application of the basic circuit laws.

Ri

Ri

x1

Ri

x2
y1

x3
y3

i0
y2

x4
y5

Rm

Rm
y4

Rm
y6

Figure 2.3 The fully dressed circuit model.

We incorporate the circuit laws in a modeling methodology that takes the form of a Strang
Quartet,
(S1)
(S2)
(S3)
(S4)

Express the voltage drops via e = Ax.

Express Ohms Law via y = Ge.
Express Kirchhos Current Law via AT y = f .
Combine the above into AT GAx = f .

The A in (S1) is the nodeedge adjacency matrix it encodes the networks connectivity. The G
in (S2) is the diagonal matrix of edge conductances it encodes the physics of the network. The f
in (S3) is the vector of current sources it encodes the networks stimuli. The culminating AT GA
11

in (S4) is the symmetric matrix whose inverse, when applied to f , reveals the vector of potentials,
x. In order to make these ideas our own we must work many, many examples.
2.2. Resistor Nets with Current Sources
With respect to the circuit of Figure 2.3, in accordance with step (S1), we express the six
potentials dierences (always tail minus head)
e1
e2
e3
e4
e5
e6

= x1 x2
= x2
= x2 x3
= x3
= x3 x4
= x4

Such long, tedious lists cry out for matrix representation, to wit

1 1
0
0
0
0 1 0

0
0 1 1
e = Ax where A =

0 1 0
0
0
0 1 1
0
0
0 1
Step (S2), Ohms law, states that the current along an edge is equal to the potential drop across
the edge divided by the resistance of the edge. In our case,
yj = ej /Ri , j = 1, 3, 5 and yj = ej /Rm , j = 2, 4, 6
or, in matrix notation,
y = Ge
where

0
0
0
0
0
1/Ri
1/Rm
0
0
0
0
0

0
0
0
0
1/Ri
0
G=

0
0
1/Rm
0
0
0
0
0
0
0
0
1/Ri
0
0
0
0
0
1/Rm

Step (S3), Kirchhos Current Law, states that the sum of the currents into each node must be
zero. In our case
i0 y1 = 0
y1 y2 y3 = 0
y3 y4 y5 = 0
y5 y6 = 0
or, in matrix terms
By = f
12

where

1 0
0
0
0
0
1 1 1 0
0
0

B=
0
0
1 1 1 0
0
0
0
0
1 1

i0
0

and f =
0
0

Turning back the page we recognize in B the transpose of A. Calling it such, we recall our main
steps
e = Ax, y = Ge, and AT y = f.
On substitution of the rst two into the third we arrive, in accordance with (S4), at
AT GAx = f.

(2.1)

This is a linear system of four simultaneous equations for the 4 unknown potentials, x1 through x4 .
As you may know, the system (2.1) may have either 1, 0, or innitely many solutions, depending
on f and AT GA. We shall devote chapters 3 and 4 to a careful analysis of the previous sentence.
For now, we simply invoke the Matlab backslash command and arrive at the response depicted
in Figure 2.4.
12

11

10

x (mV)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

z (cm)

Figure 2.4. Results of a 16 compartment simulation. cab1.m.

Once the structure of the constituents in the fundamental system (2.1) is determined it is an
easy matter to implement it, as we have done in cab1.m, for an arbitrary number of compartments.
In Figure 2.4 we see that the stimulus at the neurons left end produces a depolarization there that
then attenuates with distance from the site of stimulation.
2.3. Resistor Nets with Current Sources and Batteries
We have seen in the previous section how a current source may produce a potential dierence
across a neurons membrane. We note that, even in the absence of electrical stimuli, there is always
a dierence in potential between the inside and outside of a living cell. In fact, this dierence is
one of the biologists denition of living. Life is maintained by the fact that the neurons interior
is rich (relative to the cells exterior) in potassium ions and poor in sodium and chloride ions.
These concentration dierences establish a resting potential dierence, Em , across the cells lateral
membrane. The modied circuit diagram is given in Figure 2.5.
13

Ri

Ri

i0

Ri

Rm

Rm

Em

Rm

Em

Em

Figure 2.5 Circuit model with batteries associated with the rest potential.
The convention is that the potential dierence across the battery is Em . As the bottom terminal
of each battery is grounded it follows that the potential at the top of each battery is Em . Revisiting
steps (S14) of the Strang Quartet we note that in (S1) the even numbered voltage drops are now
e2 = x2 Em ,

e4 = x3 Em

and e6 = x4 Em .

We accommodate such things by generalizing (S1) to

(S1) Express the voltage drops as e = b Ax where b is the vector that encodes the batteries.
No changes are necessary for (S2) and (S3). The nal step now reads,
(S4) Combine (S1), (S2) and (S3) to produce
AT GAx = AT Gb + f.

(2.2)

This is the general form for a resistor network driven by current sources and batteries.
Returning to Figure 2.5 we note that
b = Em [0 1 0 1 0 1]T

and AT Gb = (Em /Rm )[0 1 1 1]T .

To build and solve (2.2) requires only minor changes to our old code. The new program is called
cab2.m and results of its use are indicated in Figure 2.6.
58

59

60

x (mV)

61

62

63

64

65

66

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

z (cm)

Figure 2.6. Results of a 16 compartment simulation with batteries, Em = 70 mV . cab2.m

14

2.4. Exercises
1. In order to refresh your matrix-vector multiply skills please calculate, by hand, the product
AT GA in the 3 compartment case and write out the 4 equations in (2.1). The second equation
should read
(x1 + 2x2 x3 )/Ri + x2 /Rm = 0.
(2.3)
2. We began our discussion with the hope that a multicompartment model could indeed adequately capture the neurons true potential and current proles. In order to check this one
should run cab1.m with increasing values of N until one can no longer detect changes in the
computed potentials.
(a) Please run cab1.m with N = 8, 16, 32 and 64. Plot all of the potentials on the same (use
hold) graph, using dierent line types for each. (You may wish to alter cab1.m so that it
accepts N as an argument).
Let us now interpret this convergence. The main observation is that the dierence equation,
(2.3), approaches a dierential equation. We can see this by noting that
dz /N
acts as a spatial step size and that xk , the potential at (k 1)dz, is approximately the value
of the true potential at (k 1)dz. In a slight abuse of notation, we denote the latter
x((k 1)dz).
Applying these conventions to (2.3) and recalling the denitions of Ri and Rm we see (2.3)
become
a2 x(0) + 2x(dz) x(2dz) 2adz
+
x(dz) = 0,
i
dz
m
or, after multiplying through by m /(adz),
am x(0) + 2x(dz) x(2dz)
+ 2x(dz) = 0.
i
dz 2
We note that a similar equation holds at each node (save the ends) and that as N and
therefore dz 0 we arrive at
d2 x(z)
2i

x(z) = 0.
(2.4)
2
dz
am
(b) With 2i /(am ) show that

x(z) = sinh( z) + cosh( z)

(2.5)
satises (2.4) regardless of and .
We shall determine and by paying attention to the ends of the neuron. At the near end
we nd
a2 x(0) x(dz)
= i0 ,
i
dz
which, as dz 0 becomes
dx(0)
i i0
(2.6)
= 2.
dz
a
15

At the far end, we interpret the condition that no axial current may leave the last node to
mean
dx()
= 0.
(2.7)
dz
(c) Substitute (2.5) into (2.6) and (2.7) and solve for and and write out the nal x(z).
(d) Substitute into x the , a, i and m values used in cab1.m, plot the resulting function
(using, e.g., ezplot) and compare this to the plot achieved in part (a).
3. Alter cab2.m to inject current at a specied compartment.

16

3. Matrix Methods for Mechanical Networks

We derive the equations of mechanical equilbrium by developing and applying the matrix forms
of Hookes Law and Conservation of Force. We solve these equations, by hand, via Gaussian
Elimination. This concrete elementary scheme reveals the importance of pivots and leads us to
the GaussJordan method of matrix inversion, the LU method of matrix factorization, and nally
the important notion of matrix determinant. We illustrate each of these ideas on two concrete
mechanical networks.
3.1. Elastic Fibers and the Strang Quartet
We connect 3 masses (nodes) with four springs (bers) between two immobile walls, as in Figure 3.1, and apply forces at the masses and measure the associated displacement.
k

x1

x2

m1

m2
f

m3
f

x3

Figure 3.1. A ber chain.

We suppose that a horizontal force, fj , is applied to each mj , and produces a horizontal displacement xj , with the sign convention that rightward means positive. The bars at the ends of
the gure indicate rigid supports incapable of movement. The kj denote the respective spring stinesses. Regarding units, we measure fj in Newtons (N) and xj in meters (m) and so stiness,
kj , is measured in (N/m). In fact each stiness is a parameter composed of both material and
geometric quantities. In particular,
kj =

Yj aj
Lj

(3.1)

where Yj is the bers Youngs modulus (N/m2 ), aj is the bers cross-sectional area (m2 ) and Lj
is the bers (reference) length (m).
The analog of potential dierence is here elongation. If ej denotes the elongation of the jth
spring then naturally,
e1 = x1 ,
or, in matrix terms,

e2 = x2 x1 ,

e3 = x3 x2 ,

and e4 = x3 ,

1
0
0
1 1
0

e = Ax where A =
0 1 1 .
0
0 1

We note that ej is positive when the spring is stretched and negative when compressed. The analog
of Ohms Law is here Hookes Law: the restoring force in a spring is proportional to its elongation.
We call this constant of proportionality the stiness, kj , of the spring, and denote the restoring
17

force by yj . Hookes Law then reads, yj = kj ej , or, in matrix terms

k1 0 0 0
0 k2 0 0

y = Ke where K =
0 0 k3 0 .
0 0 0 k4
The analog of Kirchhos Current Law is here typically called force balance. More precisely,
equilibrium is synonymous with the fact that the net force acting on each mass must vanish. In
symbols,
y1 y2 f1 = 0, y2 y3 f2 = 0, and y3 y4 f3 = 0,
or, in matrix terms
By = f

f1

where f = f2
f3

1 1 0
0
and B = 0 1 1 0 .
0 0
1 1

As is the previous section we recognize in B the transpose of A. Gathering our three important
steps
e = Ax
y = Ke
(3.2)
T
A y=f
we arrive, via direct substitution, at an equation for x. Namely
AT y = f AT Ke = f AT KAx = f.

(3.3)

These four steps, (3.2)-(3.3), comprise the Strang Quartet for mechanical networks. Assembling
AT KA we arrive at the nal system

k2
0
x1
f1
k1 + k2

k2
k2 + k3
k3
x2 = f2 .
(3.4)
0
k3
k3 + k4
x3
f3
Although Matlab solves such systems with ease our aim here is to develop a deeper understanding
of Gaussian Elimination and so we proceed by hand. This aim is motivated by a number of
important considerations. First, not all linear systems have unique solutions. A careful look at
Gaussian Elimination will provide the general framework for not only classifying those systems
that possess unique solutions but also for providing detailed diagnoses of those systems that lack
solutions or possess too many.
3.2. Gaussian Elimination and LU Decomposition
In Gaussian Elimination one rst uses linear combinations of preceding rows to eliminate nonzeros
below the main diagonal and then solves the resulting upper triangular system via backsubstitution.
To rm up our understanding let us take up the case where each kj = 1 and so (3.4) takes the form
Sx = f , i.e.,

f1
2 1 0
x1
1 2 1 x2 = f2
(3.5)
x3
f3
0 1 2
18

We eliminate the (2, 1) (row 2, column 1) element by implementing

1
new row 2 = old row 2 + row 1,
2
bringing

(3.6)

f1
2 1 0
x1
0 3/2 1 x2 = f2 + f1 /2
x3
f3
0 1 2

We eliminate the current (3, 2) element by implementing

2
new row 3 = old row 3 + row 2,
3

(3.7)

bringing the uppertriangular system

or, more precisely,

Ux = g,

(3.8)

f1
2 1 0
x1

0 3/2 1 x2 =
f2 + f1 /2
x3
f3 + 2f2 /3 + f1 /3
0 0 4/3

(3.9)

One now simply reads o

x3 = (f1 + 2f2 + 3f3 )/4.
This in turn permits, via so-called back-substition, the solution of the second equation
x2 = 2(x3 + f2 + f1 /2)/3 = (f1 + 2f2 + f3 )/2,
and, in turn,
x1 = (x2 + f1 )/2 = (3f1 + 2f2 + f3 )/4.
One must say that Gaussian Elimination has succeeded here. For, regardless of the actual elements
of f we have produced an x for which AT KAx = f .
A more careful look at back substitution reveals that the key to nding x was the fact that no
diagonal element of U vanished. These quantities are so important that we pause to name them.
Denition 3.1. The diagonal elements of the upper triangular matrix, U, achieved via the application of Gaussian Elimination to S are called the pivots of S.
Pivots provide the most concrete setting by which to dene and study the determinant. In what
follows we dene the determinant, by at, for two special classes of matrices and then use pivots
to extend the denition to all square matrices. The special classes are triangular matrices, i.e,
matrices whose elements are all zero either above or below the diagonal, and so called elementary
perturbations of the identity, i.e., matrices achieved by exchanging two rows in the identity
matrix, e.g.,

1 0 0
(3.10)
I = 0 0 1 .
0 1 0
Such matrices appear naturally in Gaussian Elimination when one is compelled to swap rows to
achieve a nonzero pivot (see below for an example).
19

Denition 3.2. If A is square we denote the determinant of A by det(A).

(i) If A is triangular then det(A) is the product of its diagonal elements.
(ii) If A is an elementary perturbation of the identity then det(A) = 1.
(iii) The determinant of an arbitrary square matrix A is (1)m times the product of the pivots
of A, where m is the number of requisite row swaps.
If det(S) = 0 then backsubstitution succeeds for every load f and in this case we say that
S is invertible. In this case we denote its inverse by S 1 and note that the matrix analog of
21 2 = 2 21 = 1 suggests that
S 1 S = SS 1 = I
where I denotes the identity matrix. The value of S 1 stems from watching what happens when it
is applied to each side of Sx = f . Namely,
Sx = f S 1 Sx = S 1 f Ix = S 1 f x = S 1 f.
Hence, to solve Sx = f for x it suces to multiply f by the inverse of S. Let us now consider how
one goes about computing S 1 . In general this takes a little more than twice the work of Gaussian
Elimination, for we interpret
SS 1 = I
as n (the size of S) applications of Gaussian elimination, with f running through the n columns of
the identity matrix. The bundling of these n applications into one is known as the Gauss-Jordan
method. Let us demonstrate it on the S appearing in (3.5). We rst augment S with I.

2 1 0 | 1 0 0
1 2 1 | 0 1 0
0 1 2 | 0 0 1
We then eliminate down, being careful to address each of the

2 1 0 | 1
0
0 3/2 1 | 1/2 1
0 0 4/3 | 1/3 2/3

3 f vectors. This produces

0
0
1

Now, rather than simple backsubstitution we instead eliminate up. Eliminating rst the (2, 3)
element we nd

2 1 0 | 1
0
0
0 3/2 0 | 3/4 3/2 3/4
0 0 4/3 | 1/3 2/3 1
Now eliminating the (1, 2) element we achieve

2 0
0 | 3/2 1 1/2
0 3/2 0 | 3/4 3/2 3/4
0 0 4/3 | 1/3 2/3 1
In the nal step we scale each row in order that the matrix on the left takes on the form of the
identity. This requires that we multiply row 1 by 1/2, row 2 by 3/2 and row 3 by 3/4, with the
result

1 0 0 | 3/4 1/2 1/4

0 1 0 | 1/2 1 1/2 .
0 0 1 | 1/4 1/2 3/4
20

Now in this transformation of S into I we have, ipso facto, transformed I to S 1 , i.e., the matrix
that appears on the right upon applying the method of GaussJordan is the inverse of the matrix
that began on the left. In this case,

3/4 1/2 1/4

S 1 = 1/2 1 1/2 .
1/4 1/2 3/4
One should check that S 1 f indeed coincides with the x computed above.
Some matrices can be inverted by inspection. An important class of such matrices is in fact
latent in the process of Gaussian Elimination itself. To begin, we build the elimination matrix that
enacts the elementary row operation spelled out in (3.6),

1 0 0
E1 = 1/2 1 0
0 0 1
Do you see that this matrix (when applied from the left to S) leaves rows 1 and 3 unsullied but
adds half of row one to two? This ought to be undone by simply subtracting half of row 1 from
row two, i.e., by application of

1
0 0
E11 = 1/2 1 0
0
0 1
Please conrm that E11 E1 is indeed I.

1 0
E2 = 0 1
0 2/3

Similarly, the matrix analogs of (3.7) and its undoing are

0
1
0
0
0 and E21 = 0
1
0
1
0 2/3 1

Again, please conrm that E2 E21 = I. Now we may express the reduction of S to U (recall (3.8))
as
(3.11)
E2 E1 S = U
and the subsequent reconstitution by
S = LU,

where L = E11 E21

1
0
0
1
0
= 1/2
0
2/3 1

One speaks of this representation as the LU decomposition of S. We have just observed that the
inverse of a product is the product of the inverses in reverse order. Do you agree that S 1 = U 1 L1 ?
LU decomposition is the preferred method of solution for the large linear systems that occur in
practice. The decomposition is implemented in Matlab as
[L U] = lu(S);
and in fact lies at the heart of Matlab s blackslash command. To diagram its use, we write Sx = f
as LUx = f and recognize that the latter is nothing more than a pair of triangular problems:
Lc = f

and Ux = c,
21

that may be solved by forward and backward substitution respectively. This representation achieves
its greatest advantage when one is asked to solve Sx = f over a large class of f vectors. For example,
if we wish to steadily increase the force, f2 , on mass 2, and track the resulting displacement we
would be well served by
[L,U] = lu(S);
f = [1 1 1];
for j=1:100,
f(2) = f(2) + j/100;
x = U \ (L \ f);
plot(x,o)
end
You are correct in pointing out that we could have also just precomputed the inverse of S and then
sequentially applied it in our for loop. The use of the inverse is, in general, considerably more costly
in terms of both memory and operation counts. The exercises will give you a chance to see this for
yourself.
We now turn to a matrix that encounters a row swap during its Gaussian Elimination. If

2 2 2
B = 1 4 0
0 5 3
then elimination in column one brings

2 2 2
EB = 0 0 4 .
0 5 3

Rather than eliminating the 5 in column 2 (and in the process destroying the good work we did in
column 1) we simply swap rows 2 and 3 and nd

2 2 2

(3.12)
U = IEB
= 0 5 3
0 0 4
where I is the elementary permutation matrix foreshadowed in (3.10). Regarding the associated
lower triangular matrix we note that I is its own inverse and so
= EB
IU

= B.
and E 1 IU

If we dene L E 1 I then we agree with the Matlab lu usage that L is a psychologically lower
triangular matrix, i.e., a product of lower triangular and elementary permutation matrices.
Finally, we ask what the lovely formulas, (3.11) and (3.12), tell us about the determinants of
products. More precisely, as elimination of S required no row swaps, from
det(S) = det(E2 E1 S) = det(U) and

det(E1 ) = det(E2 ) = 1

we infer that
det(ES) = det(E) det(S)
22

(3.13)

for arbitrary S so long as E is an elementary elimination matrix. While, as elimination of B required

one row swap we infer from

det(B) = det(U) = det(IEB)

and

= 1
det(I)

(3.14)

that
= det(I)
det(B)
det(IB)

(3.15)

for arbitrary B so long as I is an elementary perturbation matrix. Hence, as the LU decomposition

guarantees that every matrix is the product of elementary matrices it follows from (3.14) and (3.15)
that
det(AB) = det(A) det(B)
(3.16)
for every A and B in Rnn .
3.3. A Small Planar Network
We move from uni-axial to biaxial elastic nets by rst considering the swing in Figure 3.2.

k1

k3

k2

m1

m2

x1

x3

x2

x4

Figure 3.2. A simple swing.

We denote by x1 and x2 the respective horizontal and vertical displacements of m1 (positive
is right and down). Similarly, f1 and f2 will denote the associated components of force. The
corresponding displacements and forces at m2 will be denoted by x3 , x4 and f3 , f4 . In computing
the elongations of the three springs we shall make reference to their unstretched lengths, L1 , L2 ,
and L3 .
Now, if spring 1 connects (0, L1 ) to (0, 0) when at rest and (0, L1 ) to (x1 , x2 ) when stretched
then its elongation is simply

(3.17)
e1 = x21 + (x2 + L1 )2 L1 .
The price one pays for moving to higher dimensions is that lengths are now expressed in terms of
square roots. The upshot is that the elongations are not linear combinations of the end displacements
as they were in the uni-axial case. If we presume however that the loads and stinesses are matched
in the sense that the displacements are small compared with the original lengths then we may
eectively ignore the nonlinear contribution
in (3.17). In order to make this precise we need only

recall the Taylor development of 1 + t about t = 0, i.e.,

1 + t = 1 + t/2 + O(t2 )
23

where the latter term signies the remainder. With regard to e1 this allows

e1 = x21 + x22 + 2x2 L1 + L21 L1

= L1 1 + (x21 + x22 )/L21 + 2x2 /L1 L1
= L1 + (x21 + x22 )/(2L1 ) + x2 + L1 O(((x21 + x22 )/L21 + 2x2 /L1 )2 ) L1
= x2 + (x21 + x22 )/(2L1 ) + L1 O(((x21 + x22 )/L21 + 2x2 /L1 )2 ).
If we now assume that
(x21 + x22 )/(2L1 ) is small compared to x2

(3.18)

then, as the O term is even smaller, we may neglect all but the rst terms in the above and so
arrive at
e1 = x2 .
To take a concrete example, if L1 is one meter and x1 and x2 are each one centimeter than x2 is
one hundred times (x21 + x22 )/(2L1 ).
With regard to the second spring, arguing as above, its elongation is (approximately) its stretch
along its initial direction. As its initial direction is horizontal, its elongation is just the dierence
of the respective horizontal end displacements, namely,
e2 = x3 x1 .
Finally, the elongation of the third spring is (approximately) the dierence of its respective vertical
end displacements, i.e.,
e3 = x4 .
We encode these three elongations in

0 1 0 0
e = Ax where A = 1 0 1 0 .
0 0 0 1

Hookes law is an elemental piece of physics and is not perturbed by our leap from uni-axial to
biaxial structures. The upshot is that the restoring force in each spring is still proportional to its
elongation, i.e., yj = kj ej where kj is the stiness of the jth spring. In matrix terms,

k1 0 0
y = Ke where K = 0 k2 0 .
0 0 k3
Balancing horizontal and vertical forces at m1 brings
y2 f1 = 0 and y1 f2 = 0,
while balancing horizontal and vertical forces at m2 brings
y2 f3 = 0 and y3 f4 = 0.
We assemble these into
By = f

0 1 0
1 0 0

where B =
0 1 0 ,
0 0 1
24

and recognize, as expected, that B is nothing more

that x must satisfy Sx = f where

k2

0
S = AT KA =
k2
0
Applying one step of Gaussian Elimination

k2 0 k2
0 k1 0

0 0
0
0 0
0

than AT . Putting the pieces together, we nd

0 k2 0
0
k1 0
.
0 k2 0
0
0 k3

brings

0
x1
f1

0
x2 = f2
0 x3 f1 + f3
k3
x4
f4

and back substitution delivers

x4 = f4 /k3 ,
0 = f1 + f3 ,
x2 = f2 /k1 ,
x1 x3 = f1 /k2 .
The second of these is remarkable in that it contains no components of x. Instead, it provides a
condition on f . In mechanical terms, it states that there can be no equilibrium unless the horizontal
forces on the two masses are equal and opposite. Of course one could have observed this directly
from the layout of the truss. In modern, threedimensional structures with thousands of members
meant to shelter or convey humans one should not however be satised with the visual integrity of
the structure. In particular, one desires a detailed description of all loads that can, and, especially,
all loads that can not, be equilibrated by the proposed truss. In algebraic terms, given a matrix
S one desires a characterization of (1) all those f for which Sx = f possesses a solution and (2)
all those f for which Sx = f does not possess a solution. We provide such a characterization in
Chapter 3 in our discussion of the column space of a matrix.
Supposing now that f1 + f3 = 0 we note that although the system above is consistent it still
fails to uniquely determine the four components of x. In particular, it species only the dierence
between x1 and x3 . As a result both

f1 /k2
0
f2 /k1

and x = f2 /k1
x=
0
f1 /k2
f4 /k3
f4 /k3
satisfy Sx = f . In fact, one may add to either an arbitrary multiple of

1
0

z
1
0

(3.19)

and still have a solution of Sx = f . Searching for the source of this lack of uniqueness we observe
some redundancies in the columns of S. In particular, the third is simply the opposite of the rst.
25

As S is simply AT KA we recognize that the original fault lies with A, where again, the rst and
third columns are opposites. These redundancies are encoded in z in the sense that
Az = 0.
Interpreting this in mechanical terms, we view z as a displacement and Az as the resulting elongation. In Az = 0 we see a nonzero displacement producing zero elongation. One says in this case
that the truss deforms without doing any work and speaks of z as an unstable mode. Again, this
mode could have been observed by a simple glance at Figure 3.2. Such is not the case for more
complex structures and so the engineer seeks a systematic means by which all unstable modes may
be identied. We shall see in Chapter 3 that these modes are captured by the null space of A.
From Sz = 0 one easily deduces that S is singular. More precisely, if S 1 were to exist then S 1 Sz
would equal S 1 0, i.e., z = 0, contrary to (3.19). As a result, Matlab will fail to solve Sx = f
even when f is a force that the truss can equilibrate. One way out is to use the pseudoinverse, as
we shall see below.
3.4. A Large Planar Network
We close with the (scalable) example of the larger planar net in Figure 3.3. Elastic bers,
numbered 1 20, meet at nodes, numbered 1 9. We limit our observation to the motion of the
nodes by denoting the horizontal and vertical displacements of node j by x2j1 and x2j respectively.
Retaining the convention that down and right are positive we note that the elongation of ber 1 is
e1 = x2 x8
while that of ber 3 is
e3 = x3 x1 .
19

10

13

11

20

14

18

17
15

12

16

8
6

Figure 3.3. A crude tissue model.

As bers 2 and 4 are neither vertical nor horizontal their elongations, in terms of nodal displacements, are not so easy to read o. This is more a nuisance than an obstacle however, for recalling
our earlier discussion, the elongation is approximately just the stretch along its undeformed axis.
With respect to ber 2, as it makes the angle /4 with respect to the positive horizontal axis, we
nd

e2 = (x9 x1 ) cos(/4) + (x10 x2 ) sin(/4) = (x9 x1 + x2 x10 )/ 2.

26

Similarly, as ber 4 makes the angle 3/4 with respect to the positive horizontal axis, its elongation
is

e4 = (x7 x3 ) cos(3/4) + (x8 x4 ) sin(3/4) = (x3 x7 + x4 x8 )/ 2.

These are both direct applications of the general formula
ej = (x2n1 x2m1 ) cos(j ) + (x2n x2m ) sin(j )

(3.20)

for ber j, as depicted in the gure below, connecting node m to node n and making the angle j
with the positive horizontal axis when node m is assumed to lie at the point (0, 0). The reader
should check that our expressions for e1 and e3 indeed conform to this general formula and that e2
and e4 agree with ones intuition. For example, visual inspection of the specimen suggests that ber
2 can not be supposed to stretch (i.e., have positive e2 ) unless x9 > x1 and/or x2 > x10 . Does this
jibe with (3.20)?
x

2n-1

2n

original
j
x

deformed
2m

2m-1

Figure 3.4. Elongation of a generic bar, see (3.20).

Applying (3.20) to each of the remaining bers we arrive at e = Ax where A is 20-by-18, one
row for each ber, and one column for each degree of freedom. For systems of such size with such a
well dened structure one naturally hopes to automate the construction. We have done just that in
the accompanying M-le and diary. The M-le begins with a matrix of raw data that anyone with
a protractor could have keyed in directly from Figure 3.3. More precisely, the data matrix has a
row for each ber and each row consists of the starting and ending node numbers and the angle the
ber makes with the positive horizontal axis. This data is precisely what (3.20) requires in order
to know which columns of A receive the proper cos or sin. The nal A matrix is displayed in the
diary.
The next two steps are now familiar. If K denotes the diagonal matrix of ber stinesses and
f denotes the vector of nodal forces then y = Ke and AT y = f and so one must solve Sx = f
where S = AT KA. In this case there is an entire threedimensional class of z for which Az = 0
and therefore Sz = 0. The three indicates that there are three independent unstable modes of the
specimen, e.g., two translations and a rotation. As a result S is singular and x = S\f in Matlab
will get us nowhere. The way out is to recognize that S has 18 3 = 15 stable modes and that if
we restrict S to act only in these directions then it should be invertible. We will begin to make
these notions precise in Chapter 4 on the Fundamental Theorem of Linear Algebra.

27

Figure 3.5. The solid(dashed) circles correspond to the nodal positions before(after) the application of the traction force, f , in (3.21).
For now let us note that every matrix possesses such a pseudo-inverse and that it may be
computed in Matlab via the pinv command. On supposing the ber stinesses to each be one
and the edge traction to be of the form
f = [1 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 1 1]T ,

(3.21)

we arrive at x via x=pinv(S)*f and refer to Figure 3.5 for its graphical representation.
3.5. Positive Denite Matrices
Given a stable mechanical system with stiness matrix S = AT KA Rnn and a load vector
f Rn we rate candidates u Rn for its displacement based on their associated total potential
energy. Where
Total Potential Energy = Internal Strain Energy Work Done by Load
= 12 uT Su uT f.
The resulting minimum principle hinges on two key properties of S = AT KA, inherited from the
physical fact that K is a diagonal matrix with positive numbers on its diagonal. The rst is that it
is symmetric, for S T = (AT KA)T = AT K T A = S, and the second is that it is positive denite,
i.e.,
n

v T Sv = v T AT KAv = (Av)T K(Av) =
kj (Av)2j > 0, v Rn , v = 0.
(3.22)
j=1

The inequality stems from the fact that each stiness, kj > 0, and as A is stable that Av = 0. This
also helps us see why 12 v T Sv is identied as the Internal Strain Energy. For if v is the candidate
displacement then e = Av is the associated elongation, or strain. The associated internal force is
y = Ke and hence kj (Av)2j /2 = ej yj /2 is strain energy stored in the jth ber.
Proposition 3.3. The candidate with the least total potential energy is precisely the equilibrium
solution x = S 1 f . In other words
12 f T S 1 f = minn 12 uT Su uT f.
uR

(3.23)

Proof: Suppose that Sx = f . Now for each u Rn ,

( 12 uT Su uT f ) ( 12 xT Sx xT f ) = 12 xT Sx uT Sx + 12 uT Su
= 12 (x u)T S(x u) 0,

(3.24)

where the last equality uses S = S T and the last inequality uses (3.22). It follows directly from
(3.24) that x = S 1 f indeed minimizes the total potential energy. On substituion of this x into
1 T
x Sx
2

xT f

we arrive at the left side of (3.23). End of Proof.

In addition to conrming our belief that equilibrium solutions should use less energy than other
candidates, this principle can also be used to estimate important physical quantities without ever
28

having to solve Sx = f . To see this, note from (3.24) that, xT f , the actual work done by the load,
obeys
xT f = xT Sx 2uT f uT Su u Rn .
(3.25)
The key point is that we are free to try any candidate on the right hand side. Each choice will
provide a lower bound on the true work done. There are trivial choices, e.g., u = 0 informs is that
xT f 0, and nontrivial choices, e.g., u = f informs us that
xT f f T (2I AT KA)f.
This inequality, in the context of


0
xT f f1 f2 f3 1
0

our small example, (3.5), yields the interesting bound

1 0
f
f1
2


0 1 f2 = f1 f2 f3 f1 + f3 = 2f2 (f1 + f3 ).
f3
f2
1 0

Although developed (here) as a principle of mechanics this proposition has found use in many areas
of physical equilibrium. We will also have occasion to invoke it as an analytical tool. Toward that
end it seem best formulate it in a general setting and in a way too that removes the perhaps
annoying 1/2 factor on the left side of (3.23).
Proposition 3.4. If B Rnn is symmetric and positive denite and f Rn then
f T B 1 f = maxn 2xT f xT Bx,
xR

and the maximum is attained at that x for which Bx = f .

Proof: This is a simple rearrangement of (3.23). In particular, note that
maxn {2xT f xT Bx} = maxn {2( 12 xT Bx xT f )} = 2 minn { 12 xT Bx xT f }.
xR

xR

xR

End of Proof.
Finally, we note that positive denite matrices permit a simpler LU factorization.
Proposition 3.5. Cholesky Factorization If S is symmteric and positive denite then there
exists a lower triangular matrix L, with positive diagonal elements, for which
S = LLT .

(3.26)

Proof: From the symmtery of S Rnn we may begin with the representation


T
s11 S21
S=
,
s11 R, S21 Rn1 , S22 R(n1)(n1) .
S21 S22
As S is positive denite we may conclude that s11 > 0 and
S22

1
T
S21 S21
s11

is positive denite.

(3.27)

To prove the rst claim choose x = (1, 0, . . . , 0)T and note that 0 < xT Sx = s11 . To establish (3.27)
write x = (x1 , x)T with x1 R and conrm that
T
xT Sx = s11 x21 + 2x1 S21
x + xT S22 x.

29

(3.28)

Next, show that you may choose x1 such that

T
T
x S21
S21 = 0
s11 x21 + 2x1 S21

(3.29)

and conclude that this choice conrms the claim in (3.27).

With these preliminaries we proceed to construct the factorization

 

  2

T
s11 S21
l11
l11 LT21
l11
0
l11 LT21
=
=
.
S21 S22
0 LT22
l11 L21 L21 LT21 + L22 LT22
L21 L22
Identifying terms we nd

l11 = s11 ,

L21 = S21 /l11

T
and L22 LT22 = S22 S21 S21
/s11 .

(3.30)

The rst two equalities are explicit and, thanks to s11 > 0, unambiguous. Regarding the third
assignment in (3.30) we note that its right hand side is symmetric by inspection and positive denite
by the argument following (3.29). As such, the third assignment in (3.30) is simply the Cholesky
T
/s11 . Applying the above scheme to this
factorization of the n 1 dimensional matrix S22 S21 S21
will reduce our needs to the Cholesky factorization of an n 2 dimensional matrix. Continuing this
process brings us the trivial one dimensional factorization. End of Proof.
To implement the algorithm at the center of the proof of (3.26) we simply build the columns of
L from longest to shortest. For example, we build the rst column

2 0 0
4 8 16
2 4 8
8 52 92 = 4 l22 0 0 l22 l32
16 92 308
8 l32 l33
0 0 l33
by dividing the original rst column by the square root of its rst element. Proceeding on to the
second column we nd

  




l22 0
52 92
4 
l22 l32
4 8 =

92 308
8
0 l33
l32 l33
that is,

2
= 244 100 and
and nally l33

4
8
16




 
6 10
36 60
6 0
=
0 l33
60 244
10 l33
so l33 = 12. All together,

8 16
2 0 0
2 4 8
52 92 = 4 6 0 0 6 10
92 308
8 10 12
0 0 12

3.6. Exercises
1. With regard to Figure 3.1, (i) Derive the A and K matrices resulting from the removal of the
fourth spring (but not the third mass) and assemble S = AT KA.
(ii) Compute S 1 , by hand via GaussJordan, and compute L and U where S = LU by hand
via the composition of elimination matrices and their inverses. Assume throughout that with
k1 = k2 = k3 = k,
(iii) Use the result of (ii) with the load f = [0 0 F ]T to solve Sx = f by hand two ways, i.e.,
x = S 1 f and Lc = f and Ux = c.
30

2. With regard to Figure 3.2

(i) Derive the A and K matrices resulting from the addition of a fourth (diagonal) ber that
runs from the top of ber one to the second mass and assemble S = AT KA.
(ii) Compute S 1 , by hand via GaussJordan, and compute L and U where S = LU by hand
via the composition of elimination matrices and their inverses. Assume throughout that with
k1 = k2 = k3 = k4 = k.
(iii) Use the result of (ii) with the load f = [0 0 F 0]T to solve Sx = f by hand two ways, i.e.,
x = S 1 f and Lc = f and Ux = c.
3. Prove that if A and B are invertible then (AB)1 = B 1 A1 .
1 .
4. Show that if I is an elementary perturbation of I then II = I and use this to arrive at (I)
5. Note that A is invertible then AA1 = I. Use (3.16) to show that det(A1 ) = 1/ det(A).
6. Generalize gure 2.3 to the case of 16 nodes connected by 42 bers. Introduce one sti
(say k = 100) ber and show how to detect it by properly choosing f . Submit your welldocumented M-le as well as the plots, similar to Figure 3.5, from which you conclude the
presence of a sti ber.
7. We generalize Figure 3.3 to permit ever ner meshes. In particular, with reference to the gure
below we assume N(N 1) nodes where the
horizontal and vertical bers each have length
1/N while the diagonal bers have length 2/N. The top row of bers is anchored to the
ceiling.

(N1)(4N3)
N(N1)

4N2

4N1
4N

N+1

N+2

4
2

2N

4N4

4N3

10

6
7

11

4N5
N

(i) Write and test a Matlab function S=bignet(N) that accepts the odd number N and
produces the stiness matrix S = AT KA. As a check on your work we oer a spy plot of A
when N = 5. Your K matrix should reect the ber lengths as spelled out in (3.1). You may
assume Yj aj = 1 for each ber. The sparsity of A also produces a sparse S. In order to exploit
this, please use S=sparse(S) as the nal line in bignet.m.
(ii) Write and test a driver called bigrun that generates S for N = 5 : 4 : 29 and for each N
solves Sx = f two ways for 100 choices of f . In particular, f is a steady downward pull on
the bottom set of nodes, with a continual increase on the pull at the center node. This can be
done via f=zeros(size(S,1),1); f(2:2:2*N) = 1e-3/N;
31

for j=1:100,
f(N+1) = f(N+1) + 1e-4/N;
This construction should be repeated twice, with the code that closes 3.1 as your guide. In
the rst scenario, precompute S 1 via inv and then apply x = S 1 f in the j loop. In the
second scenario precompute L and U and then apply x = U\(L\f ) in the j loop. In both
cases use tic and toc to time each for loop and so produce a graph of the form
spy(A) when N=5

16

inv
lu

14

10

elapsed time (s)

12
20

30

40

50

10
8
6
4

60

10

20
nz = 179

30

40

200

400

600

800

1000

1200

1400

1600

1800

degrees of freedom

Submit your well documented code, a spy plot of S when N = 9, and a time comparison like
(will vary with memory and cpu) that shown above.
8. Prove (3.27) by following the hints in (3.28)(3.29).
9. Code Cholesky and contrast with chol.

32

4. The Column and Null Spaces

The previous chapter revealed pivots to be the crucial determinants of the solvability of linear
systems of equilibrium equations. In this and the following chapter we consider the general problem
Sx = f for arbitrary S Rmn and f Rm . Pivots will remain pivotal here, bringing us to the
important notions of linear independence, basis, rank and dimension.
4.1. The Column Space
We begin with the direct geometric interpretation of matrixvector multiplication. Recalling
(1.11), the multiplication of the vector x Rn by the matrix S Rmn produces a linear combination of the columns of S,

x1
x2

Sx = [S(:, 1) S(:, 2) S(:, n)]

(4.1)
... = x1 S(:, 1) + x2 S(:, 2) + + xn S(:, n).
xn
The picture I wish to place in your minds eye is that Sx lies in the set spanned by the columns of
S. This set occurs so frequently that we nd it useful to distinguish it with a
Denition 4.1. The column space of the matrix S Rmn is the span of its columns, i.e.,
R(S) {Sx : x Rn }.

(4.2)

This is a subset of Rm . The letter R stands for range. Hopefully our preamble as prepared you
to parse (and soon speak) the set notation used in (4.2). The braces, {} denote set and the colon
denotes such that, for which, or where. Hence, an English translation of {Sx : x Rn } would
be the set of all products of the form Sx where x lies in Rn . But lets not over analyze, we learn
by doing.
The column space of the single column

 
1
S=
1

is the line in the plane through the point (1, 1), while the column space of


1 1
S=
1 0
is the entire plane, i.e., all of R2 . Can you see how each vector in the plane can be written as a
linear combination (weighted sum) of these two columns? We are early in the chapter and so wish
to build intuition and condence so that when we venture into higher dimensions your vertigo is
balanced by your sense of wonder.
For example, the column space of the S matrix associated with the swing in Figure 3.2 is, by
denition,

1
0
1
0

1
0
0
0
4

R(S) = x1 + x2 + x3 + x4 : x R .
1
0
1
0

0
0
0
1
33

And now, although you can not fully visualize this set you can see that the rst and third columns
are colinear, i.e., lie on the same line. As a result we can get by with the more compact description

k2
0
0

0
k
0
1
3
+ x2 + x3 : x R .
R(S) = x1
k2
0
0

k
0
0
3

As the remaining three columns are linearly independent we may go no further. We recognize then
R(S) as a three dimensional subspace of R4 . In order to use these ideas with any real condence
we must establish careful denitions of subspace, independence, and dimension.
A subspace is a natural generalization of line and plane. Namely, it is any set that is closed
under vector addition and scalar multiplication. More precisely,
Denition 4.2. A subset M of Rn is a subspace of Rn when
(S1 ) p + q M whenever p M and q M, and
(S2 ) tp M whenever p M and t R.
Let us conrm now that the column space, R(S), is indeed a subspace. Regarding
(S1 ) if p R(S) and q R(S) then p = Sx and q = Sy for some x and y. Hence, p + q = Sx+ Sy =
S(x + y), i.e., (p + q) R(S). Regarding
(S2 ), tp = tSx = S(tx) so tp R(S).
Note that we used only the denition of R(S) and did not make mention of any particular S.
To show that something is not a subspace it suces to produce one instance that violates one
of the two conditions. For example, to prove that the circle
C = {x R2 : x21 + x22 = 1}
is not a subspace we note that (1, 0) C and (0, 1) C while their sum (1, 1)  C and so (S1 ) is
violated. We could, for good measure, violate condition (S2 ) by noting that 2(1, 0)  C.
We have established that every column space is a subspace. The converse is also true. Every
subspace of Rn is the column space of some matrix. To make sense of this we should more carefully
explain what we mean by span.
Denition 4.3. A collection of vectors {s1 , s2 , . . . , sn } in a subspace M is said to span M when
M = R(S) where S = [s1 s2 sn ].
We will be interested in how a subspace is situated in its ambient space and so will have occasion
to speak of complementary subspaces and even the sum of two subspaces. Lets take care of the
latter right now,
Denition 4.4. If M and Q are subspaces of the same ambient space, Rd , we dene their direct
sum
M Q {p + q : p M and q Q}
as the union of all possible sums of vectors from M and Q.
Do you see how R3 may be written as the direct sum of R1 and R2 ?
4.2. The Null Space
34

If the product of two real numbers is zero then we know that one of them must have be zero.
This inference is false in higher dimensions. For example

   
1 1
1
0
=
.
2 2
1
0
Given a matrix S, we will see that it pays to keep track of those vectors that S annihilates.
Denition 4.5. The null space of S Rmn is the collection of those vectors in Rn that S maps
to the zero vector in Rm . More precisely,
N (S) {x Rn : Sx = 0}.
Let us conrm that N (S) is in fact a subspace. If both x and y lie in N (S) then Sx = Sy = 0
and so S(x + y) = 0. In addition, S(tx) = tSx = 0 for every t R.
As an example we remark that the null space of the S matrix associated with Figure 3.2 is

N (S) = t : t R ,
1

a line in R4 .
The null space answers the question of uniqueness of solutions to Sx = f . For, if Sx = f and
Sy = f then S(x y) = Sx Sy = f f = 0 and so (x y) N (S). Hence, a solution to Sx = f
will be unique if, and only if, N (S) = {0}.
Recalling (4.1) we note that if x N (S) and x = 0, say, e.g., x1 = 0, then Sx = 0 takes the
form
n

xj
s1 =
sj .
x
1
j=2
That is, the rst column of S may be expressed as a linear combination of the remaining columns
of S. Hence, one may determine the (in)dependence of a set of vectors by examining the null space
of the matrix whose columns are the vectors in question.
Denition 4.6. The vectors {s1 , s2 , . . . , sn } are said to be linearly independent if N (S) = {0}
where S = [s1 s2 sn ].
As lines and planes are described as the set of linear combinations of one or two generators, so
too subspaces are most conveniently described as the span of a few basis vectors.
Denition 4.7. A collection of vectors {s1 , s2 , . . . , sn } in a subspace M is a basis for M when the
matrix S = [s1 s2 sn ] satises
(B1 ) M = R(S), and
(B2 ) N (S) = {0}.
The rst stipulates that the columns of S span M while the second requires the columns of S to
be linearly independent. For example, the columns of


1 2
3 4
35

comprise a basis for R2 , while the columns of neither





1 1
1 2 3
nor
2 2
3 4 5
comprise bases for R2 .
4.3. A Blend of Theory and Practice
Let us compute bases for the null and column spaces of the adjacency matrix associated with
the ladder below
1

7
3

8
4

Figure 4.1. An unstable ladder?

The ladder has 8 bars and 4 nodes, so 8 degrees of freedom. Continuing to denote the horizontal
and vertical displacements of node j by x2j1 and x2j we arrive at the A matrix

1
0
0
0
0 0 0 0
1 0
1
0
0 0 0 0

0 1 0
0 0 0 0
0

0
0 1 0 0
0 1 0
A=

0
0 1 0 0 0 1
0
0
0
0
0
1 0 0 0

0
0
0
0 1 0 1 0
0
0
0
0
0 0 1 0
To determine a basis for R(A) we must nd a way to discard its dependent columns. A moments
reection reveals that columns 2 and 6 are colinear, as are columns 4 and 8. We seek, of course,
a more systematic means of uncovering these, and perhaps other less obvious, dependencies. Such
dependencies are more easily discerned from the row reduced form

1 0 0 0 0 0 0 0
0 1 0 0 0 1 0 0

0 0 1 0 0 0 0 0

0 0 0 1 0 0 0 1
Ared = rref(A) =

0 0 0 0 1 0 0 0
0 0 0 0 0 0 1 0

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
Recall that rref performs the elementary row operations necessary to eliminate all nonzeros below
the diagonal.
Each nonzero row of Ared is called a pivot row. The rst nonzero in each row of Ared is called
a pivot. Each column that contains a pivot is called a pivot column. On account of the staircase
nature of Ared we nd that there are as many pivot columns as there are pivot rows. In our example
36

there are six of each and, again on account of the staircase nature, the pivot columns are the linearly
independent columns of Ared . One now asks how this might help us distinguish the independent
columns of A. For, although the rows of Ared are linear combinations of the rows of A no such thing
is true with respect to the columns. The answer is: pay attention only to the indices of the pivot
columns. In our example, columns {1, 2, 3, 4, 5, 7} are the pivot columns. In general
Proposition 4.8. Suppose A Rmn If {Ared (:, cj ) : j = 1, . . . , r} are the pivot columns of Ared
then {A(:, cj ) : j = 1, . . . , r} constitutes a basis for R(A).
Proof: Note that the pivot columns of Ared are, by construction, linearly independent. Suppose,
however, that columns {cj : j = 1, . . . , r} of A are linearly dependent. In this case there exists a
nonzero x Rn for which Ax = 0 and
xk = 0,

k  {cj : j = 1, . . . , r}.

(4.3)

Now Ax = 0 necessarily implies that Ared x = 0, contrary to the fact that the {Ared (:, cj ) : j =
1, . . . , r} are the pivot columns of Ared . (The implication Ax = 0 Ared x = 0 follows from the
fact that we may read row reduction as a sequence of linear transformations of A. If we denote the
product of these transformations by T then T A = Ared and you see why Ax = 0 Ared x = 0. The
reverse implication follows from the fact that each of our row operations is reversible, or, in the
language of the land, invertible.)
We now show that the span of {A(:, cj ) : j = 1, . . . , r} is indeed R(A). This is obvious if r = n,
i.e., if all of the columns are linearly independent. If r < n there exists a q  {cj : j = 1, . . . , r}.
Looking back at Ared we note that its qth column is a linear combination of the pivot columns with
indices not exceeding q. Hence, there exists an x satisfying (4.3) and Ared x = 0 and xq = 1. This x
then necessarily satises Ax = 0. This states that the qth column of A is a linear combination of
those in {A(:, cj ) : j = 1, . . . , r}. End of Proof.
Let us now exhibit a basis for N (A). We exploit the already mentioned fact that N (A) =
N (Ared ). Regarding the latter, we partition the elements of x into so called pivot variables,
{xcj : j = 1, . . . , r}
and free variables
{xk : k  {cj : j = 1, . . . , r}}.
There are evidently n r free variables. For convenience, let us denote these in the future by
{xcj : j = r + 1, . . . , n}.
One solves Ared x = 0 by expressing each of the pivot variables in terms of the nonpivot, or free,
variables. In the example above, x1 , x2 , x3 , x4 , x5 and x7 are pivot while x6 and x8 are free. Solving
for the pivot in terms of the free we nd
x7 = 0, x5 = 0, x4 = x8 , x3 = 0, x2 = x6 , x1 = 0,
37

or, written as a vector,

0
0
1
0


0
0


0
1
(4.4)
x = x6 + x8 ,
0
0
1
0


0
0
0
1
where x6 and x8 are free. As x6 and x8 range over all real numbers the x above traces out a plane
in R8 . This plane is precisely the null space of A and (4.4) describes a generic element as the
linear combination of two basis vectors. Compare this to what Matlab returns when faced with
null(A,r). Abstracting these calculations we arrive at
Proposition 4.9. Suppose that A Rmn has pivot indices {cj : j = 1, . . . , r} and free indices
{cj : j = r + 1, . . . , n}. A basis for N (A) may be constructed of n r vectors {z1 , z2 , . . . , znr }
where zk , and only zk , possesses a nonzero in its cr+k component.
With respect to our ladder the free indices are c7 = 6 and c8 = 8. You still may be wondering
what R(A) and N (A) tell us about the ladder that we did not already know. Regarding R(A)
the answer will come in the next chapter. The null space calculation however has revealed two
independent motions against which the ladder does no work! Do you see that the two vectors in
(4.4) encode rigid vertical motions of bars 4 and 5 respectively? As each of these lies in the null
space of A the associated elongation is zero. Can you square this with the ladder as pictured in
gure 3.1? I hope not, for vertical motion of bar 4 must stretch bars 1,2,6 and 7. This contradiction
is resolved by reecting on the assumption in (3.18).
We close with a few more (modest size) examples. We compute bases for the column and null
spaces of


1 1 0
A=
1 0 1
Subtracting the rst row from the second lands us at


1 1 0
Ared =
0 1 1
hence both rows are pivot rows and columns 1 and 2 are pivot columns. Proposition 4.8 then
informs us that the rst two columns of A, namely
   
1
1
,
(4.5)
1
0
comprise a basis for R(A). In this case, R(A) = R2 .
Regarding N (A) we express each row of Ared x = 0 as the respective pivot variable in terms of
the free. More precisely, x1 and x2 are pivot variables and x3 is free and Ared x = 0 reads
x1 + x2 = 0
x2 + x3 = 0
Working from the bottom up we nd
x2 = x3

and x1 = x3
38

and hence every vector in the null space is of the form

1

x = x3 1 .
1
In other words

N (A) = x3 1 : x3 R

and

1
1
1

constitutes a basis for N (A).

We append a new column and arrive at


1 1 0 2
.
B=
1 0 1 3
The column space of A was already the whole space and so adding a column changes, with respect
to R(A), nothing. That is, R(B) = R(A) and (4.5) is a basis for R(B).
Regarding N (B) we again subtract the rst row from the second,


1 1 0 2
Bred =
0 1 1 1
and identify x1 and x2 as pivot variables and x3 and x4 as free. We see that Bred x = 0 means
x1 + x2 + 2x4 = 0
x2 + x3 + x4 = 0
or, equivalently,
x2 = x3 + x4
and so

and x1 = x3 3x4

1
3

1
1

N (B) = x3 + x4 : x3 R, x4 R
1
0

0
1

and

1
3

1
, 1
1 0

0
1

constitutes a basis for N (B). From these examples we may abstract one useful generality. If m < n
then there will always be at least one free variable. As a result,
Proposition 4.10. If A Rmn and m < n then there exists a nonzero x Rn for which Ax = 0.
In other words, any collection of more than m vectors in Rm will be linearly dependent.
The number of pivots, r, of A Rmn appears to be an important indicator. We shall refer to it
from now on as the rank of A. Our canonical bases for R(A) and N (A) possess r and nr elements
respectively. The number of elements in a basis for a subspace is typically called the dimension
of the subspace. This of course presumes that every basis is the same size. Let us conrm that.
Proposition 4.11. If M is a subspace of Rn and both X = {x1 , . . . , xp } and Y = {y1, . . . , yq } are
bases for M then p = q.
39

Proof: We establish the contrapositive by supposing p = q, or without loss, p < q and then
arguing that the yj are not linearly independent. As each yj is in the span of X it follows that
yj = Xaj for some nonzero aj Rp . We collect these in A Rpq . As q > p we may infer from
the previous proposition the existence of a nonzero z Rq for which Az = 0. Now Y z = XAz = 0
informs us that the columns of Y are linearly dependent. End of Proof.
4.4. The Structure of a Nilpotent Matrix
The matrix A is said to be nilpotent when Am
For example,

2 2

A= 5 1
1 5

= 0 while Am1 = 0 for some integer m > 1.

2
3
3

(4.6)

obeys A3 = 0 while A2 = 0. Nilpotent matrices arise naturally when studying the eigenvalue
problem. In fact, as we will see in Chapter 11, they are the only obstruction to a clean eigen
decomposition. We study them here because they may be fully decomposed by their growing
sequence of null spaces
N (A) N (A2 ) N (Am1 ) N (Am ) = Rn .

(4.7)

In fact, we will show that the structure of a nilpotent matrix is completely encoded in the sequence
of dimensions,
dj dim N (Aj ), for j = 1, . . . , m.
Our rst interesting observation is that these dimensions can not grow too quickly. More precisely,
dj dj1 d1 ,

for j = 2, . . . , m.

(4.8)

We build up to the full decomposition by rst putting a nilpotent matrix in strictly (only zeros
on the diagonal) upper triangular form. We begin with a basis, X1 = {xj : j = 1, . . . , d1 } for N (A)
and then expand this to basis of N (A2 ) by appending X2 = {xj : j = d1 + 1, . . . , d2}. We continue
this process until we arrive at a basis {xj : j = 1, . . . , n} for all of Rn . Regarding the example in
(4.6) we nd

1
x1 = 1
2
to be a basis for N (A) and

2
v1 = 3
3

0
and v2 = 1
1

to be a basis for N (A2 ). As v1 = 2x1 + v2 we choose x2 = v2 . To nd x3 we simply look for anything

not in the span of x1 and x2 . Note that

0

x3 = 0
1
40

will do. Let us now examine the action of A onto our basis vectors. The rst is easy, Ax1 = 0. For
the second, as x2 N (A2 ) we nd that Ax2 N (A) and hence Ax2 must be a multple of x1 . In
fact
Ax2 = 4x1 .
Finally, by the same token, Ax3 N (A2 ) and so Ax3 is a linear combination of x1 and x2 . In fact
Ax3 = 2x1 x2 .
If we gather our ndings we nd
A[x1 x2 x3 ] = [0 4x1 2x1 x2 ] = [x1 x2 x3 ]U
where U is the strictly upper triangular matrix

0 4 2
U = 0 0 1 .
0 0 0

(4.9)

(4.10)

We write (4.9) as AX = XU and note that by construction the columns of X are linearly independent. As such X is invertible and we may multiply across by X 1 and arrive at
X 1 AX = U.

(4.11)

It is no more dicult to establish the general case.

Proposition 4.12. If A is nilpotent then there exists an invertible matrix X and a strictly upper
triangular matrix, U for which (4.11) holds.
Proof: We begin with a basis, X1 = {xj : j = 1, . . . , d1 } for N (A) and then expand this to basis
of N (A2 ) by appending X2 = {xj : j = d1 +1, . . . , d2}. The key observation is that, by construction,
for each xj X2 we know that
Axj = 0 while A(Axj ) = 0.
In other words, Axj is a nontrivial linear combination of elements of N (A). As as result, Axj = 0
for the j d1 while Axj for d1 + 1 j d2 is a linear combination of the strictly prior xj , i.e., for
j d1 . Proceeding on, if necessary, the xj added to complete a basis for N (A3 ) can be expressed
as a linear combination of strictly prior xj . End of Proof.
This transformation of A into U by X is called a similarity transformation and in this case
A and U are said to be similar. It follows from (1.14) and (3.16) that if two matrices are similar
then they same determinant and trace.
Corollary 4.13. If A is nilpotent then det(A) = tr(A) = 0.
To see the ner structure we recognize that we have a lot a freedom in how we choose the many
bases. The key point is to start at the right end. It also helps to have the right denition.
Denition 4.14. Let U and V be subspaces of Rn and U V . We say that the set of vectors
{v1 , , vp } V is linearly independent mod U when
p


ai vi U implies that each ai = 0.

i=1

41

We say that {v1 , , vp } is a basis of V mod U when it is linearly independent mod U and
V = U + span{v1 , , vp }
It is easy to nd a basis of V mod U: simply choose a basis {u1, . . . , uk } for U and extend it to
a basis {u1 , . . . , uk , v1 , . . . , vp } for V . The vectors {v1 , . . . , vp } are then a basis of V mod U.
A Jordan block of size s is an s-by-s matrix of zeros save the superdiagonal, that is populated
by ones. For example, the Jordan blocks of sizes 1, 2 and 3 are



0 1 0
0 1
J1 = 0, J2 =
, J3 = 0 0 1 .
(4.12)
0 0
0 0 0
We may now establish the fundamental result.
Proposition 4.15 If Am1 = 0 and Am = 0 then there exists an invertible matrix X and a Jordan
matrix J for which
X 1 AX = J.
(4.13)
J is the block diagonal matrix begining with c1 Jordan blocks of size 1, c2 Jordan blocks of size 2
up through cm Jordan blocks of size m. The chain numbers, cj , are determined by the null space
dimensions, dj , via

2 1

1 2 1

.
.
.

.
.
.
(4.14)
c = Sm d where Sm =
.
.
.

1 2 1
1 1
is the stiness matrix (recall Exer. 3.1) for the hanging elastic chain with m unit stinesses.
Proof: We follow the recipe,
1
cm
Step 1: Choose a basis {vm
, . . . , vm
} of N (Am ) mod N (Am1 ). There are cm = dm dm1 such
basis vectors. Each of these vectors will generate a chain of length m and a Jordan block of size m.
i
N (Am1 ).
Step 2: If m = 1, stop. Else, apply A to the vectors constructed above, obtaining Avm
Then the key point is that
1
cm
{Avm
, . . . , Avm
} is linearly independent mod N (Am2 ).
c

m1
1
, . . . , vm1
}.
Now extend these (if necessary) to a basis of N (Am1 ) mod N (Am2 ), by appending {vm1
Each of these new appended vectors will generate a chain of length m 1 and a Jordan block of
size m 1. Since we have extended a list of size cm to reach a length dm1 dm2 , there are
cm1 = (dm1 dm2 ) (dm dm1 ) chains of length m 1.
Step 3: Repeat Step 2, with m replaced by m 1.
Once the algorithm terminates, we may arrange the chosen basis as follows:

X = {v11 , . . . , v1c1 , Av21 , v21 , . . . , Av2c2 , v2c2 , , . . . , Ai1 vi1 , . . . , vi1 , . . . , Ai1 vici , , . . . , vici , . . .}
and so AX = XJ where J is the block diagonal matrix begining with c1 zeros, then c2 blocks of
size 2, then c3 blocks of size 3 up through cm blocks of size m. The formulas for cm and cm1 in
Steps 1 and 2 are precisely those implemented by (4.14). End of Proof.
42

For example,

1 1 1
A = 2 2 2
1 1 1

obeys A2 = 0, and has null space dimensions, d1 = 2 and d2 = 3. The proposition then dictates
that we will encounter c1 = 2d1 d2 = 1 chain of length 1, and c2 = d2 d1 = 1 chain of length 2.
We compute the associated basis vectors according to the recipe above. Following Step 1 we note
(any vector not in N (A) will suce) that

0
1

v2 = 0
1
is a basis of N (A2 ) mod N (A). Step 2 leads us to

1
Av21 = 2
1
which, we are happy to conrm, indeed lies in N (A). We continue within Step 2 to complete a
basis for N (A) by appending

0
1

v1 = 1 .
1
This completes all of the steps and so we assemble
X = [v11 Av21 v21 ]
and develop

0 0 0
AX = [0 0 x2 ] = XJ where J = 0 0 1 .
0 0 0
We have been a little cavalier regarding the extending of bases. Our examples have been so small
that we have been able to proceed by visual inspection. What we need is something like a recipe.
We begin with with the decreasing shrinking sequence of column spaces
R(A) R(A2 ) R(Am1 ) R(Am ) = 0

and note that

dim(N (Am ) mod N (Am1 )) = n dm1
is precisely the dimension of R(Am1 ). Now for each pivot column yk of Am1 we set xk to the
associated coordinate vector. It follows that Am1 xk = yk which ensures that xk = N (Am1 ). As
the xk are linearly independent by construction, they comprise a basis for N (Am ) mod N (Am1 ).
As a result, Step 1 may be accomplished concretely via
Step 1: Let pj denote the index of the jth pivot column of Am1 , where j = 1, . . . , dim R(Am1 ).
j
to be zero in every element except pj , where it take the value 1.
Dene vm
Each of the extensions in the subsequent steps may be done in a similar fashion.
4.5. Exercises
43

1. Which of the following subsets of R3 are actually subspaces? Check both conditions and show
your work.
(a) All vectors whose rst component x1 = 0.
(b) All vectors whose rst component x1 = 1.
(c) All vectors whose rst two components obey x1 x2 = 0.
(d) The vector (0, 0, 0).
(e) All linear combinations of the pair (1, 1, 0) and (2, 0, 1).
(f) All vectors for which x3 x2 + 3x1 = 0.
2. I encourage you to use rref and null for the following. (i) Add a diagonal crossbar between
nodes 3 and 2 in Figure 4.1 and compute bases for the column and null spaces of the new
adjacency matrix. As this crossbar fails to stabilize the ladder we shall add one more bar. (ii)
To the 9 bar ladder of (i) add a diagonal cross bar between nodes 1 and the left end of bar 6.
Compute bases for the column and null spaces of the new adjacency matrix.
3. We wish to show that N (A) = N (AT A) regardless of A.
(i) We rst take a concrete example. Report the ndings of null when applied to A and AT A
for the A matrix associated with Figure 4.1.
(ii) For arbitrary A show that N (A) N (AT A), i.e., that if Ax = 0 then AT Ax = 0.
(iii) For arbitrary A show that N (AT A) N (A), i.e., that if AT Ax = 0 then Ax = 0. (Hint:
if AT Ax = 0 then xT AT Ax = 0 and this says something about Ax.)
4. Suppose that A Rmn and N (A) = Rn . Argue that A must be the zero matrix.
5. Show that if AB = 0 then R(B) N (A).
6. Show that if Am1 v = 0 and Am v = 0 then the chain {v, Av, . . . , Am1 v} is linearly independent.
7. Suppose that A Rnn obeys Am = 0 while Am1 = 0. (i) Prove that (4.7) holds. (ii) Prove
that (4.8) holds.
8. Suppose that A and B are both n-by-n. Show that if one of them is invertible then AB is
similar to BA.
9. Consider the nilpotent matrix

2
7

A=
1
1
1

2
1
7
1
1

2
1
1
7
1

2
1
1
1
7

4
5

5
.
5
5

Use Matlab to conrm that dj = dim N (Aj ) = j for j = 1, . . . , 5. Conclude that A is similar
to J5 . Find a basis for N (A5 ) mod N (A4 ) via Steps 1 of 4.4. Use this basis vector to complete
the Jordan chain and write out the the full transformation matrix, X.

44

5. The Fundamental Theorem of Linear Algebra

The previous chapter, in a sense, only told half of the story. In particular, A Rmn maps
R into Rm and its null space lies in Rn and its column space lies in Rm . Having seen examples
where R(A) was a proper subspace of Rm one naturally asks about what is left out. Similarly, one
wonders about the subspace of Rn that is complementary to N (A). These questions are answered
by the column space and null space of AT .
n

5.1. The Row Space

As the columns of AT are simply the rows of A we call R(AT ) the row space of A. More precisely
Denition 5.1. The row space of A Rmn is the span of its rows, i.e.,
R(AT ) {AT y : y Rm }.
This is a subspace of Rn .
Regarding a basis for R(AT ) we recall that the rows of Ared rref(A) are merely linear combinations of the rows of A and hence
R(AT ) = R((Ared )T ).
Recalling that pivot rows of Ared are linearly independent and that all remaining rows of Ared are
zero leads us to
Proposition 5.2. The pivot rows of Ared constitute a basis for R(AT ).
As there are r pivot rows of Ared we nd that the dimension of R(AT ) is r. Recalling Proposition
2.2 we nd the dimensions of N (A) and R(AT ) to be complementary, i.e., they sum to the dimension
of the ambient space, n. Much more in fact is true. Let us compute the inner product of an arbitrary
element x R(AT ) and z N (A). As x = AT y for some y we nd
xT z = (AT y)T z = y T Az = 0.
This states that every vector in R(AT ) is perpendicular to every vector in N (A).
Let us test this observation on the A matrix stemming from the unstable ladder of 3.4. Recall
that


0
0
1
0


0
0


0
1
z1 = and z2 =
0
0
1
0


0
0
0
1
45

constitute a basis for N (A) while the pivot rows of Ared are



1
0
0
0
1
0



0
0
1



0
0
0
x1 = , x2 = , x3 = ,
0
0
0
0
1
0



0
0
0
0
0
0
and

0
0

0

1
x4 = ,
0
0

0
1

0
0

0

0
x5 = ,
1
0

0
0

0
0

0

0
x6 = .
0
0

1
0

Indeed, each zj is perpendicular to each xk . As a result,

{z1 , z2 , x1 , x2 , x3 , x4 , x5 , x6 }
comprises a set of 8 linearly independent vectors in R8 . These vectors then necessarily span R8 .
For, if they did not, there would exist nine linearly independent vectors in R8 ! In general, we nd
Proposition 5.3 Fundamental Theorem of Linear Algebra (Preliminary). Suppose A Rmn has
rank r. The row space, R(AT ), and the null space, N (A), are respectively r and n r dimensional
subspaces of Rn . Each x Rn may be uniquely expressed in the form
x = xR + xN ,

where xR R(AT ) and xN N (A).

(4.1)

Recalling Denition 4 from Chapter 3 we may interpret (4.1) as

Rn = R(AT ) N (A).
As the constituent subspaces have been shown to be orthogonal we speak of Rn as the orthogonal
direct sum of R(AT ) and N (A). We have worked with a large example that ows from a real
problem. It is however dicult to fully visualize the resulting spaces. So we consider

1 1 1
A = 1 1 1
(5.1)
1 1 1
note that its row space is spanned by

x1 = 1
1

1
and x2 = 1
1
46

and that its null space is spanned by

1
x3 = 0 .
1
We illustrate in Figure 5.1 the complementary piercing of R(AT ) by N (A).

x1

0.5

0.5

1
1
1

0.5
0.5

0
0

0.5

0.5
1

Figure 5.1. A depiction of R3 = R(AT ) N (A) for the A of (5.1). Here a nite piece (gray) of
the R(AT ) plane is generated by x1 and x2 while the a nite piece of the N (A) line is generated
by x3 .
5.2. The Left Null Space
The Fundamental Theorem will more than likely say that Rm = R(A) N (AT ). In fact, this
is already in the preliminary version. To coax it out we realize that there was nothing special
about the choice of letters used. Hence, if B Rpq then the preliminary version states that Rq =
R(B T )N (B). As a result, letting B = AT , p = n and q = m, we nd indeed Rm = R(A)N (AT ).
That is, the left null space, N (AT ), is the orthogonal complement of the column space, R(A).
The word left stems from the fact that AT y = 0 is equivalent to y T A = 0, where y acts on A
from the left.
In order to compute a basis for N (AT ) we merely mimic the construction of the previous section.
Namely, we compute (AT )red and then solve for the pivot variables in terms of the free ones.
With respect to the A matrix associated with the unstable ladder of 3.4, we nd

1 1 0
0
0 0 0
0
0 0
0 1 0 0 0
0

0 0 0
0
0 1 1 0

0
0 1 0 0
0
0 0
AT =

0
0
0 1 1 0
0 0
0 0
0
1
0 0 0
0

0 0
0
0
0 0 1 1
0 0
0
0
1 0 0
0
47

and

(AT )red

1
0

0
= rref(AT ) =
0
0

0
0

0 1 0 0 0 0 0
1 1 0 0 0 0 0

0 0 1 0 0 0 0

0 0 0 1 0 0 0
.
0 0 0 0 1 0 1
0 0 0 0 0 1 1

0 0 0 0 0 0 0
0 0 0 0 0 0 0

We recognize the rank of AT to be 6, with pivot and free indices

{1, 2, 4, 5, 6, 7} and {3, 8}
respectively. Solving (AT )red x = 0 for the pivot variables in terms of the free we nd
x7 = x8 , x6 = x8 , x5 = 0, x4 = 0, x2 = x3 , x1 = x3 ,
or in vector form,

1
0
1
0


1
0


0
0
x = x3 + x8 .
0
0
0
1


0
1
0
1

These two vectors constitute a basis for N (AT ) and indeed they are both orthogonal to every column
of A. We have now exhibited means by which one may assemble bases for the four fundamental
subspaces. In the process we have established
Proposition 5.4 Fundamental Theorem of Linear Algebra. Suppose A Rmn has rank r. One
has the orthogonal direct sums
Rn = R(AT ) N (A) and Rm = R(A) N (AT )
where the dimensions are

dimR(A) = dimR(AT ) = r
dimN (A) = n r
dimN (AT ) = m r.

5.3. Synthesis
We pause here to reect on, assess, and hopefully synthesize the work of these last two, theory
heavy, chapters. For they provide a complete answer to the two part question:
Q: Given an A Rmn (1) for which b Rm does there an exist an x Rn such that Ax = b and,
(2) when such an x exists is it unique?
Admittedly this looks like a question only of interest to theoreticians. The practical engineer
states My trusted model has produced an A and a b and Gaussian Elimination either delivers an
48

x or it does not. In the case where it does not I believe that among the engineers colleagues,
superiors, or customers someone would raise the question why?, not merely out of idle speculation
but out of fear that perhaps the engineer has placed too much trust in her model. If she, by reference
to the Fundamental Theorem is able to quell their fears, I imagine their next question being, so
what are you going to do about it?
We have seen this dialog played out in Chapter 3 where, when GE failed we were saved by the
pseudo-inverse. That got us over (not really) the uniqueness question, Q2, but ignored existence
Q1. One simply needs a theoretical foundation to diagnose problems that defeat our intuition. So
lets get on with it.
Regarding Q1 we have a complete answer in Chapter 4 that simply named a notion we already
nailed in Chapter 1. Namely, given A, we may only solve Ax = b when b is a linear combination of
the columns of A, i.e., when b R(A). And how does one see whether or not b R(A)? We have
oered two approaches. The direct path: apply Gaussian Elimination to the augmented system
[A|b], and the indirect path: check whether b is orthogonal to N (AT ). And how does one execute
mr
for N (AT ) and check the m r inner products, bT yj .
that indirect path? Find a basis, {yj }j=1
In practice the rank, r, is close m and so this can be easier than the direct approach although
the direct approach actually delivers an x when b R(A). In case b = R(A) there of course is no
x. The remedy we propose in the next chapter is to nd that x for which Ax is as close to b as
possible. The implementation of the remedy makes crucial use of the Fundamental Theorem.
Regarding Q2 we recall that if Ax = b and Ay = b then A(xy) = bb = 0, i.e., (xy) N (A).
Conversely, if Ax = b and z N (A) then A(x + z) = b. Hence the full answer to Q2 is that Ax = b
has at most one solution if and only if N (A) = 0. And how does one conrm this? The direct path
is: Solve Az = 0 via Gaussian Elimination. The indirect path is: show that R(AT ) = Rn . Either
way, you must show that the rank, r, is full, i.e., r = n. An equivalent rephrasing of the direct path
is: show that the columns of A are linearly independent. In the case that r < n no solution will
be unique and one must turn to physical or mathematical criteria for selecting among the (innite
number of) solutions. We shall see in Chapter 13 that the pseudo-inverse selects the solution with
the smallest norm.
So, can we synthesize some lessons that let us quickly address our two fundamental questions?
Suppose m = n. If r = m then, for each b there exists a unique x such that Ax = b. In this case,
the columns of A are linearly independent and A is invertible. If r < m then there are b for which
there does not exist an x for Ax = b and, even for those b for which such an x exists, this x is not
unique.
Suppose m < n. As the rank can not exceed the smaller of m and n it follows that r < n and
hence the columns are linearly dependent, dim N (A) > 0, and solutions to Ax = b are not unique.
If r = m then solutions exist for every b. If r < m then solutions exist for only some b.
Suppose m > n. As the rank can not exceed the lesser of m and n it follows that r < m and so
Ax = b will have a solution only for some b. If r = n then these solutions will be unique. If r < n
no solutions will be unique.
5.4. Vector Spaces and Linear Transformations
Our study has been motivated by the equations of equilibrium of electrical and mechanical
systems. We then abstracted these square systems to rectangular systems and captured the action
of an m n matrix via its four fundamental subspaces of two Euclidean spaces, Rn and Rm . These
49

spaces were introduced as the collection of real vectors with n and m components respectively. It
is natural, and we shall see necessary, to abstract further to vectors of complex numbers, complex
matrices, complex functions and beyond. All of these are captured by the notion of vector space.
Denition 5.5. A vector space over the complex numbers, C, is a set V together with two
operations, one of vector addition and one of scalar multiplication, that satisfy the eight axioms
listed below. In the list below, let u, v and w be arbitrary vectors in V , and a and b arbitrary
complex numbers.
1. Associativity of vector addition, u + (v + w) = (u + v) + w.
2. Commutativity of vector addition, u + v = v + u.
3. Identity element of vector addition. There exists an element 0 V , called the zero vector,
such that v + 0 = v for each v V .
4. Inverse elements of vector addition. For every v V , there exists an element v V , called
the additive inverse of v, such that v + (v) = 0.
5. Distributivity of scalar multiplication with respect to vector addition, c(u + v) = cu + cv.
6. Distributivity of scalar multiplication with respect to complex addition, (a + b)v = av + bv.
7. Compatibility of scalar multiplication with complex multiplication, a(bv) = (ab)v.
8. Identity element of scalar multiplication, 1v = v.
If you are not familiar with complex arithmetic you may, for now, assume that V is built over
the reals, R. We will cover complex arithmetic, from the beginning, in Chapter 9. Do you see
that Rn is a vector space? Also note that the collection of matrices Rmn is a vector space. As
is the collection of all polynomials with complex coecients. All of the key spacial concepts so
far discussed in Rn , notably, subspace, basis and dimension extend intact to all vector spaces. For
example the set of 2by2 upper triangular matrices is a 3 dimensional subspace of R22 with basis






0 0
0 1
1 0
.
,
,
0 1
0 0
0 0
If A is a rule for transforming elements of the vector space V into elements of the vector space
W we call it linear if
A(av + bw) = aAv + bAw,

v V, w V, a C, b C.

(5.2)

If V and W are nite dimensional, with bases {v1 , v2 , . . . , vn } and {w1 , w2 , . . . , wm } respectively
then (5.2) implies that
A(c1 v1 + c2 v2 + + cn vn ) = c1 A(v1 ) + c2 A(v2 ) + + cn A(vn ),
for arbitrary c Cn . Now, as each A(vj ) W it may be expressed in terms of the wi . In particular,
we nd
A(vj ) = a1j w1 + a2j w2 + + amj wm .
And so

A(c1 v1 + c2 v2 + + cn vn ) = (a11 c1 + a12 c2 + + a1n cn )w1

+ (a21 c1 + a22 c2 + + a2n cn )w2
+ + (am1 c1 + am2 c2 + + amn cn )wm .
50

We now view A as transforming the coecients of the vj into the coecients of the wi via matrix
vector multiplication. Where as this operation was ordained in 1.2 we see it here follow as a
natural consequence of linearity.
For example, the transformation of the plane that reects vectors over the horizontal axis,
A(x1 , x2 ) (x1 , x2 )
is linear. In the standard basis,
v1 = (1, 0),

v2 = (0, 1)

we nd A(v1 ) = v1 and so a11 = 1 and a12 = 0. Similarly A(v2 ) = v2 and so a21 = 0 and a22 = 1
and we arrive at the standard matrix representation


1 0
As =
.
0 1
If instead however we choose the tilted basis
w2 = (1, 1)

w1 = (1, 1),

then A(w1 ) = w2 and A(w2 ) = w1 and we arrive at the tilted matrix representation


0 1
At =
.
1 0
As As and At represent the same transformation it should not be surprising that the change of
basis matrix serves as a similarity transformation between the representations. In particular, note
that


1 1
B
1 1
is the basis changer, i.e., w1 = Bv1 , w2 = Bv2 , and as a result the simililarity transformation
At = B 1 As B.
Other than providing a rational for matrixvector multiplication this can appear to be much ado
about nothing. There are however many linear transformations that lack an obvious matrix representation. For example, the transpose operation is a linear transformation of Cmn to Cnm . Also
note that the derivative is a linear transformation on the space of polynomials.
5.5. Exercises
1. True or false: support your answer.
(i) If A is square then R(A) = R(AT ).
(ii) If A and B have the same four fundamental subspaces then A=B.
2. Construct bases (by hand) for the four subspaces associated with


1 1 1
A=
.
1 0 1
Also provide a careful sketch of these subspaces.
51

3. Show that if AB = 0 then R(B) N (A).

4. Why is there no matrix whose row space and null space both contain the vector [1 1 1]T ?
5. Write down a matrix with the required property or explain why no such matrix exists.
(a) Column space contains [1 0 0]T and [0 0 1]T while row space contains [1 1]T and [1 2]T .
(b) Column space has basis [1 1 1]T while null space has basis [1 2 1]T .
(c) Column space is R4 while row space is R3 .
6. One often constructs matrices via outer products, e.g., given v Rn let us consider A = vv T .
(a) Show that v is a basis for R(A),
(b) Show that N (A) coincides with all vectors perpendicular to v.
(c) What is the rank of A?

52

6. Least Squares
We learned in the previous chapter that for a given A Rmn , the equation Ax = b need not
possess a solution when the number of rows of A exceeds its rank, i.e., when r < m. We detail
two natural engineering contexts, where material parameters are to be inferred from experiment, in
which the governing A matrix has r < m. To resolve the inconsistent system Ax = b we project the
faulty b into the column space of A and proceed to solve the consistent, so-called normal equations,
AT Ax = AT b. We then develop the associated theory of projection matrices and show how this
permits us to transform linearly independent collections of vectors into orthonormal collections.
6.1. The Normal Equations
The goal is to choose x such that Ax is as close as possible to b. Measuring closeness in terms
of the sum of the squares of the components we arrive at the least squares problem of minimizing
Ax b2 (Ax b)T (Ax b)

(6.1)

over all x Rn . The path to the solution is illuminated by the Fundamental Theorem. More
precisely, we write
b = bR + bN

where bR R(A) and bN N (AT ).

On noting that (i) (Ax bR ) R(A) for every x Rn and (ii) R(A) N (AT ) we arrive at the
Pythagorean Theorem
Ax b2 = Ax bR bN 2 = Ax bR 2 + bN 2 ,

(6.2)

As bN is what it is, (6.2) states that the best x is the one that satises
Ax = bR .

(6.3)

As bR R(A) this equation indeed possesses a solution. We have yet however to specify how one
computes bR given b. Although an explicit expression for bR , the so called orthogonal projection
of b onto R(A), in terms of A and b is within our grasp we shall, strictly speaking, not require it.
To see this, let us note that if x satises (6.3) then
Ax b = Ax bR bN = bN .

(6.4)

As bN is no more easily computed than bR you may claim that we are just going in circles. The
practical information in (6.4) however is that (Ax b) N (AT ), i.e., AT (Ax b) = 0, i.e.,
AT Ax = AT b.

(6.5)

As AT b R(AT ) regardless of b this system, often referred to as the normal equations, indeed
has a solution. This solution is unique so long as the columns of AT A are linearly independent, i.e.,
so long as N (AT A) = {0}. Recalling Exer. 4.3, we note that this is equivalent to N (A) = {0}. We
summarize our ndings in
Proposition 6.1. Suppose that A Rmn . The set of x Rn for which the mist Ax b2 is
smallest is composed of those x for which
AT Ax = AT b.
53

There is always at least one such x. There is exactly one such x if and only if N (A) = {0}.
As a concrete example, we take

1 1
A = 0 1
0 0

1
and b = 1 .
1

(6.6)

and plot R(A) and the associated decomposition of b in Figure 6.1.

0.8

y3

0.6
0.4
0.2
0
2

R(A)
1.5

2
1

0.5

0
0

0.5

1
1.5

Figure 6.1. The decomposition of b = bR + bN into its column space and left null space components.
As this b = R(A) there is no x such that Ax = b. Indeed,
Ax b2 = (x1 + x2 1)2 + (x2 1)2 + 1 1,
with the minimum uniquely attained at

 
0
,
x=
1

in agreement with the unique solution of (6.5), for



 
1 1
1
T
T
A A=
and A b =
.
1 2
2
We now recognize, a posteriori, that

bR = Ax = 1
0

is the orthogonal projection of b onto the column space of A.

We consider a more typical example, attempting to determine the neuronal membrane conductance, G, and reversal potential, E, by tting Ohms law for the putative neuronal membrane
current,
b = G(v E),
(6.7)
54

where we have m current measurements, bj , at m prescribed voltage levels, vj . The key step is to
recognize (6.7) as an m-by-2 system of equations for the unknown biophysical parameters x1 = G
and x2 = GE, via
vj x1 x2 = bj , j = 1, . . . , m
and to translate this into the least squares problem Ax = b where

v1 1
v2 1
A=
.
..
...
.
vm 1
We illustrate this in Figure 6.2 on noisy synthetic data generated by the true values, G = 1.2 mS
and E = 70 mV . We suppose we have 21 accurate voltage measurements, between 75 mV and
65 mV in steps of 0.5 mV . We then generate the 21 true values of b via (6.7) and then soil these
with randn noise.
8
6
4

b (A)

2
0
2
4
6
8
75

70

65

v (mV)

Figure 6.2. The X points are the synthetic noisy measurements. On solving the associated
least squares problem, AT Ax = AT b we recover G = 1.18 and E = 68.75, which when used in
(6.7), yields the solid straight line.
6.2. Probabilistic and Statistical Interpretations
The previous example oers entry into the fascinating world of probability theory. To say that
our jth experiment was tainted with Gaussian noise (of zero mean and variance 2 ) is to say that
aj x = bj + j

(6.8)

where aj is the jth row of A and that j is drawn from a probability distribution with density
1
(6.9)
p() exp(2 /(2 2 )).
2
This permits us to write the probability of observing bj given aj and the candidate x, as
1
p(bj |aj ; x) = exp((aj x bj )2 /(2 2 )).
2
55

Next, as we expect the jth and kth experiments to be independent of one another (i.e., errors in
one should not eect errors in the other) we can write the probability of observing bj and bk given
aj and ak and the candidate x, as the product of the individual probabilities
p((bj , bk )|(aj , ak ); x) =

1
2 2

exp(((aj x bj )2 (ak x bk )2 /(2 2 )).

Combining now all of the experiments, we nd

p(b|A; x) =

1
exp(Ax b2 /(2 2 )).
m
m/2
(2)

This is often abbreviated as L(x) and interpreted as the likelihood of A producing b given x. The
principle of maximum likelihood is to choose x to maximize this likelihood. This is clearly the
x that minimizes Ax b2 .
This rationale generalizes naturally to the case where each experiment produces more than one
measurement. In this case each j Rn is drawn from the multivariate density
1
exp(T C 1 /2).
p() 
n
(2) det(C)

(6.10)

where C Rnn is the symmetric and positive denite covariance matrix. It follows that the
probability of measuring bj Rn given aj Rnm and x Rm is
p(bj |aj ; x) = 

1
(2)n

det(C)

exp((aj x bj )T C 1 (aj x bj )/2).

We would like to write this as a simple square. The way out is via the Cholesky factorization (recall
(3.26))
C = LLT
for then (exercise)
1
exp(L1 (aj x bj )2 /2).
p(bj |aj ; x) = 
(2)n det(C)
and so using independence, after stacking and blocking
p(b|A; x) =

((2)n

1
exp(L1 (Ax b)2 /2).
det(C))m/2

and so the x that maximizes the likelihood is the x that satises the weighted least squares problem
AT C 1 Ax = AT C 1 b.
From the statistical point of view we call ELS (AT A)1 AT the Least Squares Estimator.
We note that ELS b is clearly linear in b and we show here that if j is simply assumed to be
independent and drawn from a distribution with mean zero and variance 2 then ELS b is the Best
Linear Unbiased Estimate.
A linear estimator E is called unbiased if the mean value of Eb is x. But this is easy,
mean(Eb) = mean(EAx E) = EAx + Emean() = EAx
56

and so E is unbiased i EA = I. Please note that ELS is indeed unbiased. It follows that the
variance of the unbiased estimate is simply
var(Eb) mean((Ebx)(Ebx)T ) = mean((E(Ax)x)(E(Ax)x)T ) = Evar()E T . (6.11)
From here we can now establish that ELS b is best in the sense that it has the least variance, in the
sense of postivive denite matrices. To begin we write E = ELS + D and note that EA = I implies
T
= ELS D = 0. As such, in the single measurement case, i.e., var() = 2 I,
DA = 0 and so DELS
we nd
var(Eb) = Evar()E T = 2 EE T = 2 (ELS + D)(ELS + D)T = var(ELS b) + 2 DD T .
As DD T is a positive semidenite we have shown that
y T var(ELS b)y y T var(Eb)y
for every vector y and every Linear Unbiased Estimator, E.
In the multivariate case where var() = C we again invoke its Cholesky factorization, C = LLT ,
to transform b = Ax + (or) to
where b = L1 b, A = L1 A and = L1 .

b = Ax + ,

This decorrelates the error terms for

var() = var(L1 ) = L1 CLT = L1 LLT LT = I.
And so we may argue as above that
var(Eb) = Evar(b)E T = (ELS + D)(ELS + D)T = var(ELS b) + DD T
where

1
1 T T 1
T 1
A)1 AT C 1
ELS = (A A)1 A = (AT LT
n Ln A) A Ln Ln = (A C

as above.
6.3. Applying Least Squares to the Biaxial Test Problem
We progress from identifying 2 electrical parameters from noisy voltagecurrent measurements
to identifying 20 ber stiness in Figure 3.3 from noisy forcedisplacement measurements.
We envision loading the 9 nodes with a known force vectors, f R18 , and measuring the
associated 18 displacements, x. From knowledge of x and f we wish to infer the twenty components
of K = diag(k) where k is the vector of unknown ber stinesses. The rst, and most important,
step is to recognize that
AT KAx = f
may be written as
Bk = f

where B = AT diag(Ax).

(6.12)

Though conceptually simple this is not of great use in practice, for B is 18-by-20 and hence (6.12)
possesses many solutions. The way out (as in our previous example) is to conduct more experiments.
We will see that, for our small sample, 2 experiments will suce.
57

To be precise, we suppose that x(1) is the displacement produced by loading f (1) while x(2) is the
displacement produced by loading f (2) . We then piggyback the associated pieces in

 T
A diag(Ax(1) )
B=
AT diag(Ax(2) )


and f =


f (1)
.
f (2)

This B is 36-by-20 and so the system Bk = f is overdetermined and hence ripe for least squares.
We proceed then to assemble B and f . We suppose f (1) and f (2) to correspond to horizontal
and vertical stretching
f (1) = [1 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 1 0]T
f (2) = [0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 1 0 1]T
respectively. For the purpose of our example we suppose that each kj = 1 except k8 = 5. We
assemble AT KA as in Chapter 3 and solve
AT KAx(j) = f (j)
with the help of the pseudoinverse. In order to impart some reality to this problem we taint
each x(j) with 10 percent noise prior to constructing B. Please see the attached Mle for details.
Regarding
B T Bk = B T f
we note that Matlab solves this system when presented with k=B\f when B is rectangular. We
have illustrated the results of this procedure in Figure 6.3.
4.5
4

fiber stiffness

3.5
3
2.5
2
1.5
1
0.5
0

10

15

20

fiber number

Figure 6.3. Results of a successful biaxial test.

We see that even in the presense of noise that the sti ber is readily identied.
58

6.4. Projections
From an algebraic point of view (6.5) is an elegant reformulation of the least squares problem.
Though easy to remember it unfortunately obscures the geometric content, suggested by the word
projection, of (6.4). As projections arise frequently in many applications we pause here to develop
them more carefully.
With respect to the normal equations we note that if N (A) = {0} then
x = (AT A)1 AT b
and so the orthogonal projection of b onto R(A) is

Dening

bR = Ax = A(AT A)1 AT b.

(6.13)

P = A(AT A)1 AT ,

(6.14)

(6.13) takes the form bR = P b. Commensurate with our notion of what a projection should be we
expect that P map vectors not in R(A) onto R(A) while leaving vectors already in R(A) unscathed.
More succinctly, we expect that P bR = bR , i.e., P P b = P b. As the latter should hold for all b Rm
we expect that
(6.15)
P 2 = P.
With respect to (6.14) we nd that indeed
P 2 = A(AT A)1 AT A(AT A)1 AT = A(AT A)1 AT = P.
We also note that the P in (6.14) is symmetric. We dignify these properties through
Denition 6.2. A matrix P that satises P 2 = P is called a projection. A symmetric projection
is called an orthogonal projection.
We have taken some pains to motivate the use of the word projection. You may be wondering
however what symmetry has to do with orthogonality. We explain this in terms of the tautology
b = P b + (I P )b.
Now, if P is a projection then so too is (I P ). Moreover, if P is symmetric then the inner product
of bs two constituents is
(P b)T (I P )b = bT P T (I P )b = bT (P P 2 )b = bT 0b = 0,
i.e., P b is orthogonal to (I P )b.
As an example, for the A of (6.6) we nd

1 0 0
P = A(AT A)1 AT = 0 1 0 ,
0 0 0
as suggested by Figure 6.1. It is very useful to even develop expressions for the projection onto a
line. In this case A is the single column, let us denote it by a, and the associated projection matrix
is the scaled outer product
(6.16)
P = aaT /(aT a).
59

For example,



1 1 1
P =
2 1 1

(6.17)

is orthogonal projection onto the line through a = (1, 1). We illustrate this in Figure 6.4.
3

b
2.5

Pb

1.5

0.5

0
1

0.5

0.5

1.5

2.5

Figure 6.4. The projection of b = (1, 3) onto the line through a = (1, 1) via the projection
matrix, (6.17).
6.5. The QR Decomposition
While linear independence remains a central concept we shall see that representations are often
more concise and computations are always more robust when the relevant basis vectors are chosen
to be maximally independent, i.e., orthogonal. Projections are a natural means for transforming
a basis for a space M to an orthonormal basis for M.
This process, known as the GramSchmidt Procedure, takes n basis vectors, xj , for a subspace M and returns n orthonormal vectors, qj , in M.
GS1: Set y1 = x1 and q1 = y1 /y1 .
GS2: y2 = x2 minus the projection of x2 onto the line spanned by q1 . That is
y2 = x2 q1 (q1T q1 )1 q1T x2 = x2 q1 q1T x2 .
Set q2 = y2 /y2  and Q2 = [q1 q2 ].
GS3: y3 = x3 minus the projection of x3 onto the plane spanned by q1 and q2 . That is
y3 = x3 Q2 (QT2 Q2 )1 QT2 x3
= x3 q1 q1T x3 q2 q2T x3 .
Set q3 = y3 /y3  and Q3 = [q1 q2 q3 ].
Continue in this fashion through step
GSn: yn = xn minus its projection onto the subspace spanned by the columns of Qn1 . That is
yn = xn Qn1 (QTn1 Qn1 )1 QTn1 xn
= xn

n1


qj qjT xn .

j=1

60

Set qn = yn /yn  and Q = [q1 , q2 , . . . , qn ].

As the resulting Q Rmn has orthonormal columns it follows that QT Q = I. We call such a Q
an orthonormal matrix. It follows that if m = n then QT = Q1 .
To take a simple example, let us orthogonalize the following basis for R3 ,
x1 = [1 0 0]T ,

x2 = [1 1 0]T ,

x3 = [1 1 1]T .

(6.18)

GS1: q1 = y1 = x1 .
GS2: y2 = x2 q1 q1T x2 = [0 1 0]T , and so q2 = y2 .
GS1: y3 = x3 q1 q1T x3 q2 q2T x3 = [0 0 1]T , and so q3 = y3 .
We have arrived at the canonical basis
q1 = [1 0 0]T ,

q2 = [0 1 0]T ,

q3 = [0 0 1]T .

(6.19)

Once the idea is grasped the actual calculations are best left to a machine. Matlab accomplishes
this via the orth command. Its implementation is a bit more sophisticated than a blind run of
steps GS1n. As a result, there is no guarantee that it will return the same basis. For example,
here is a Matlab diary of orth applied to (6.19),
>> X=[1 1 1;0 1 1;0 0 1];
>> Q=orth(X)
Q =
0.7370 -0.5910 0.3280
0.5910 0.3280 -0.7370
0.3280 0.7370 0.5910
This ambiguity does not bother us, for one orthogonal basis is as good as another. In fact, we can
often get by just knowing that an orthonormal basis exists. For example, lets show that orthogonal
bases permit us to easily see the rank of a projection.
Proposition 6.3. If P = P 2 then tr(P ) = rank(P ).
Proof: We suppose that P Rnn and denote by r the rank of P . We suppose that {q1 , . . . , qr }
is an orthonormal basis for R(P ) and that {qr+1 , . . . , qn } is an orthonormal basis for N (P T ). By
the Fundamental Theorem we know that Q = [q1 , q2 , , qn ] is an orthonormal basis of Rn . As a
result QT Q = I, and so, recalling the product formula (1.14) for the trace function we nd
tr(QT P Q) = tr(QT QP ) = tr(IP ) = tr(P ).

(6.20)

It remains to show that the left hand side is r, the rank of P . Our choice of Q makes this a very easy
calculation. Note that the (i, j) element of QT P Q is the inner product of qi with the jth column of
P Q, i.e., with P qj . Now if j r we know that qj R(P ) and so P qj = qj . As Q is orthonormal
we know that qiT qj = ij . If instead j > r then qj N (P T ) and so P qj = 0 and so qiP qj = 0 for
every i. We can summarize our ndings in

1 if i = j r
(QT P Q)ij = qiT P qj =
0 otherwise.
As QT P Q is a diagonal matrix with r ones it follows that tr(QT P Q) = r.End of Proof.
61

A more concrete use for the GramSchmidt Procedure will follow from viewing it as a factorization of X. More precisely, we wish to interpret the procedure as expressing each xj as a linear
combination of the qi for i j. This is simple for j = 1, namely GS1 states
x1 = (q1T x1 )q1 .

(6.21)

x2 = (q1T x2 )q1 + x2 (q1T x2 )q1 q2 ,

(6.22)

Unpacking GS2 we next nd that

is indeed as a linear combination of q1 of q2 . The awkward norm term can be reduced by simply
taking the inner product of each side of (6.22) with q2 . As qiT qj = ij this yields
q2T x2 = x2 (q1T x2 )q1 
and so, in fact (6.22) takes the form
x2 = (q1T x2 )q1 + (q2T x2 )q2 .

(6.23)

We next continue this line of argument and nd that

xj = (q1T xj )q1 + (q2T xj )q2 + + (qjT xj )qj .
for each j up to n. That each xj is expressed in terms of qi for i j
decomposition. Namely, we recognize that (6.21), (6.23) and (6.24),
T
q1 x1 q1T x2
0
q2T x2 q2T x3

0
0
q3T x3
[x1 , x2 , . . . , xn ] = [q1 , q2 , . . . , qn ]
.
..
..
..
.
.
0

(6.24)

manifests itself in a triangular

when collected, take the form

q1T xn
q2T xn

q3T xn .
..
..
.
.
T
0 qn xn

This matrix of qj is of course the original Q from GramSchmidt. The remaining upper triangular
matrix is typically denoted R. As none of its diagonal elements may vanish, it is invertible. We
have now established
Proposition 6.4. If X Rmn has linearly independent columns then there exists an orthonormal
Q Rmn and a nonsingular upper triangular R Rnn such that
X = QR.

(6.25)

This result oers a potentially dramatic rewrite of the least squares problem, Ax = b. For recall
that if x = R(A) we instead must solve the normal equations, AT Ax = AT b. Of course we may
solve this via the LU factorization of Chapter 3. If we instead factor A = QR then the normal
equations become RT Rx = RT QT b. However, RT inherits its nonsingularity from R and so we may
multiply each side by (RT )1 and arrive at the reduced normal equations
Rx = QT b.

(6.26)

As R is upper triangular, this may be solved by a single sweep of back substitution, without ever
even having to construct AT A. We note that Matlab generates Q and R via its qr function and
that it indeed solves (6.26) when confronted with the least squares problem x=A\b.
62

To take a concrete example, with the A and b of (6.6) we nd

 


1 0
1
1
1
T
Q = 0 1 , R =
and Q b =
1
0 1
0 0
and so (6.26) takes the form

   

x1
1 1
1
,
=
0 1
1
x2

and x2 = 1 and x1 = 0 as expected.

6.6. Detecting Integer Relations
The previous section demonstrated the value of QR for resolving overdetermined systems. In
this section we will discuss its use in an underdetermined system that arises in a beautiful algorithm
for detecting integer relations between real numbers. We denote the set of integers by Z and the
lattice of integers in Rn by Zn . Given an x Rn we call a nonzero z Zn an integer relation for
x if
(6.27)
z T x = 0.
In the plane one can nd such relations by merely computing the greatest common divisor of the
scaled entries. For example, to nd z Z2 such that 3.1z1 + 0.04z2 = 0 we multiply by 100 to clear
fractions and arrive at 310z1 + 4z2 = 0. We then compute g = gcd(310, 4) = 2 and note that
z1 = 4/g = 2 and z2 = 310/g = 155
provide an integer relation. The gcd is computed via a classic division algorithm that rst appeared
in Euclid. Attempts to generalize this to higher dimensions have only recently succeeded with
the resulting algorithm recognized as one of the ten best of the 20th century, and its application in
the right hands has detected some amazing new patterns - opening for example new vistas on .
We will use it to uncover the integers in putative functional relations like
z1 sin(5) + z2 sin() + z3 sin3 () + z4 sin5 () = 0.

(6.28)

The algorithm is known as PSLQ where PS stands for Partial Sums and LQ for the the Lower Triangular Orthogonal Decomposition, a transposed variant of QR. We have found it more convenient
to present the method as PSQR.
First, regarding partial sums, we assume without loss that x = 1 and no xj = 0 (otherwise
there is an obvious integer relation), build
s2j

x2k .

k=j

and use these to populate the (n 1) n upper triangular matrix

s2 x2 x1 x3 x1
xn x1

s1
s1 s2
s1 s2
s1 s2
s3
x3 x2
xn x2

s2
s2 s3
s2 s3

..
..

..
..
.
.
Ux =
.
.
.

sn1
xn1 xn2
xn xn2

sn2
sn1 sn2
sn1 sn2
sn
sn1

63

xn xn1
sn sn1

(6.29)

This matrix enjoys two lovely identities

and UxT Ux = In xxT

Ux UxT = In1

(6.30)

on which the entire algorithm depends. To establish the rst we rst examine the diagonal term
Ux (i, :)Ux (i, :)T = (si+1 /si )2 +

1
(si+1 si )2

+ x2i

x2k =

k=i+1

s2i+1 x2i
+ 2 = 1.
s2i
si

Regarding the odiagonal terms we suppose, without loss, i < j and nd

Ux (i, :)Ux (j, :)T =

n

xj xi sj+1
xi xj
xj xi sj+1 xi xj sj+1
+
x2k =
+
= 0.
si si+1 sj
si si+1 sj sj+1 k=j+1
si si+1 sj
si si+1 sj

This proves the rst identity in (6.30). The second may be proven in much the same way and is
left as an exercise. An immediate consequence of the second is that
Px UxT Ux

(6.31)

is orthogonal projection onto the orthogonal complement to the line through x, and
Px z = z

for any integer relation z Zn for x.

(6.32)

With this preparation we may establish

Proposition 6.5. Suppose that A Znn is invertible and that Ux A = QR where Q R(n1)(n1)
is orthogonal, R R(n1)n is upper triangular with no zeros on its diagonal, and Ux is given in
(6.29). If z is an Integer Relation for x then
1 zRmax

(6.33)

where Rmax is the magnitude of the largest diagonal element of R.

Proof: From Ux A = QR we deduce Px A = UxT Ux A = UxT QR. So if z is an Integer Relation for x
then by (6.31) and (6.32) it follows that z T = z T Px and z T A = z T Px A = z T (UxT Q)R. Now, as A is
invertible z T A = 0. Let j be the least j for which z T A:,j = 0, so z T A:,k = 0 for k < j. If j = 1 skip
to (6.34). If j > 1 note that, as R is upper triangular,
0 = z T A:,1 = z T (UxT Q)R:,1 = r1,1 z T (UxT Q):,1 ,
and, as r1,1 = 0 we conclude that z T (UxT Q):,1 = 0. Continuing in this fashion,
0 = z T A:,2 = z T (UxT Q)R:,2 = r1,2 z T (UxT Q):,1 + r2,2 z T (UxT Q):,2 .
In this way we see that z T (UxT Q):,k = 0 for each k < j and so
z T A:,j = rj,j z T (UxT Q):,j .
Finally, as the left side of (6.34) is a nonzero integer and each (UxT Q):,j  = 1 we nd
1 |z T A:,j | = |z T (UxT Q):,j rj,j | |rj,j |z(UxT Q):,j  = |rj,j |z Rmax z
as a consequence of the CauchySchwarz inequality. End of Proof.
64

(6.34)

This result sets the stage for a tractable, constructive means for identifying an integer relation
or for showing that one is improbable, i.e., of astronomically big norm. The idea is to successively
choose the free integer matrix A so to decrease Rmax (and so increase the size of any permissible
integer relation) while monitoring the diagonal of R for zeros which then reveal integer relations in
the row(s) of A1 .
The decrease in Rmax is achieved through a simple integer variation on Gaussian Elimination. It
begins, given an upper triangular R R(n1)n , by generating an upper triangular, unit diagonal,
D Rnn such that RD is diagonal and diag(R) = diag(RD). We build this elimination matrix
via successive construction of its superdiagonals
di,i = 1,

di,i+1 = ri,i+1 /ri,i ,

di,i+2 = (ri,i+1 di+1,i+2 + ri,i+2 di+2,i+2 )/ri,i ,

and in general
di,i+j =

j
1 
ri,i+k di+k,i+j .
ri,i k=1

(6.35)

For example, if


1 2 3
R=
0 4 5

1 2 1/2
then D = 0 1 5/4
0 0
1



1 0 0
and RD =
.
0 4 0

As we will use D to transform A we must respect its integrality. This is accomplished by sequentially
rounding to the nearest integer in our super-diagonal construction. That is we replace (6.35) with


j
1 
di,i+j = round
(6.36)
ri,i+k di+k,i+j .
ri,i k=1
On application to the small example above,

1 2 1
D = 0 1 1
0 0
1


and RD =


1 0 0
.
0 4 1

Although RD is not necessarily diagonal, we can still prove

|(RD)i,j | |rj,j |/2.

(6.37)

The construction of D from R via (6.36) is known as Hermite Reduction and as such we will denote
it D = Hred (R). We now have all of the pieces required of the
Integer Relation Algorithm

Initialize. Given x Rn build Ux per (6.29). Set U = Ux and A = In and choose w > 2/ 3.
While neither x nor diag(U) possess any zero elements
1. Reduce: Compute D = Hred (U) and set
x = D 1 x,

U = UD
65

and A = AD.

2. Exchange: Let r be such that w j |uj,j | is maximal for j = r. Let I be the elementary
perturbation of In where columns r and r + 1 are swapped, and set

x = Ix,

U = U I and A = AI.

If r = n 1 then U remains upper triangular and we return to top of while. Else,

3. Fix: Construct F = In1 except the 2 2 block




1
ur,r
fr,r+1
ur+1,r
fr,r
= 2
.
fr+1,r fr+1,r+1
(ur,r + u2r+1,r )1/2 ur+1,r ur,r
Set U = F U and return to top of while.
end of While
We see from the exchange step that w is a weight that can assist the motion of large diagonal elements down the diagonal and eventually o the diagonal and into column n. We have implemented
this algorithm in psqr. For example, with x = (1.5, 2.3, 3.2)T and = 3 psqr returns

4 4 1
A1 = 9 17 8
1 2 1
and we recognize two integer relations for x in its rst two rows.
6.7. Autoregressive Models and Levinsons Algorithm
We investigate a classic least squares problem from random signal theory. We suppose that
x RN is drawn from a zeromean second order stationary process. That is,
mean(xj ) = 0 and mean(xj xj+ ) c = c

(6.38)

and attempt to t it to an autoregressive model of order L,

xj =

a xj

(6.39)

=1

where L is typically much less than N. We use (6.38) to arrive at a consistent and deterministic
system for a RL . Multiplying each side of (6.39) by xj+1 and taking means
mean(xj+1 xj ) =

a mean(xj+1 xj ),

reveals c1 =

=1

a c1 ,

(6.40)

=1

while multiplying (6.39) by xj+k and taking means reveals

ck =

a ck .

(6.41)

=1

All together we arrive at the system

CL a = c1:L
66

(6.42)

where c1:L = (c1 , c2 , . . . , cL )T and

c0
c1
CL =
...

cL1
cL2
..
.

c1
c0
..
.

cL1 cL2

c0

The system (6.42) is known as the YuleWalker equations for the autoregressive model, (6.39).
In order to develop some intuition about the behavior of such models we simulate and analyze the
rst order process
xj = a1 xj1 + j ,
(6.43)
where j is drawn from the Gaussian distribution with density (6.9). Squaring and taking means of
each side reveals
c0 = mean(x2j ) = mean(a21 x2j1 + 2a1 xj1 j + 2j ) = a21 c0 + 0 + 2 ,
and so,
c0 =

2
.
1 a21

(6.44)

In a similar fashion,
ck = mean(xj xj+k ) = mean(xj (a1 xj+k1 + j+k )) = a1 ck1 = a21 ck2 = = ak1 c0 .

(6.45)

We illustrate in Figure 6.5 the processes and their autocovariances when a1 = 0.95 and = 1.
N = 10000
exact

Autocovariance, AR(1), a1 = 0.95

Sample path, AR(1), a = 0.95

10
8
6
4
2
0
2
4
6
8

10
4000

10

4100

4200

4300

4400

10

4500

20

30

40

50

60

70

80

Autocovariance, AR(1), a1 = 0.95

10

Sample path, AR(1), a = 0.95

90

100

lag

index

10

10

N = 10000
exact

8
6
4
2
0
2
4
6
8
10

4000

4100

4200

4300

4400

10

4500

20

30

40

50

lag

index

67

60

70

80

90

100

Figure 6.5 Sample paths and exact and empirical autocovariances for the rst order model, (6.43),
with = 1.
We next address the most parsimonious choice of model order, L. To distinguish the model
orders we denote the solution to (6.39) at order L by a(L) and the associated least squares error by

2
N
L
 (L)
1 
(L)
mean xj
a xj = c0 cT1:L a(L) .
N j=1
=1
The competing objectives are to render both the error, (L), and the model order, L, small. We
accomplish this, following (Akaike69), by minimizing the Relative Final Prediction Error
RF P E(L) =

N + L + 1 (L)
.
N L 1 c0

(6.46)

The search for the minimum is eased by the fact that we may solve for a(L) via a fast and cheap
update of a(L1) . The resulting recursive algorithm was rst discovered in this context by Norman
Levinson. We follow [Ammar Gragg] and derive it as a recursive inverse Cholesky factorization of
CL ,


RL1 rL1
T
RL CL RL = DL , where DL = diag (d1:L ) and RL =
(6.47)
0
1
as a consequence of the persymmetry of CL , i.e.,
EL CL EL = CL
where EL = [eL eL1 e1 ] is the exchange matrix obtained by exchanging, or reversing, the
columns of the L by L identity matrix. We construct the L = 2 factors by hand




1 c1 /c0
c0
0
and D2 =
R2 =
0
1
0 c0 c21 /c0
and proceed to establish the general case.
Proposition 6.6. Commencing from r1 = c1 /c0 , d2 = c0 c21 /c0 and a(1) = c1 /c0 and k = 1, we
chart the Levinson recurrence relations
k+1 = (cT1:k rk + ck+1 )/dk+1
 (k) 
 
r
a
(k+1)
a
=
k+1 k
1
0
 


0
1
rk+1 =
+ k+1
rk
Ek rk

(6.48)

2
)dk+1.
dk+2 = (1 k+1

Proof: To set up the recursion we recall that c1:k = (c1 , c2 , . . . , ck )T and express the growth of
our Toeplitz matrix in terms of the associated exchange matrix:
 


c0 cT1:k
Ek c1:k
Ck
=
(6.49)
Ck+1 = T
c1:k Ek
c0
c1:k Ck
68

and the growth of its associated Cholesky factor via





Rk rk
k+1
, rk+1 =
Rk+1 =
.
0 1
sk

(6.50)

To begin, we use the rst representation of Ck+1 in (6.49) and equate last columns in Ck+1Rk+1 =
T
Rk+1
Dk+1 , nding
(6.51)
Ck rk + Ek c1:k = 0
and
cT1:k Ek rk + c0 = dk+1.

(6.52)

The former becomes, on using persymmetry Ck = Ek Ck Ek and multiplying across by EkT ,

Ck Ek rk + c1:k = 0.

(6.53)

We next increment k by 1 in (6.51), nding Ck+1 rk+1 + Ek+1 c1:k+1 = 0. We unpack this using (6.49)
(2nd part) and (6.50) and nd
k+1c0 + cT1:k sk + ck+1 = 0
(6.54)
and
k+1 c1:k + Ck sk + Ek c1:k = 0.

(6.55)

Subtracting (6.51) from (6.55) and using (6.53) we arrive at

This completes the expression of

rk+1

sk = rk + k+1 Ek rk .

(6.56)

 


0
1
=
+ k+1
,
rk
Ek rk

(6.57)

which is precisely the third statement in (6.48). Using (6.56) in (6.54) we nd

c0 k+1 + cT1:k (rk + Ek rk k+1) + ck+1 = 0.
Applying (6.52) here reveals the rst formula in (6.48). This formula, together with (6.52) (after
incrementing k) and (6.57) yield
 
r
T
T
T
2
)dk+1 ,
dk+2 = c0 + c1:k+1Ek+1 rk+1 = c0 + c1:k Ek rk + c1:k+1 k k+1 = (1 k+1
1
the fourth piece of the stated recursion. Finally,
1
RT c1:k+1
a(k+1) = Rk+1 Dk+1
 k+11

 T 
Rk rk
Rk 0
Dk
0
=
c
0 1
rkT 1 1:k+1
0 1/dk+1



Rk rk
Dk1 RkT c1:k
=
0 1
(rkT c1:k + ck+1 )/dk+1
  (k) 
 

r
Rk Dk1 RkT c1:k k+1 rk
a
k+1 k
=
=
1
k+1
0

lls in the last part of the recursion. End of Proof.

69

We illustrate its performance in identifying a 10th order model in Figure 6.6.

0.6

12

1.1

10
0.4

1
8

0.2

0.4

0.9

6
4

RFPE

autocovariance, c

0.2

2
0
2

0.8

0.7

0.6

4
0.6

0.5
6

0.8

100

200

300

index

400

500

10

15

20

25

30

0.4

lag, L

10

15

20

25

30

Figure 6.6 (A) A sample path of a 10th order process. (B) Its empirical autocovariance. (C)
The relative nal prediction error. AkaLev.m
6.8. Exercises
1. A elastic cable was stretched to lengths  = 6, 7, and 8 feet under applied forces of f = 1, 2,
and 4 tons. Assuming Hookes law  L = cf , nd its compliance, c, and original length, L,
by least squares. In particular
(i) Formulate the question as Ax = b, with numerical values in A and b. What does x signify?
(ii) Solve the normal equations, AT Ax = AT b by hand for x.
(iii) Graph the three data points in the (f, ) plane as well as the straight line t corresponding
to the x found in (ii).
2. With regard to the example of 6.3 note that, due to the the random generation of the noise
that taints the displacements, one gets a dierent answer every time the code is invoked.
(i) Write a loop that invokes the code a sucient number of times (until the averages settle
down) and submit bar plots of the average ber stiness and its standard deviation for each
ber, along with the associated Mle.
(ii) Experiment with various noise levels with the goal of determining the level above which it
becomes dicult to discern the sti ber. Carefully explain your ndings.
3. Find the matrix that projects R3 onto the line spanned by [1 0 1]T .
4. Find the matrix that projects R3 onto the plane spanned by [1 0 1]T and [1 1 1]T .
5. If P is the projection of Rm onto a kdimensional subspace M, what is the rank of P and what
is R(P )?

6. Show that if P is an othogonal projection onto a subspace of dimension r then P F = r.

7. Show that if P is a projection then so too are P T and I P .
8. Show that the only invertible projection is the identity matrix.
9. Not all projections are symmetric. Please conrm that



1
0
0
1 0
0
0
and 1/2
1 0
1/4 1/2 1
70

are projections. Sketch the column space of the rst, and depict the oblique projection of
b = (1, 3)T onto this space. How does your sketch dier from Figure 6.4?
10. Show that orthonormal matrices stretch no vector, i.e., if QT Q = I then Qx = x for all x.
11. Show that if Q is a square orthonormal matrix then det(Q) = 1.
12. Use the GramSchmidt procedure, by hand (dont even use a calculator, for they replace
pregnant square roots with chaotic decimals), to compute orthonormal bases for the four
fundamental subspaces of

1 2 5 3
3 1 5 4

A=
(6.58)
2 1 0 1 .
1 2 5 3
13. Construct, by hand, the QR decomposition of

1 1
A = 2 1
4 1
and explain its relationship to our rst exercise.
14. Establish the second identity in (6.30). Recall that x = 1 and then deduce, with little work,
that Ux x = 0.
15. Conrm that (6.37) is a consequence of the fact that the largest rounding error is 1/2. That
is, |t round(t)| 1/2 for every t R. Here is how to start in one concrete case
(RD)1,2 = r1,1 d1,2 + r1,2
= r1,1 round(r1,2 /r1,1 ) + r1,2
.
= r1,1 round(r1,2 /r1,1 ) + r1,2 + r1,1 (r1,2 /r1,1 ) r1,1 (r1,2 /r1,1 )
= r1,1 (round(r1,2 /r1,1 ) (r1,2 /r1,1 ))
Finish this argument and then extend it to the rest of the superdiagonals.
16. Complete the integer relation (6.28) by invoking psqr at many random values of .
17. The statistical modeling of 6.7 is precisely that needed to develop the nite Wiener Filter.
Here we suppose we have measurements, yk , of a true signal, xk , contaminated by additive
noise, k , i.e.,
yk = xk + k .
We suppose known the two autocovariances, cyy and cxx , and the covariance, cxy , and we seek
the nite lter a for which
k

akm ym
xk =
m=kN

71

minimizes mean((xk xk )2 ). Show that the

cyy (0) cyy (1)

..
..
.
.
cyy (1)
.
.
.
..
..
..

..
.
cyy (N)
cyy (1)

best a obeys the YuleWalker equations

cyy (N) a c (0)

0
xy
..
. a1 cxy (1)
. = .
.. ..
cyy (1)

cxy (N)
aN
cyy (0)

18. Let us extend the Levinson Algorithm of 6.7 to the multivariate setting, where X is drawn
from an nvalued zeromean second order stationary process. That is, each Xj Rn ,
T
mean(Xj ) = 0 and mean(Xj Xj+k
) = Ck

(6.59)

and so Ck Rnn . We t X to an autoregressive model of order L,

Xj =

A Xj

(6.60)

=1

where each A R .
(i) By taking products and means of (6.60) when L = 2 derive the associated YuleWalker
equations





C0 C1
(2)
(2)
= C1 C2
A1 A2
C C
nn

and proceed to construct the explicit block Cholesky factorization



 


I
0
C0
0
C0 C1
I C01 C1
=
.
C1 C0
C1 C01 I
0
I
0 C0 C1 C01 C1
(ii) In general, for a model of order k, derive

C0
C1 Ck1
C0 Ck2
(k)
(k)
(k) C1
= (C1 C2 Ck )
(A1 A2 Ak )
..
..
.
..
.
.
Ck1 Ck2 C0
and express it as
A(k) Ck = C1:k .
(iii) Conrm the persymmetry Enk Ck Enk = Ck , adopt the convention




Rk Rk
k+1
Rk+1 =
, Rk+1 =
.
0 In
Sk

(6.61)

(6.62)

and establish the multivariate Levinson algorithm: Commencing from R1 = C1 C01 , D2 =

C0 C1 C01 C1 and A(1) = C1 C01 and k = 1,
1
k+1 = Dk+1
(C1:k Rk + Ck+1 )

 T


1
Rk In
A(k+1) = A(k) 0 Dk+1 k+1 Dk+1
  

0
In
Rk+1 =
+
k+1
Enk Rk
Rk

Dk+2 = Dk+1 (In 2k+1 ).

72

(6.63)

7. Matrix Methods for Metabolic Networks

Metabolism, the conversion of food into energy, can be seen to function in three stages. In
the rst, fats, polysaccharides and proteins are broken down into fatty acids and glycerol, glucose
and other sugars, and amino acids. In the second stage these metabolites are largely converted
into acetyl units of acetyl CoA. In the third stage Acetyl CoA brings acetyl into the Citric Acid
Cycle. With each turn of the cycle an acetyl group is oxidized and the associated ow of electrons
is harnessed to generate ATP, lifes principal currency of energy.
We derive and solve linear programming problems stemming from ux balance subject to thermodynamic constraints.
7.1. Flux Balance and Optimal Yield
In models of metabolism the concentration of individual metabolites is governed by the reactions
in which they participate. Although metabolites undergo signicant conversion we assume that the
net flux of each metabolite is zero. In other words, nothing is lost (or gained) in its conversion.
As there are typically many more reactions than metabolites (in E. Coli the ratio is ten to one)
the modeler is typically faced with an underdetermined system of equilibrium equations. As such
there is typically a large class of balanced ux distributions. Scientists and engineers have recently
turned this freedom to their advantage by directing ow down pathways that deliver an optimal (or
at least improved) yield of a desired product.
To illustrate this we consider the network of Figure 7.1 and suppose that we have a steady ow
into metabolite 1 and wish to accumulate as much as possible of metabolite 6.
1
m1

v3
v1
m4

v6

m2

v2
m3

v5

v7

v4

m5

m6

Figure 7.1. A metabolic network

There are many pathways from m1 to m6 and our task is discover or design one that produces
the greatest yield. Along the way we must obey ux balance and energy considerations. The former
simply means that ux into an internal metabolite must balance ux out and that each of the uxes
has an energetically preferred direction.
73

More precisely, Figure 7.1 indicates that m1 is supplied with rate one and then
m1 + m2
m1
m1 + m3
m5
m3
m2
m2 + m4
m3 + m5

m4 at rate v1
m2 + m3 at rate v2
m5 at rate v3
m3 at rate v4
m2 at rate v5
m4 at rate v6
m6 at rate v7
m6 at rate v8 .

(7.1)

To balance the ux of mj we balance its total out and total in rates. Its total out rate is the
sum of the reaction rates in which it appears as a reactant (on the left), while its total in rate
is the sum of the reaction rates in which it appears as a product (on the right). By this logic, the
reaction scheme of (7.1) gives rise to this system of ux balances,
v1 + v2 + v3 = 1
(v1 + v6 + v7 ) (v2 + v5 ) = 0
(v3 + v5 + v8 ) (v2 + v4 ) = 0
v7 (v1 + v6 ) = 0
(v4 + v8 ) v3 = 0.

(7.2)

Please note that we have balanced the ux of only the rst 5 metabolites, for our goal is to in fact
upset the balnce of m6 is our favor. It is now a simple manner to translate (7.2) into the matrix
equation Sv = b where

1
1
1
0
0
0 0 0
1
1 1 0

0
0
1
1
1
0

and b = 0 .
0
1
1
1
1
0
0
1
S=

1 0
0
0
0
0 1 1 0
0
0 1 1
0
0 0 1
0
While ux balance has been reduced to Sv = b we note that each of our reactions is typically
catalyzed by an enzyme with prescribed reactants and products. To say that we cannot interchange
these is to say that the reaction is not reversible and this in turn implies that each reaction rate has
a sign. For the sake of deniteness we suppose each vj 0. As the ux (or yield) of the desidered
(sixth) metabolite is v7 + v8 we may pose our design problem as
max(v7 + v8 ) where P {v R8 : Sv = b, v 0}.
vP

(7.3)

The maximization of a linear combination of variables subject to both linear and sign constraints
is called a Linear Programming problem.
7.2. Linear Programming
It seems natural to write the last two equations in Sv = b as
v7 = v1 + v6

and v8 = v3 v4
74

and to therefore reduce Sv = b to

v1 + v2 + v3 = 1
2v1 v2 v5 + 2v6 = 0
v2 + 2v3 2v4 + v5 = 0

(7.4)

Which has the general solution Sv = b is

v1 = (1/4) (1/4)v4 + (1/2)v5 (3/4)v6
v2 = (1/2) (1/2)v4 + (1/2)v6
v3 = (1/4) + (3/4)v4 (1/2)v5 + (1/4)v6

(7.5)

y = v1 + v3 v4 + v6 = 1/2 (1/2)v4 + (1/2)v6.

(7.6)

and associated yield

The easiest way to guarantee that all rates are nonnegative is to set each of the free rates to zero.
This gives us the simple starting solution
v = (1/4 1/2 1/4 0 0 0)

(7.7)

and the corresponding yield of 1/2. To improve the yield we note (with respect to (7.6)) that
increasing v6 is the only good idea (for it is the only variable with a positive weight). But how far
can we go? We note that, with v4 = v5 still zero that v1 0 requires v6 (1/3) (and that positivity
of v2 and v3 provide no upper bounds on v6 ). Hence we set v6 = 1/3 and note that
v1 = 0,

v2 = 2/3 and v3 = 1/3,

brings an increased yield of y = 2/3. We can interpret the setting of v1 = 0 to imply that {v1 , v2 , v3 }
was not the best choice of basic variables. Hence, we might be better served by a system that
expresses the new basic variables {v2 , v3 , v6 } in terms of the new free variables {v1 , v4 , v5 }. Well
v6 = (1/3) (4/3)v1 (1/3)v4 + (2/3)v5
v2 = (2/3) (2/3)v1 (2/3)v4 + (1/3)v5
v3 = (1/3) (1/3)v1 + (2/3)v4 (1/3)v5

(7.8)

is such a system. As its yield is

y = v1 + v3 v4 + v6 = (2/3) v1 (2/3)v4 + (1/3)v5 ,
we consider increasing v5 . Its only bound is v5 1 and at this bound (with v1 = v4 = 0) we nd
v6 = 1,

v2 = 1 and v3 = 0

and an improved, optimal, yield of y = 1. To conrm optimality we attempt another step. We

express the new basic variables {v2 , v5 , v6 } in terms of the new free variables {v1 , v3 , v4 }. Well
v6 = 1 2v1 2v3 + v4
v2 = 1 v1 v3
v5 = 1 v1 3v3 + 2v4
is such is system. As its yield
y = v1 + v3 v4 + v6 = 1 v1 v3 ,
75

(7.9)

has no rates with positive weights we conclude that no growth is possible.

7.3. The Simplex Method
We now attempt to formalize these steps. The step from (7.4) to (7.5) begins by row reduction of
Sv = b and subsequent identication of the basic, or pivot, variables indexed by B, and nonbasic,
or free, variables indexed by N. This permits us to express Sv = b as S(:, B)v(B)+S(:, N)v(N) = b,
and to solve for the basic in terms of the nonbasic
v(B) = S(:, B)\(b S(:, N)v(N)).

(7.10)

With B = (1, 2, 3) and N = (4, 5, 6) this takes the form

v1
1 1 1
1
0 0
0
v4
v2 = 2 1 0 0 0 1 2 v5
0 1 2
0
2 1 0
v3
v6
and this is precisely (7.5). With f = (1, 0, 0, 0, 0, 0)T we can now express the yield
y = f T v = f T (B)v(B) + f T (N)v(N)
= f T (B)S(:, B)\b + (f T (N) f T (B)S(:, B)\S(:, N))v(N),
as exactly in (7.6). Setting vN = 0 gives us our initial iterate
v (1) = (1/4, 1/2, 1/4, 0, 0, 0).
Now to increase the yield we scan the vector
[val, ind] = max(f T (N) f T (B)S(:, B)\S(:, N))

(7.11)

for the most positive number. If val 0 we are done. If val > 0 and N(ind) = j then we set
vj = max{vj : v(B) 0, v(N \ j) = 0}.

(7.12)

The basic variable, vk , that vanishes in the process then exchanges places with vj . That is, j moves
from N to B and k moves from B to N. In our example, j = 6 and k = 1 and so our second iterate
is
v (2) = (0, 2/3, 1/3, 0, 0, 1/3).
One then returns to (7.10) with the new B and N and then (7.11) and (7.12) until val 0.
1. Solve SBT y = fB .
2. Choose an entering column, a = SN (:, e), by requiring that y T a < fN (e).
3. Solve SB d = a.
4. Find the largest t such that vB td 0. The variable for which equality holds will now leave.
5. Set the value of the entering variable to t and replace the values vB of the basic variables by
vB td 0. Replace the leaving column of SB by the entering column.
76

These steps, when coupled with a procedure for initializing the index sets B and N, comprise
the Simplex Algorithm.
7.4. The Geometric Point of View
The nal iterate is
v (3) = [0 1 0 0 1 1 1 0]T

(7.13)

Can you see it? One way is to attempt to decompose P. We will do this with the help of S + , the
pseudoinverse of S, and E, an m-by-q matrix whose columns comprise a basis for N (S). Now
P {v = S + b + Ey : y Pred }
where
Pred = {y Rq : Ey S + b}
In this light I ask that you see that Pred is the intersection of half spaces (for each inequality in
Ey S + b holds on one side of the hyperplane where Ey = S + b). In the case of Figure 7.1 we
nd

1 1
0
5
0
6
1
1

1 2 1
5

1 2
1 2 2
+
S b=

and E =
1 1
0
16 0
1
2
0
0

0
3
1
0
0
0
1
3
and so Pred may be seen with our eyes (Figure 7.2). Do you recognize it as the intersection of
halfspaces? Where 2 of these intersect we have an edge. Where 3 intersect we nd a vertex.
With the reduced polytope representation also comes the reduced cost,
f T v = f T (S + b + Ey) = f T S + b + y T E T f.
The rst term in the last expression is completely determined by the network, while the latter is
determined by the y values in Pred . For the problem at hand
y T E T f = y1 + y2 .
Do you see that this achieves its maximum on Pred at precisely the rst vertex
y=

1
[18 13 3]T
16

and do you see that this gives the v in (7.13)?

77

The Reduced Polytope

0.2

0.2

0.1

0.1

The Reduced Polytope

0.1

5
4

0.1

0.2
0

0.2
0

1
0.5

0.5

1.5

y1

0.5

1
0.5

0.5

y2

1.5

0.5

Figure 7.2. The reduced polytope associated with the network in gure 7.1 with hidden lines
hidden (left) and revealed (right). We have labeled the 5 vertices in the right panel.
7.5. Exercises

78

8. Matrix Methods for Dynamical Systems

Up to this point we have largely been concerned with (i) deriving linear systems of algebraic
equations from considerations of static equilibrium and (ii) the solution of such systems via Gaussian
elimination.
In this section we hope to begin to persuade the reader that our tools extend in a natural fashion
to the class of dynamic processes. More precisely, we argue that (i) Matrix Algebra plays a central
role in the derivation of mathematical models of dynamical systems and that, with the aid of the
Laplace transform in an analytical setting or the Backward Euler method in the numerical setting,
(ii) Gaussian elimination indeed produces the solution.
8.1. Neurons and the Dynamic Strang Quartet
A nerve bers natural electrical stimulus is not direct current but rather a short burst of current,
the socalled nervous impulse. In such a dynamic environment the cells membrane behaves not
only like a leaky conductor but also like a charge separator, or capacitor.
Ri
x1

y2
y1

Ri

x2

y6

y3
Rm

Cm

x3

Rm

Rm
i0

Cm

y4

y7
y5

Em

Cm
y8

Em

Em

Figure 8.1. An RC model of a neuron.

The typical value of a cells membrane capacitance is
c = 1 (F/cm2 )
where F denotes microFarad. The capacitance of a single compartment is therefore
Cm = 2a(/N)c
and runs parallel to each Rm , see Figure 8.1. We ask now how the static Strang Quartet of chapter
one should be augmented. Regarding (S1) we proceed as before. The voltage drops are
e1 = x1 , e2 = x1 Em , e3 = x1 x2 ,
e5 = x2 Em , e6 = x2 x3 , e7 = x3 ,
and so

0
1

0

0
e = b Ax where b = Em
1
0

0
1
79

e4 = x2 ,
e8 = x3 Em ,

1 0
0
1 0
0

0
1 1

0 1 0
and A =

0 1 0
0 1 1

0
0 1
0
0 1

In (S2) we must now augment Ohms law with voltagecurrent law obeyed by a capacitor, namely
the current through a capacitor is proportional to the time rate of change of the potential across
it. This yields, (denoting d/dt by  ),
y1 = Cm e1 , y2 = e2 /Rm ,
y5 = e5 /Rm , y6 = e6 /Ri ,
or, in matrix terms,
where

y3 = e3 /Ri ,
y7 = Cm e7 ,

y4 = Cm e4 ,
y8 = e8 /Rm ,

y = Ge + Ce
G = diag(0 Gm Gi 0 Gm Gi 0 Gm )
C = diag(Cm 0 0 Cm 0 0 Cm 0)

are the conductance and capacitance matrices.

As Kirchhos Current law is insensitive to the type of device occupying an edge, step (S3)
proceeds exactly as above.
i0 y1 y2 y3 = 0 y3 y4 y5 y6 = 0 y6 y7 y8 = 0,
or, in matrix terms,
AT y = f

where f = [i0 0 0]T .

Step (S4) remains one of assembling,

AT y = f AT (Ge + Ce ) = f AT (G(b Ax) + C(b Ax )) = f,
becomes

AT CAx + AT GAx = AT Gb + f + AT Cb .

(8.1)

This is the general form of the potential equations for an RC circuit. It presumes of the user
knowledge of the initial value of each of the potentials,
x(0) = X.

(8.2)

Regarding the circuit of Figure 8.1 we nd

0
Gi
0
Cm 0
Gi + Gm
2Gi + Gm
Gi
AT CA = 0 Cm 0 AT GA = Gi
0
Gi
Gi + Gm
0
0 Cm


Gm
0
AT Gb = Em Gm and AT Cb = 0 .
Gm
0
and an initial (rest) potential of
x(0) = Em [1 1 1]T .
We shall now outline two modes of attack on such problems. The Laplace Transform is an analytical
tool that produces exact, closedform, solutions for small tractable systems and therefore oers
insight into how larger systems should behave. The BackwardEuler method is a technique for
solving a discretized (and therefore approximate) version of (8.1). It is highly exible, easy to code,
and works on problems of great size. Both the BackwardEuler and Laplace Transform methods
80

require, at their core, the algebraic solution of a linear system of equations. In deriving these
methods we shall nd it more convenient to proceed from the generic system
x = Bx + g.

(8.3)

B = (AT CA)1 AT GA

(Gi + Gm )
Gi
0
1

Gi
(2Gi + Gm )
Gi
=
Cm
0
Gi
(Gi + Gm )

(8.4)

With respect to our ber problem

and

Em Gm + i0
1
Em Gm .
g = (AT CA)1 (AT Gb + f ) =
Cm
Em Gm

8.2. The Laplace Transform

The Laplace Transform is typically credited with taking dynamical problems into static problems.
Recall that the Laplace Transform of the function h is
!
(Lh)(s)
exp(st)h(t) dt.
0

where s is a complex variable. We shall soon take a 2 chapter dive into the theory of complex
variables and functions. But for now let us proceed calmly and condently and follow the lead of
Matlab . For example
>> syms t
>> laplace(exp(t))
ans = 1/(s-1)
>> laplace(t*exp(-t))
ans = 1/(s+1)2
The Laplace Transform of a matrix of functions is simply the matrix of Laplace transforms of
the individual elements. For example

 

exp(t)
1/(s 1)
L
=
.
t exp(t)
1/(s + 1)2
Now, in preparing to apply the Laplace transform to (8.3) we write it as
Lx = L(Bx + g)

(8.5)

and so must determine how L acts on derivatives and sums. With respect to the latter it follows
directly from the denition that
L(Bx + g) = LBx + Lg = BLx + Lg.
81

(8.6)

Regarding its eect on the derivative we nd, on integrating by parts, that

"
!
!
"


"
Lx =
exp(st)x (t) dt = x(t) exp(st)" + s
exp(st)x(t) dt.
0

Supposing that x and s are such that x(t) exp(st) 0 as t we arrive at

Lx = sLx x(0).

(8.7)

Now, upon substituting (8.6) and (8.7) into (8.5) we nd

sLx x(0) = BLx + Lg,
which is easily recognized to be a linear system for Lx, namely
(sI B)Lx = Lg + x(0).

(8.8)

The only thing that distinguishes this system from those encountered since chapter 1 is the presence
of the complex variable s. This complicates the mechanical steps of Gaussian Elimination or the
GaussJordan Method but the methods indeed apply without change. Taking up the latter method,
we write
Lx = (sI B)1 (Lg + x(0)).
The matrix (sI B)1 is typically called the resolvent of B at s. We turn to Matlab for its
symbolic calculation. For example,
>> syms s
>> B = [2 -1;-1 2]
>> R = inv(s*eye(2)-B)
R =
[ (s-2)/(s*s-4*s+3), -1/(s*s-4*s+3)]
[ -1/(s*s-4*s+3), (s-2)/(s*s-4*s+3)]

(8.9)

We note that (sI B)1 is well dened except at the roots of the quadratic, s2 4s + 3. This
quadratic is the determinant of (sIB) and is often referred to as the characteristic polynomial
of B. The roots of the characteristic polynomial are called the eigenvalues of B. We will develop
each of these new mathematical objects over the coming chapters. We mention them here only to
point out that they are all latent in the resolvent.
As a second example let us take the B matrix of (8.4) with the parameter choices specied in
fib3.m, namely

5 3
0
1
3 8 3 .
(8.10)
B=
10
0
3 5
The associated resolvent is
(sI B)1

s + 1.3s + 0.31 0.3s + 0.15

0.09
1
0.3s + 0.15
0.3s + 0.15
s2 + s + 0.25
=
B (s)
2
0.09
0.3s + 0.15 s + 1.3s + 0.31

where
B (s) = s3 + 1.8s2 + 0.87s + 0.11
82

(8.11)

is the characteristic polynomial of B. Assuming a current stimulus of the form i0 (t) = t exp(t/4)/1000,
and Em = 0 brings

1.965/(s + 1/4)2

0
(Lg)(s) =
0
and so (8.11) persists in
Lx = (sI B)1 Lg

2
s + 1.3s + 0.31
1.965
0.3s + 0.15
=
2
3
(s + 1/4) (s + 1.8s2 + 0.87s + 0.11)
0.09

Now comes the rub. A simple linear solve (or inversion) has left us with the Laplace transform of
x. The accursed No Free Lunch Theorem informs us that we shall have to do some work in order
to recover x from Lx.
In coming sections we shall establish that the inverse Laplace transform of a function h is
!
1
1
(L h)(t) =
h(s) exp(st) ds,
(8.12)
2i C
where s runs along C, a closed curve in the complex plane that encircles all of the singularities of
h. We dont suppose the reader to have yet encountered integration in the complex plane and so
please view (8.12) as a preview of coming attractions.
With the inverse Laplace transform one may express the solution of (8.3) as
x(t) = L1 (sI B)1 (Lg + x(0)).

(8.13)

As an example, let us take the rst component of Lx, namely

1.965(s2 + 1.3s + 0.31)
.
Lx1 (s) =
(s + 1/4)2(s3 + 1.8s2 + 0.87s + 0.11))

(8.14)

The singularities, or poles, are the points s at which Lx1 (s) blows up. These are clearly the roots
of its denominator, namely
11/10, 1/2, 1/5 and

1/4,

(8.15)

and hence the curve, C, in (8.12), must encircle these. We turn to Matlab however to actually
evaluate (8.12). Referring to fib3.m for details we note that the ilaplace command produces

x1 (t) = 1.965 8 exp(t/2) (1/17) exp(t/4)(40912/17 + 76t)
+ (400/3) exp(t/5) + (200/867) exp(11t/10)
83

(8.16)

6
x

x (mV)

10

15

20

25

30

35

40

45

50

t (ms)

Figure 8.2. The 3 potentials associated with Figure 8.1.

The other potentials, see Figure 8.2, possess similar expressions. Please note that each of the poles
of Lx1 appear as exponents in x1 and that the coecients of the exponentials are polynomials whose
degrees are determined by the orders of the respective poles. fib3.m
8.3. The BackwardEuler Method
Where in the previous section we tackled the derivative in (8.3) via an integral transform we
pursue in this section a much simpler strategy, namely, replace the derivative with a nite dierence
quotient. That is, one chooses a small dt and replaces (8.3) with
x(t) x(t dt)
= B x(t) + g(t).
(8.17)
dt
The utility of (8.17) is that it gives a means of solving for x at the present time, t, from knowledge
of x in the immediate past, t dt. For example, as x(0) = x(0) is supposed known we write (8.17)
as
(I/dt B)
x(dt) = x(0)/dt + g(dt).
Solving this for x(dt) we return to (8.17) and nd
(I/dt B)
x(2dt) = x(dt)/dt + g(2dt)
and solve for x(2dt). The general step from past to present,
x(jdt) = (I/dt B)1 (
x((j 1)dt)/dt + g(jdt)),

(8.18)

is repeated until some desired nal time, T dt, is reached. This equation has been implemented in
fib3.m with dt = 1 and B and g as above. The resulting x (run fib3 yourself!) is indistinguishable
from 8.2.
Comparing the two representations, (8.13) and (8.18), we see that they both produce the solution
to the general linear system of ordinary equations, (8.3), by simply inverting a shifted copy of B.
The former representation is hard but exact while the latter is easy but approximate. Of course we
should expect the approximate solution, x
, to approach the exact solution, x, as the time step, dt,
approaches zero. To see this let us return to (8.18) and assume, for now, that g 0. In this case,
one can reverse the above steps and arrive at the representation
x(jdt) = ((I dtB)1 )j x(0).
84

(8.19)

Now, for a xed time t we suppose that dt = t/j and ask whether
x(t) = lim ((I (t/j)B)1 )j x(0).
j

This limit, at least when B R, yields the exponential

x(t) = exp(Bt)x(0),

(8.20)

clearly the correct solution to (8.3) in the absense of the forcing term, g.
If we now presume that g = 0 we must replace (8.19) by
x(jdt) = ((I dtB)1 )j x(0) + ((I dtB)1 )j dt(g(dt) + (I dtB)g(2dt) + + (I dtB)j1 g(jdt)).
Again setting dt = t/j we nd
x(t) = ((I(t/j)B)1 )j x(0)+((I(t/j)B)1 )j (t/j)(g(t/j)+(I(t/j)B)g(2t/j)+ +(I(t/j)B)j1 g(t)).
We concentrate on the latter sum and note that the coecients of the g terms
1 = exp(B(t/j)) + O(1/j)
(I (t/j)B) = exp(B(2t/j)) + O(1/j)
(I (t/j)B)2 = exp(B(3t/j)) + O(1/j)
=
(I (t/j)B)j1 = exp(Bt) + O(1/j)
and so
(t/j)(g(t/j) + (I (t/j)B)g(2t/j) + + (I (t/j)B)j1 g(t))
=(t/j)(exp(Bt/j)g(t/j) + exp(B2t/j)g(2t/j) + + exp(Bt)g(t))
+ (t/j)(g(t/j) + g(2t/j) + + g(t))O(1/j),
= (t/j)

exp(Btk/j)g(tk/j) + O(1/j)(t/j)

k=1

lim (t/j)

lim (t/j)

g(tk/j).

k=1

and

and

g(tk/j) =

g(y) dy
0

k=1

!
exp(Btk/j)g(tk/j) =

exp(By)g(y) dy.
0

k=1

and so x converges to
!

x(t) = exp(Bt)x(0) +

exp((t y)B)g(y) dy.

(8.21)

We rcognize this as the homogeneous solution, (8.20), plus the convolution (dene better in an
exercise) of exp(Bt) and g(t). On reconciling (8.21) with its equivalent formulation, (8.13), it
appears that
exp(Bt) = L1 (sI B)1 .
(8.22)
85

When B = 1 this concurs with our very rst example. Proper understanding of this representation
in the matrix case will have to wait until we have mastered the inverse Laplace transform.
8.4. Dynamics of Mechanical Systems
Regarding the ber nets of Chapter 3, we may move from the equilibrium equations, for the
displacement x due to a constant force, f ,
Sx = f,

where S = AT KA,

to the dynamical equations for the displacement, x(t), due to a time varying force, f (t), and or
nonequilibrium initial conditions, by simply appending the Newtonian inertial terms, i.e.,
Mx (t) + Sx(t) = f (t),

x(0) = x0 ,

x (0) = v0 ,

(8.23)

where M is the diagonal matrix of node masses, x0 denotes their initial displacement and v0 denotes
their initial velocity.
We transform this system of second order dierential equations to an equivalent rst order system
by introducing
u1 x and u2 u1
and then noting that (8.23) takes the form
u2 = x = M 1 Su1 + M 1 f (t).
As such, we nd that u = (u1 u2 )T obeys the familiar
u = Bu + g,


where
B=


0
I
,
M 1 S 0

u(0) = u0


g=

M 1 f


,

(8.24)
 
x0
u0 =
.
v0

(8.25)

Let us consider the concrete example of the chain of three masses in Fig. 2.1. If each node has mass
m and each spring has stiness k then

2 1 0
k
1 2 1 .
(8.26)
M 1 S =
m
0 1 2
The associated characteristic polynomial of B is
B (s) = s6 + 6cs4 + 10c2 s2 + 4c3 , where c k/m,
(8.27)

is a cubic in s2 with simple roots at 2c and 2c 2c. And so the eigenvalues of B are the three
purely imaginary numbers



1 = i 2c 2c, 2 = i 2c, 3 = i 2c + 2c
(8.28)
and their complex conjugates, 4 = 1 , 5 = 2 and 6 = 3 . Next, if the exogenous force, f ,
is 0, and the initial disturbance is simply x1 (0) = 1 then
2

3c s + 4cs3 + s5
1
cs(s2 + 2c) .
(8.29)
Lu1(s) =
B (s)
c2 s
86

On computing the inverse Laplace Transform we (will) nd

x1 (t) =

exp(j t)(3c j +

4c3j

"
j ) ""
,
B (s) "s=j

(s
5j )

j=1

(8.30)

that is, x1 is a weighted sum of exponentials. As each of the rates are purely imaginary it will
follow from Eulers formula (9.13) that our masses will simply oscillate according to weighted sums
of three sinusoids. For example, note that

(8.31)
exp(2 t) = cos( 2ct) + i sin( 2ct) and exp(5 t) = cos( 2ct) i sin( 2ct).
Of course such sums may reproduce quite complicated behavior. We have illustrated each of the
displacements in Figure 8.3 in the case that c = 1. Rather than plotting the complex, yet explicit,
expression (8.30) for x1 , we simply implement the Backward Euler scheme of the previous section.
5

4+x1

2+x

10

20

30

40

50

60

70

80

90

100

Figure 8.3. The displacements of the 3 masses in Figure 3.1, with k/m = 1, following an initial displacement of the rst mass. For viewing purposes we have oset the three displacements.
chain.m

87

8.5. Exercises

1. Compute, without the aid of a machine, the Laplace transforms of exp(t) and t exp(t). Show
all of your work.
2. Extract from fib3.m analytical expressions for x2 and x3 .
3. Use eig to compute the eigenvalues of B as given in (8.10). Use poly to compute the characteristic polynomial of B. Use roots to compute the roots of this characteristic polynomial.
Compare these to the results of eig. How does Matlab compute the roots of a polynomial?
(type type roots) for the answer). Submit a Matlab diary of your ndings.
4. Adapt the Backward Euler portion of fib3.m so that one may specify an arbitrary number
of compartments, as in fib1.m. As B, and so S, is now large and sparse please create the
sparse B via spdiags and the sparse I via speye, and then prefactor S into LU and use U\L\
rather than S\ in the time loop. Experiment to nd the proper choice of dt. Submit your
well documented M-le along with a plot of x1 and x50 versus time (on the same well labeled
graph) for a 100 compartment cable.
5. Derive (8.19) from (8.18) by working backwards toward x(0). Along the way you should explain
why (I/dt B)1 /dt = (I dtB)1 .
6. Show, for scalar B, that ((1 (t/j)B)1 )j exp(Bt) as j . Hint: By denition
((1 (t/j)B)1 )j = exp(j log(1/(1 (t/j)B)))
now use LHopitals rule to show that j log(1/(1 (t/j)B)) Bt.
7. If we place a viscous damper in parallel with each spring in Figure 3.1 as below

k1

m
1

k3

k2

f1

f3

f2

k4

Figure 8.4. A damped chain.

then the dynamical system (8.24) takes the form
Mx (t) + Dx (t) + Sx(t) = f (t)

(8.32)

where D = AT diag(d)A where d is the vector of damping constants. Modify chain.m to solve
this new system and use your code to reproduce Figure 8.5
88

4+x1

2+x

10

20

30

40

50

60

70

80

90

100

Figure 8.5. The displacement of the three masses in the weakly damped chain, where k/m = 1
and d/m = 1/10.

89

9. Complex Numbers, Functions and Derivatives

In this Chapter we investigate the algebra and geometry of the complex plane, C, and begin the
study of the calculus of functions from C to C. Ostensibly this is to prepare the way for the complex
integration required to make sense of the eigenvalue problem and the inverse Laplace transform and
their role in understanding dynamics.
9.1. Complex Numbers
A complex number is simply a pair of real numbers. In order to stress however that the two
algebras dier we separate the two real pieces by the symbol +i. More precisely, each complex
number, z, may be uniquely expressed by the combination x + iy where x and y are real and i
denotes 1. We call x the real part and y the imaginary part of z. We now summarize the
main rules of complex arithmetic. If
z1 = x1 + iy1
then

and z2 = x2 + iy2

(x1 + x2 ) + i(y1 + y2 )
(x1 + iy1 )(x2 + iy2 ) = (x1 x2 y1 y2 ) + i(x1 y2 + x2 y1 )
x1 iy1 ,
z1 z 2
(x1 x2 + y1 y2 ) + i(x2 y1 x1 y2 )

=
z2 z 2
x22 + y22


|z1 | z1 z 1 = x21 + y12 .

z1 + z2
z1 z2
z1
z1
z2

In addition to the Cartesian representation z = x + iy one also has the polar form
z = |z|(cos + i sin ), where (, ] and

/2,
if x = 0,

if x = 0,
/2,
= atan2(y, x) arctan(y/x)
if x > 0,

arctan(y/x) + if x < 0,

arctan(y/x) if x < 0,

y > 0,
y < 0,

(9.1)

y 0,
y < 0.

This form is especially convenient with regards to multiplication. More precisely,

z1 z2 = |z1 ||z2 |{(cos 1 cos 2 sin 1 sin 2 ) + i(cos 1 sin 2 + sin 1 cos 2 )}
= |z1 ||z2 |{cos(1 + 2 ) + i sin(1 + 2 )}.
As a result,
z n = |z|n (cos(n) + i sin(n)).
A complex vector (matrix) is simply a vector (matrix) of complex numbers. Vector and matrix
addition proceed, as in the real case, from elementwise addition. The inner product of two complex
vectors requires, however, some care. This is evident when we try to use the old notion to dene
the length of complex vector. To wit, note that if


1+i
z=
1i
90

then
z T z = (1 + i)2 + (1 i)2 = 1 + 2i 1 + 1 2i 1 = 0.
Now norm should measure the distance from a point to the origin and should only be zero for the
zero vector. The x, as you have probably guessed, is to sum the squares of the magnitudes of
the components of z. This is accomplished by simply conjugating one of the vectors. Namely, we
dene the norm of a complex vector via

(9.2)
z z T z.
In the example above this produces


|1 + i|2 + |1 i|2 =

4 = 2.

As each real number is the conjugate of itself, this new denition subsumes its real counterpart.
The notion of magnitude also gives us a way to dene limits and hence will permit us to introduce
complex calculus. We say that the sequence of complex numbers, {zn : n = 1, 2, . . .}, converges to
the complex number z0 and write
zn z0

or z0 = lim zn ,
n

when, presented with any > 0 one can produce an integer N for which |zn z0 | < when n N.
As an example, we note that (i/2)n 0, for given an we note that if n > N = log2 (1/) then
|(i/2)n | < .
9.2. Complex Functions
A complex function is merely a rule for assigning certain complex numbers to other complex
numbers. The simplest (nonconstant) assignment is the identity function f (z) z. Perhaps the
next simplest function assigns to each number its square, i.e., f (z) z 2 . As we decomposed the
argument of f , namely z, into its real and imaginary parts, we shall also nd it convenient to
partition the value of f , z 2 in this case, into its real and imaginary parts. In general, we write
f (x + iy) = u(x, y) + iv(x, y)
where u and v are both realvalued functions of two real variables. In the case that f (z) z 2 we
nd
u(x, y) = x2 y 2 and v(x, y) = 2xy.
With the tools of the previous section we may produce complex polynomials
f (z) = z m + cm1 z m1 + + c1 z + c0 .
We say that such an f is of degree m. We shall often nd it convenient to represent polynomials
as the product of their factors, namely
f (z) = (z 1 )p1 (z 2 )p2 (z h )ph .

(9.3)

Each j is a root of f of degree pj . Here h is the number of distinct roots of f . We call j a

simple root when pj = 1. In chapter 8 we observed the appearance of ratios of polynomials or so
called rational functions. Suppose
f (z)
r(z) =
g(z)
91

is rational, that f is of order at most m 1 while g is of order m with the simple roots {1 , . . . , m }.
It should come as no surprise that such a r should admit a Partial Fraction Expansion
r(z) =

m

j=1

rj
.
z j

One uncovers the rj by rst multiplying each side by (z j ) and then letting z tend to j . For
example, if
r1
r2
1
=
+
(9.4)
2
z +1
z+i zi
then multiplying each side by (z + i) produces
1
r2 (z + i)
= r1 +
.
zi
zi
Now, in order to isolate r1 it is clear that we should set z = i. So doing we nd r1 = i/2. In order
to nd r2 we multiply (9.4) by (z i) and then set z = i. So doing we nd r2 = i/2, and so
z2

i/2
i/2
1
=
+
.
+1
z+i zi

(9.5)

Returning to the general case, we encode the above in the simple formula
rj = lim (z j )r(z).
zj

(9.6)

You should be able to use this to conrm that

z2

1/2
1/2
z
=
+
.
+1
z+i zi

(9.7)

Recall that the resolvent we met in Chapter 8 was in fact a matrix of rational functions. Now, the
partial fraction expansion of a matrix of rational functions is simply the matrix of partial fraction
expansions of each of its elements. This is easier done than said. For example, the resolvent of


0 1
B=
1 0
is
(zI B)


1
z
= 2
z + 1 1

1
1/2
=
z + i i/2

1
z




(9.8)
1
i/2
1/2 i/2
+
.
1/2
z i i/2 1/2
The rst line comes from either Gauss-Jordan by hand or via the symbolic toolbox in Matlab ,
e.g.,
>> syms z; R = inv(z*eye(2)-B)
(9.9)
while the second line in (9.8) is simply an amalgamation of (9.4) and (9.5). We may achieve the
two coecient matrices in Matlab via
>> P1 = subs(simple((z+i)*R),z,-i),
>> P2 = subs(simple((z-i)*R),z,i).
92

(9.10)

Regarding P1 this line rst multiplies by the oending fraction and simplies (in order to clear
the fraction) before substituting z=-i. You may at times also choose to run simple on R. And
whenever expressions ll the screen you may view individual elements in a nicer light via pretty,
e.g., >> pretty(R(1,1)).
Complex matrices have nally entered the picture. We will devote all of Chapter 11 to uncovering
the remarkable properties enjoyed by the matrices that appear in the partial fraction expansion of
(zI B)1 . Have you noticed that, in our example, the two matrices are each projections, that
they sum to I, and that their product is 0? Could this be an accident? To answer this we will also
need to develop (zI B)1 in a geometric expansion. At its simplest, the n-term geometric series,
for z = 1, is
n1

1 zn
zk =
.
(9.11)
1

z
k=0
We will prove this in the exercises and use it to appreciate the beautiful orthogonality of the columns
of the Fourier Matrix.
In Chapter 8 we were confronted with the complex exponential when considering the Laplace
Transform. By analogy to the real exponential we dene


zn

exp(z)

n=0

(9.12)

n!

and nd that, for R,

exp(i) = 1 + i + (i)2 /2 + (i)3 /3! + (i)4 /4! +
= (1 2 /2 + 4 /4! ) + i( 3 /3! + 5 /5! )
= cos + i sin .

(9.13)

This should hopefully clear up any mystery remaining from (8.31). With these observations, the
polar form is now simply z = |z| exp(i). One may just as easily verify that
cos =

exp(i) + exp(i)
2

and

sin =

exp(i) exp(i)
.
2i

sin z

exp(iz) exp(iz)
.
2i

These suggest the denitions, for complex z, of

cos z

exp(iz) + exp(iz)
2

and

As in the real case the exponential enjoys the property that

exp(z1 + z2 ) = exp(z1 ) exp(z2 )
and in particular
exp(x + iy) = exp(x) exp(iy) = exp(x)(cos y + i sin y).
Finally, the inverse of the complex exponential is the complex logarithm,
ln z ln(|z|) + i,

One nds that ln(1 + i) = ln 2 + i3/4.

for z = |z| exp(i).

93

9.3. Fourier Series and Transforms

The complex exponential is fundamental to the two central transforms of modern science,
those named after Fourier and Laplace. We pause here to develop the key properties of Fourier
Series and Transforms.
We observed in 8.4 that a mechanical system with 3 degrees of freedom vibrates at 3 characteristic frequencies, (8.28). One of the most common applications of Fourier tools is to the problem of
spectral analysis, i.e., the determination of the spectral or frequency components of a signal.
Most everything follows from the fact that exp(2imt) is orthogonal to exp(2int) in the sense
that, for integers m and n,

! 1
! 1
0 if m = n,
exp(2int) exp(2imt) dt =
exp(2i(n m)t) dt =
.
(9.14)
1 if m = n.
0
0
The key idea is now to use these orthonormal exponentials as a basis for a class of functions dened
on 0 t 1. More precisely, given a function f we develop it in a Fourier Series

f (t) =

fn exp(2int).

(9.15)

n=

To determine the Fourier coecients, fm , multiply each side of (9.15) by exp(2imt) then integrate
and invoke (9.14):
! 1
fm =
f (t) exp(2imt) dt.
(9.16)
0

For example, if
1
f (t) = t then f0 =
2
while if

and fn =

1
f (t) = t(1 t) then f0 =
6

i
,
2n

and fn =

|n| > 0,

1
,
2(n)2

(9.17)

|n| > 0.

(9.18)

We speak of fm exp(2imt) as the projection of f onto exp(2imt) and so interpret fm as the

amount of f at frequency m. Regarding the sense of negative frequency we note that if f is
real then fn = fn and so (9.15) takes the form
f (t) = f0 + 2

{fn exp(2int)}.

(9.19)

n=1

This in turn suggests that we expand

! 1
!

fm =
f (t) exp(2imt) dt =
0

c
s
f (t)(cos(2mt) i sin(2mt)) dt = 2(fm
ifm
)

when
c
fm

1
=
2

f (t) cos(2mt) dt and

0

s
fm

1
=
2

94

f (t) sin(2mt) dt,

0

m = 1, 2, . . .

(9.20)

In which case (9.19) becomes

f (t) = f0 +

fnc cos(2nt) + fns sin(2nt).

(9.21)

1  cos(2nt)
.
and (1 t)t =
6 n=1 n2 2

(9.22)

n=1

Returning to our two examples, we nd

1  sin(2nt)
t=
2 n=1
n

It is very instructive to visually interogate the convergence of these sums by graphing

fN (t) = f0 +

fnc cos(2nt) + fns sin(2nt)

(9.23)

n=1

of (9.15).
0.3

1.2
exact
N=3
N=5

exact
N=1
N=3

0.25

0.8
0.2

0.6
0.15

0.4
0.1
0.2
0.05

0.2

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

Figure 9.1 (A) The exact f (t) = t and two of its low frequency Fourier approximants. (B) The
exact f (t) = (1 t)t and two of its low frequency Fourier approximants. (fftexa.m)
In practice we are most often confronted not with an analytical expression of a function on the
unit interval but rather with N samples over an interval of duration T :
(N )
f (mdt),
fm

dt T /N,

m = 0, . . . , N 1.

We now attempt to develop f (N ) in a discrete Fourier series of the form

(N )
fm

N 1
1  (N )
f exp(2inmdt/T ).
=
N n=0 n

(9.24)

On dening
wN exp(2i/N)
we note that Eq. (9.24) takes the very simple form,
N 1
1  mn (N )
(N )
wN fn = fm
.
N n=0

95

(9.25)

This in turn may be written as the matrix equation

1
FN f(N ) = f (N )
N

(9.26)

N
= 1,
where, noting that wN

1
1
1
2
1 wN
wN

2
4
wN
FN =
1 wN

2(N 1)
N 1
1 wN
wN

1
N 1

wN
2(N 1)
.
wN

(N 1)2
wN

(9.27)

We now exploit this very special structure of FN and arrive at an elegant solution to Eq. (9.26).
To begin we examine the jk element of FNH FN , i.e., row j of FNH (the conjugate transpose of FN )
times column k of FN ,
2(j1)

k1
(FNH FN )jk = 1 1 + w j1
+ wN
N wN
m(j1)

2(k1)

wN

(N 1)(j1)

+ + wN

(N 1)(k1)

wN

m(k1)

wN
= exp(2im(j1)) exp(2im(j1)) = 1 for each m and (FNH FN )jj =
If j = k then w N
(j1) (k1)
N. If j = k we let z = wN wN
and nd the nite geometric series (recall (9.11))
(FNH FN )jk = 1 + z + z 2 + + z N 1 =

1 zN
= 0,
1z

(9.28)

where the nal equality stems from z N = 1. Gathering the above computations, we have shown
that
1
FNH FN = NI and so FN1 = FNH and f(N ) = FNH f (N )
(9.29)
N
is the solution to Eq. (9.26). We speak of f(N ) as the Discrete Fourier Transform of f (N ) and
note the latter equation in (9.29) may be expressed in component form as
(N )
=
fm

N
1


(N )
w mn
N fn

n=0

N
1


exp(2i(m/T )(ndt))f (ndt).

(9.30)

n=0

As such we interpret m/T as the associated discrete frequencies. It also follows from Eq. (9.30)
that if f (N ) is real then
(N )

(N )
fN/2+j = fN/2j ,

j = 1, 2, . . . , N/2 1,

(9.31)

and as such only the rst 1 + N/2 frequencies

m = m/T,

m = 0, . . . , N/2,

carry information. This observation in fact leads to a fast method, known as the fft, for implementing the Discrete Fourier Transform. The basic idea is that a DFT of order N can be implemented
by 2 DFTs of order N/2.
We illustrate it by recovering the three frequencies of vibration in the 3 signals of Figure 8.3.
96

0.18
x
0.16

a1=0.95

300

0.12
250
0.1


cxx

350

x3

0.14

abs(fft(x ))/N

a =0.95

x2

200

0.08
150
0.06
100

0.04

50

0.02
0
0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.1

0.2

(Hz)

, (Hz)

0.3

0.4

0.5

Figure 9.2 (A) N = 1, 000, 001 samples. Note that x2 has no 2 . Red lines at k /(2), recall
(8.28), c = 1. (chainfreq.m) (B) The power spectra of xj+1 a1 xj = j with a1 = 0.95.
We consider a Stationary Process, x, of 6.7 and dene its Power Spectrum to be the Fourier
Transform of its autocovariance:


cxx ()
cxx (k) exp(2ikdt) cxx (k) mean(xj xj+k )
(9.32)
k=

where dt is the time step between samples of x.

If x is the AR(1) process of (6.43) and dt = 1 s then, recalling (6.44)(6.45),
cxx () = c0 + c0

ak1 {exp(2ik) + exp(2ik)}

k=1

a1 exp(2i)
a1 exp(2i)
+ c0
1 a1 exp(2i)
1 a1 exp(2i)
2
1 a1
2
=
= c0
1 2a1 cos(2) + a21
1 2a1 cos(2) + a21
= c0 + c0

(9.33)

We illustrate this in Figure 9.2(B).

For our next class we consider moving averages
xn =

aj n+j ,

mean(m n ) = mn .

j=

We argue that the Fourier Transform of its autocovariance

cxx (n) mean(xm+n xm ) =

ajn aj

j=

can be represented in terms of the Fourier Transform of a:

a
()

ak exp(2ik),

k=

for

!
ak =

1/2

a() exp(2ik) d
1/2

97

(9.34)

and so
! 1/2
1/2

cxx () exp(2in) d =

ajn aj =

j=

aj

a
() exp(2in)
1/2

a
() exp(2i(n j)) d

1/2

j=

1/2

1/2

aj exp(2ij) d

j=

1/2

1/2

a
() exp(2in)a() d =

=
1/2

1/2

|a()|2 exp(2in) d

and so if x is a moving average of the form (9.34) then

cxx () = |a()|2.
More generally, we consider the discrete convolution of two time series,

ap xjp
(a  x)j
p

and establish
Proposition 9.1 If y = a  x then
(i) y() = a
()
x().
(ii) If x is stationary then cxy = a  cxx and cyy = (E a)  (a  cxx ).
(iii) If x is stationary then cxy () = a
()
cxx () and cyy () = |a()|2 cxx ().
Proof: Regarding (i) we nd

(a  x)k exp(2ikdt)
y() =
k

ap xkp exp(2i(k p + p)dt)

ap exp(2ipdt)

xkp exp(2i(k p)dt) = a()

x().

Regarding (ii) we nd
cxy (k) = mean(xj (a  x)j+k ) = mean(xj
=

ap mean(xj xj+kp ) =

cyy (k) = mean((a  x)j (a  x)j+k ) = mean(



ap mean(xjp

ap

ap xj+kp)

ap cxx (k p) = (a  cxx )k ,

and

ap xjp

aq xj+kq ) =

aq cxx (k + p q) =

98

ap

aq xj+kq )

aq mean(xjp xj+kq )

ap (a  cxx )k+p = ((E a)  (a  cxx ))k .

(9.35)


Finally (iii) follows from (i) and (ii) on noting that E
. End of Proof.
a = a
We apply this to the AR(1) process, xj a1 xj1 = j , so a() = 1 a1 exp(2i) and so
c () = (1 2a1 cos(2) + a21 )
cxx ().
As c () = 2 we have arrived at (9.33) by dierent means.
We next dene the coherence of two processes to be the normalized crossspectrum
Rxy () 

cxy ()
cxx ()
cyy ()

(9.36)

()/|a()|. As a simple
and note that Proposition 9.1 asserts that if y = a  x then Rxy () = a
example, if yk = xk+d is a delayed copy of x then y = a  x with aj = 0 save ad = 1 in which case
() = exp(2di).
Rxy () = a
Given two processes, x and y, we now search for the lter a for which y is closest to a  x in the
sense that the error
= y (a  x)
(9.37)
has minimal power spectrum at each frequency. Arguing as in the proof of Proposition 9.1, the
autocovariance of each side of (9.37) spells
c = cyy (E a)  cxy a  cyx + (E a)  (a  cxx ).
and so the associated error Spectrum is
c () = cyy () a
cxy a()
cxy () + |a()|2cxx ()
cxx |2 + cyy (1 |Rxy |2 ).
= cxx |a cxy /

(9.38)

To minimize this we choose

cxx ().
a
() = cxy ()/

(9.39)

In which case c () = cyy ()(1 |Rxy ()|2). As the a that results from (9.39) requires knowledge
of covariances at an innite number of lags it is known as the lter with Infinite Impulse Response,
by contrast of the Finite Impulse Response of Exer.6.17. For real-time lters the obstacle is not
access to the innite past but rather access to the future. To design best lters that require no
knowledge of the future will require knowledge of residues.
9.4. Complex Dierentiation and the First Residue Theorem
The complex function f is said to be dierentiable at z0 if
f (z) f (z0 )
zz0
z z0
lim

exists, by which we mean that

f (zn ) f (z0 )
zn z0

converges to the same value for every sequence {zn } that converges to z0 . In this case we naturally
call the limit f  (z0 ).
99

Example: The derivative of z 2 is 2z.

z 2 z02
(z z0 )(z + z0 )
= lim
= 2z0 .
zz0 z z0
zz0
z z0
lim

Example: The exponential is its own derivative.


exp(z) exp(z0 )
exp(z z0 ) 1
(z z0 )n
= exp(z0 ).
= exp(z0 ) lim
= exp(z0 ) lim
zz0
zz0
zz0
z z0
z z0
(n
+
1)!
n=0

lim

Example: The real part of z is not a dierentiable function of z.

We show that the limit depends on the angle of approach. First, when zn z0 on a line parallel
to the real axis, e.g., zn = x0 + 1/n + iy0 , we nd
x0 + 1/n x0
= 1,
n x0 + 1/n + iy0 (x0 + iy0 )
lim

while if zn z0 in the imaginary direction, e.g., zn = x0 + i(y0 + 1/n), then

x0 x0
= 0.
n x0 + i(y0 + 1/n) (x0 + iy0 )
lim

This last example suggests that when f is dierentiable a simple relationship must bind its
partial derivatives in x and y.
Proposition 9.2. If f is dierentiable at z0 then
f  (z0 ) =

f
f
(z0 ) = i (z0 ).
x
y

Proof: With z = x + iy0 ,

f (z) f (z0 )
f (x + iy0 ) f (x0 + iy0 )
f
(z0 ).
= lim
=
zz0
xx0
z z0
x x0
x

f  (z0 ) = lim

Alternatively, when z = x0 + iy then

f
f (z) f (z0 )
f (x0 + iy) f (x0 + iy0 )
= i (z0 ).
= lim
zz0
yy0
z z0
i(y y0 )
y

f  (z0 ) = lim
End of Proof.

In terms of the real and imaginary parts of f this result brings the CauchyRiemann equations
v
v
u
u
(9.40)
=
and
= .
x
y
x
y
Regarding the converse proposition we note that when f has continuous partial derivatives in a
region obeying the CauchyRiemann equations then f is in fact dierentiable in that region.
We remark that with no more energy than that expended on their real cousins one may uncover
the rules for dierentiating complex sums, products, quotients, and compositions.
100

As one important application of the derivative let us attempt to expand in partial fractions a
rational function whose denominator has a root with degree larger than one. As a warm-up let us
try to nd r1,1 and r1,2 in the expansion
z+2
r1,1
r1,2
=
.
+
2
(z + 1)
z + 1 (z + 1)2
Arguing as above it seems wise to multiply through by (z + 1)2 and so arrive at
z + 2 = r1,1 (z + 1) + r1,2 .

(9.41)

On setting z = 1 this gives r1,2 = 1. With r1,2 computed (9.41) takes the simple form z + 1 =
r1,1 (z + 1) and so r1,1 = 1 as well. Hence
1
1
z+2
=
.
+
2
(z + 1)
z + 1 (z + 1)2
This latter step grows more cumbersome for roots of higher degree. Let us consider
r1,1
r1,3
(z + 2)2
r1,2
=
+
.
+
(z + 1)3
z + 1 (z + 1)2 (z + 1)3
The rst step is still correct: multiply through by the factor at its highest degree, here 3. This
leaves us with
(z + 2)2 = r1,1 (z + 1)2 + r1,2 (z + 1) + r1,3 .
(9.42)
Setting z = 1 again produces the last coecient, here r1,3 = 1. We are left however with one
equation in two unknowns. Well, not really one equation, for (9.42) is to hold for all z. We exploit
this by taking two derivatives, with respect to z, of (9.42). This produces
2(z + 2) = 2r1,1 (z + 1) + r1,2

and 2 = 2r1,1 .

The latter of course needs no comment. We derive r1,2 from the former by setting z = 1. We
generalize from this example and arrive at
Proposition 9.3. The First Residue Theorem. The ratio, r = f /g, of two polynomials where
the order of f is less than that of g and g has h distinct roots {1 , . . . , h } of respective degrees
{p1 , . . . , ph }, may be expanded in partial fractions
r(z) =

pj
h 

j=1 k=1

rj,k
(z j )k

(9.43)

where, as above, the residue rj,k is computed by rst clearing the fraction and then taking the
proper number of derivatives and nally clearing their powers. That is,
rj,k = lim

zj

dpj k
1
{(z j )pj r(z)}.
(pj k)! dz pj k

(9.44)

We will be most interested in the residues of matrices of rational functions. For example, the matrix

1 0 0
(9.45)
B = 1 3 0
0 1 1
101

has the resolvent

R(z) = (zI B)1 =

1
z1
1
(z1)(z3)
1
(z1)2 (z3)

0
0

0
1
z3
1
(z1)(z3)

(9.46)

1
z1

and so we ought to be able to collect terms and express the matrix residues, Rj,k , in the partial
fraction expansion
1
1
1
R1,1 +
R2,1 .
R(z) =
R1,2 +
(9.47)
2
z1
(z 1)
z3
At this point, this is merely bookeeping and the only challenging term is the (3,1) element of (9.46).
We write
r1,1
r1,2
1
r2,1
=
+
.
+
2
2
(z 1) (z 3)
z 1 (z 1)
z3
It follows from (9.44) that


1
1
r1,1 =
(1) =
z3
4


and
r2,1 =


and r1,2 =

1
(z 1)2

1
z3


(1) =

1
2

1
(3) = .
4

Collecting terms reveals the coecient matrices in (9.47)

1
0
0
0
0 0
0
0 ,
0 0 ,
R1,1 = 1/2
R1,2 = 0
1/4 1/2 1
1/2 0 0

(9.48)

(9.49)

and R2,1

0
0 0
= 1/2 1 0 . (9.50)
1/4 1/2 0

In closing, we note that these may be computed, as in (9.10), via Matlab ,

>> R11 = subs(simple((z-1)*R),z,1),
>> R12 = subs(simple(diff((z-1)^2*R),z,1),
>> R21 = subs(simple((z-3)*R),z,3),

(9.51)

where the coecient of the double pole required a symbolic derivative, i.e., diff.
9.5. Exercises
1. Express | exp(x + iy)| in terms of x and/or y.
2. Suppose z = 1 and dene the n-term geometric series

n1


zk ,

k=0

and show, by brute force, that z = 1 z n . Derive (9.11) from this result.
3. Conrm that exp(ln z) = z and ln exp(z) = z.
4. Find the real and imaginary parts of cos z and sin z. Express your answers in terms of regular
and hyperbolic trigonometric functions.
102

5. Show that cos2 z + sin2 z = 1.

6. As in the real calculus,
the exponential and natural log permit one to dene arbitrary powers.

Please compute i via the denition z p exp(p ln z).

2
= wN/2 to arrive at
7. Suppose N is even and use (9.30) and wN

N/21
(N )
=
fm

n=0
m+N/2

From here use wN

N/21
(N )
wmn
N/2 f2n

wm
N

(N )

m = 0, 1, . . . , N 1.

wmn
N/2 f2n+1 ,

n=0

m
= wN
to establish

(N )
(No )
fm
= (FN/2 f (Ne ) )m + w m
)m
N (FN/2 f
(N )
(No )
)m
fm+N/2 = (FN/2 f (Ne ) )m wm
N (FN/2 f

m = 0, 1, . . . , N/2 1,

where f (Ne ) denotes the even elements and f (No ) denotes the odd elements. Compute the
savings.
8. Lets build a multivariate generalization of Prop. 9.1. For Xj Rn and Ap Rnn dene

Ap Xjp
(9.52)
Yj = (A  X)j
p

and establish
X().

(i) Y () = A()
CXX ()AH ().
(ii) If X is stationary then CY Y = A  E (CXX  AT ) and CY Y () = A()
9. Verify that sin z and cos z satisfy the Cauchy-Riemann equations (9.40) and use Proposition
9.1 to evaluate their derivatives.
10. Compute, by hand the partial fraction expansion of the rational function that we arrived at in
(8.14). That is, nd r1,1 , r1,2 , r2 , r3 and r4 in
s2 + 1.3s + 0.31
(s + 1/4)2(s + 1/2)(s + 1/5)(s + 11/10)
r1,2
r3
r4
r1,1
r2
+
+
+
.
=
+
s + 1/4 (s + 1/4)2 s + 1/2 s + 1/5 s + 11/10

r(s) =

Contrast your answer with the explicit expression in (8.16).

11. Submit a Matlab diary documenting your calculation, via the symbolic toolbox as in (9.51),
of the partial fraction expansion of the resolvent of

2 1 0
B = 1 2 1 ,
0 1 2
103

a matrix that appeared in (8.26) during our discussion of dynamics.

2
1

1
1

(sI B)1 =
2
2

s (2 + 2) 4
1
1
2

1
1 0 1
1
1 1
1

0 0 0 +
+
2
s 2 2 1 0 1
s (2 2) 4
1

104

You should achieve

2 1
2 .
2
2 1

10. Complex Integration

Our goal here is to develop the main results of complex integrtation theory and to apply these
to partial fraction expansion of the resolvent of a matrix and to the Inverse Laplace Transform of a
rational function. With these tools and applications we will have placed the dynamics work in 8.2
on a solid foundation.
10.1. Cauchys Theorem
We will be integrating complex functions over complex curves. Such a curve is parametrized
by one complex valued or, equivalently, two real valued, function(s) of a real parameter (typically
denoted by t). More precisely,
C {z(t) = x(t) + iy(t) : t1 t t2 }.
For example, if x(t) = y(t) = t from t1 = 0 to t2 = 1, then C is the line segment joining 0 + i0 to
1 + i.
We now dene
! t2
!
f (z) dz
f (z(t))z  (t) dt.
C

t1

For example, if C = {t + it : 0 t 1} as above and f (z) = z then

! 1
! 1
!
z dz =
(t + it)(1 + i) dt =
{(t t) + i2t} dt = i,
C

while if C is the unit circle {exp(it) : 0 t 2} then

! 2
! 2
!
!
z dz =
exp(it)i exp(it) dt = i
exp(i2t) dt = i
C

{cos(2t) + i sin(2t)} dt = 0.

Remaining with the unit circle but now integrating f (z) = 1/z we nd
! 2
!
1
z dz =
exp(it)i exp(it) dt = 2i.
C

We generalize this calculation to arbitrary (integer) powers over arbitrary circles. More precisely,
for integer m and xed complex a we integrate (z a)m over
C(a, ) {a + exp(it) : 0 t 2},

(10.1)

the circle of radius centered at a. We nd

!
! 2
m
(z a) dz =
(a + exp(it) a)m i exp(it) dt
C(a,)

m+1

exp(i(m + 1)t) dt

= i

(10.2)

= i
{cos((m + 1)t) + i sin((m + 1)t)} dt
0

2i if m = 1,
=
0
otherwise,
m+1

105

regardless of the size of !

When integrating more general functions it is often convenient to express the integral in terms
of its real and imaginary parts. More precisely
!
!
f (z) dz = {u(x, y) + iv(x, y)}{dx + idy}
C
!
!C
= {u(x, y) dx v(x, y) dy} + i {u(x, y) dy + v(x, y) dx}
C
C
! b
{u(x(t), y(t))x(t) v(x(t), y(t))y (t)} dt
=
a
! b
+i
{u(x(t), y(t))y (t) + v(x(t), y(t))x (t)} dt.
a

The second line should invoke memories of

Proposition 10.1. Greens Theorem. If C is a closed curve and M and N are continuously
dierentiable realvalued functions on Cin , the region enclosed by C, then

!! 
!
N
M

{M dx + N dy} =
dxdy
x
y
C
Cin
Applying this proposition to the situation above, we nd, so long as C is closed, that


!! 
!! 
!
v u
u v
f (z) dz =
+
dxdy + i

dxdy.
x y
x y
C
Cin
Cin
At rst glance it appears that Greens Theorem only serves to muddy the waters. Recalling the
CauchyRiemann equations however we nd that each of these double integrals is in fact identically
zero! In brief, we have proven
Proposition 10.2. Cauchys Theorem. If f is dierentiable on and in the closed curve C then
!
f (z) dz = 0.
C

Strictly speaking, in order to invoke Greens Theorem we require not only that f be dierentiable
but that its derivative in fact be continuous. This however is simply a limitation of our simple mode
of proof, Cauchys Theorem is true as stated.
This theorem, together with (10.2), permits us to integrate every proper rational function. More
precisely, if r = f /g where f is a polynomial of degree at most m 1 and g is an mth degree
polynomial with h distinct zeros at {j }hj=1 with respective multiplicities of {pj }hj=1 we found that
r(z) =

pj
h 

j=1 k=1

rj,k
.
(z j )k

(10.3)

Observe now that if we choose the radius j so small that j is the only zero of g encircled by
Cj C(j , j ) then by Cauchys Theorem
!
pj
!

1
r(z) dz =
rj,k
dz.
k
(z

j)
Cj
C
j
k=1
106

In (10.2) we found that each, save the rst, of the integrals under the sum is in fact zero. Hence
!
r(z) dz = 2irj,1.
(10.4)
Cj

With rj,1 in hand, say from (9.44) or residue, one may view (10.4) as a means for computing the
indicated integral. The opposite reading, i.e., that the integral is a convenient means of expressing
rj,1, will prove just as useful. With that in mind, we note that the remaining residues may be
computed as integrals of the product of r and the appropriate factor. More precisely,
!
r(z)(z j )k1 dz = 2irj,k .
(10.5)
Cj

It is a simple, but highly important, matter to extend this representation to a matrix of rational
functions. More precisely, if R(z) (zI B)1 is the resolvent associated with B then (10.3) and
(10.5) state that
pj
h 

Rj,k
R(z) =
(z j )k
j=1 k=1
where
Rj,k

1
=
2i

!
R(z)(z j )k1 dz.

(10.6)

Cj

Lets consider these in the concrete setting of (9.45), with resolvent (9.46). It now follows from
(10.6) that

#
dz
0
0
# dz
1 # C1 z1
dz
0
R1,1 =
(10.7)
.
# C1 (z1)(z3) # C1 z3
#
2i
dz
dz
dz
C1 (z1)2 (z3)

C1 (z1)(z3)

C1 z1

Evaluation of the upper triangle is a simple matter of recalling our brute force calculation (10.2).
Regarding the lower triangle, we fall back on partial fractions before again invoking (10.2). More
precisely
!
!
!
1
1
1
dz
1/2
1/2
1
=
dz +
dz =
(10.8)
2i C1 (z 1)(z 3)
2i C1 z 1
2i C1 z 3
2
as the latter integrand has no pole in C1 , and similarly
!
!
!
!
1
1
1
dz
1/4
1/2
1/4
1
1
=
dz +
dz =
. (10.9)
dz +
2
2
2i C1 (z 1) (z 3)
2i C1 z 1
2i C1 (z 1)
2i C1 z 3
4
On assembling these integrals we nd
R1,1

1
0
0
0
0
= 1/2
1/4 1/2 1

in agreement with (9.50). In the next section we will develop more general integration methods
that both accelerate the computations in (10.8)(10.9) and permit us to integrate over regions with
multiple poles. The latter will be necessary to tease out global properties of the resolvent and to
nally come to grips with the inverse Laplace Transform.
107

10.2. The Second Residue Theorem

After (10.4) and (10.5) perhaps the most useful consequence of Cauchys Theorem is the freedom
it grants one to choose the most advantageous curve over which to integrate. More precisely,
Proposition 10.3. Suppose that C2 is a closed curve that lies inside the region encircled by the
closed curve C1 . If f is dierentiable in the annular region outside C2 and inside C1 then
!
!
f (z) dz =
f (z) dz.
C1

C2

Proof: With reference to the gure below we introduce two vertical segments and dene the
closed curves C3 = abcda (where the bc arc is clockwise and the da arc is counter-clockwise) and
C4 = adcba (where the ad arc is counter-clockwise and the cb arc is clockwise). By merely following
the arrows we learn that
!
!
!
!
f (z) dz =
f (z) dz +
f (z) dz +
f (z) dz.
C1

C2

C3

C4

As Cauchys Theorem implies that the integrals over C3 and C4 each vanish, we have our result.
End of Proof.
a
C1

C2
c
d

Figure 10.1. The Curve Replacement Lemma.

As an example, recalling (10.3) and (10.4), we may express the integral of a rational function
around a curve that encircles all of its poles as a sum of residues
!
r(z) dz =
C

pj !
h 

j=1 k=1

Cj

h

rj,k
dz = 2i
rj,1 .
(z j )k
j=1

(10.10)

To take a slightly more complicated example let us integrate f (z)/(z a) over some closed curve
C inside of which f is dierentiable and a resides. Our Curve Replacement Lemma now permits us
to claim that
!
!
f (z)
f (z)
dz =
dz.
C z a
C(a,) z a
It appears that one can go no further without specifying f . The alert reader however recognizes
that the integral over C(a, ) is independent of and so proceeds to let 0, in which case z a
and f (z) f (a). Computing the integral of 1/(z a) along the way we are lead to the hope that
!
f (z)
dz = f (a)2i.
C z a
108

In support of this conclusion we note that



!
!
f (z)
f (z)
f (a)
f (a)
dz =
+

dz
za za za
C(a,) z a
C(a,)
!
!
1
f (z) f (a)
dz +
dz.
= f (a)
za
C(a,) z a
C(a,)
Now the rst term is f (a)2i regardless of while, as 0, the integrand of the second term
approaches f  (a) and the region of integration approaches the point a. Regarding this second term,
as the integrand remains bounded (in fact it tends to f  (a)) as the region of integration shrinks to
a point, the integral must tend to zero. We have just proven
Proposition 10.4. Cauchys Integral Formula. If f is dierentiable on and in the closed curve
C then
!
f (z)
1
dz
(10.11)
f (a) =
2i C z a
for each a lying inside C.
The consequences of such a formula run far and deep. We shall delve into only one or two. First,
we note that, as a does not lie on C, the right hand side of (10.11) is a perfectly smooth function
of a. Hence, dierentiating each side, we nd
!
!
1
1
d f (z)
f (z)
df (a)

=
dz =
dz
(10.12)
f (a) =
da
2i C da z a
2i C (z a)2
for each a lying inside C. Applying this reasoning n times we arrive at a formula for the nth
derivative of f at a,
!
n!
f (z)
dn f
(a) =
dz
(10.13)
n
da
2i C (z a)1+n
for each a lying inside C. The upshot is that once f is shown to be dierentiable it must in fact be
innitely dierentiable. As a simple extension let us consider
!
f (z)
1
dz
2i C (z 1 )(z 2 )2
where f is dierentiable on and in C and C encircles both 1 and 2 . By the curve replacement
lemma this integral is the sum
!
!
f (z)
f (z)
1
1
dz +
dz
2
2i C1 (z 1 )(z 2 )
2i C2 (z 1 )(z 2 )2
where j now lies in only Cj . As f (z)/(z 2 ) is well behaved in C1 we may use (10.11) to conclude
that
!
f (z)
f (1 )
1
dz =
.
2
2i C1 (z 1 )(z 2 )
(1 2 )2
Similarly, As f (z)/(z 1 ) is well behaved in C2 we may use (10.12) to conclude that
"
!
f (z)
d f (a) ""
1
dz =
.
2i C2 (z 1 )(z 2 )2
da (a 1 ) "a=2
109

These calculations can be read as a concrete instance of

Proposition 10.5. The Second Residue Theorem. If g is a polynomial with roots {j }hj=1 of
degree {pj }hj=1 and C is a closed curve encircling each of the j and f is dierentiable on and in C
then
!
h

f (z)
dz = 2i
res(f /g, j )
C g(z)
j=1
where

1
dpj 1
res(f /g, j ) = lim
zj (pj 1)! dz pj 1

f (z)
(z j )
g(z)

pj

(10.14)

is called the residue of f /g at j by extension of (9.44).

The generality of this statement, and the notation required to specify the residue, should not
obscure the fact that it permits us to compute important integrals by merely evaluating the good
parts at the bad places. The bad places are of course the poles of the integrand and the good part
of the integrand is what remains after multiplying by the oending factor, and perhaps taking a
few derivatives. Let us put this interpretation to work on a few examples. Revisiting (10.8), where
C1 encircles only z = 1,
!
dz
1
,
2i C1 (z 1)(z 3)
we note that the pole, z = 1, is simple and that the good part, 1/(z 3) is 1/2 there. Revisiting
(10.9),
!
dz
1
,
2i C1 (z 1)2 (z 3)
the pole z = 1 is now double and so we must evaluate the derivative of the good part, still 1/(z 3),
at z = 1, and nd 1/4. Next, we consider
!
1
exp(z)
dz, p = 1, 2, . . .
2i C(0,1) z p
and note that z = 0 is a pole of order p and so we must take p 1 derivatives of the good part,
exp(z), at z = 0, and nally divide by (p 1)!. Hence,
!
exp(z)
1
1
.
dz
=
2i C(0,1) z p
(p 1)!
One of the most useful applications of the Second Residue Theorem is the formula for the inverse
Laplace transform of a rational function.
10.3. The Inverse Laplace Transform and Return to Dynamics
If r is a rational function with poles {j }hj=1 then the inverse Laplace transform of r is
1
(L r)(t)
2i
1

!
r(z) exp(zt) dz
C

110

(10.15)

where C is a curve that encloses each of the poles of r. As a result

(L r)(t) =

res(r(z) exp(zt), j ).

(10.16)

j=1

Let us put this lovely formula to the test.

We take our examples from dynamaical systems of Chapter 8. Let us rst compute the inverse
Laplace Transform of
r(z) =

1
.
(z + 1)2

According to (10.16) it is simply the residue of r(z) exp(zt) at z = 1, i.e.,

d
exp(zt) = t exp(t).
z1 dz

res(r(z) exp(zt), 1) = lim

This closes the circle on the example begun in 8.3 and continued in Exer. 8.1. For our next example
we recall from (8.14) (ignoring the leading 1.965),
(s2 + 1.3s + 0.31)
(s + 1/4)2(s3 + 1.8s2 + 0.87s + 0.11))
(s2 + 1.3s + 0.31)
,
=
(s + 1/4)2(s + 1/2)(s + 1/5)(s + 11/10)

Lx1 (s) =

and so (10.16) dictates that

"
"
(s2 + 1.3s + 0.31)
d
"
x1 (t) = exp(t/4)
ds (s + 1/2)(s + 1/5)(s + 11/10) "s=1/4
"
"
(s2 + 1.3s + 0.31)
"
+ exp(t/2)
(s + 1/4)2(s + 1/5)(s + 11/10) "s=1/2
"
"
(s2 + 1.3s + 0.31)
"
+ exp(t/5)
(s + 1/4)2(s + 1/2)(s + 11/10) "s=1/5
"
"
(s2 + 1.3s + 0.31)
"
+ exp(11t/10)
2
(s + 1/4) (s + 1/2)(s + 1/5) "

(10.17)

s=10/11

Evaluation of these terms indeed conrms our declaration, (8.16), and the work in Exer. 7.9.
111

2.5
2
1.5
1
0.5

CL()
C ()
A

0
0.5
1
1.5
2
2.5
3

Figure 10.2 An illustration of the contour in (10.20) when = 2.

The curve replacement lemma of course gives us considerable freedom in our choice of the curve C
used to dene the inverse Laplace transform, (10.15). As in applications the poles of r are typically
in the left half of the complex plane (why?) it is common to choose C to be the half circle, see
Figure 10.2,
(10.18)
C = CL () CA (),
comprised of the line segment, CL , and arc, CA ,
CL () {i : } and CA () { exp(i) : /2 3/2},
where is chosen large enough to encircle the poles of r. With this concrete choice, (10.15) takes
the form
!
!
1
1
1
r(z) exp(zt) dz +
r(z) exp(zt) dz
(L r)(t) =
2i CL
2i CA
(10.19)
!
! 3/2

1
r(i) exp(it) d +
r( exp(i)) exp( exp(i)t) exp(i) d.
=
2
2 /2
Although this second term appears unwieldy it can be shown to vanish as , in which case
we arrive at
!
1
1
(L r)(t) =
r(i) exp(it) d,
(10.20)
2
the conventional denition of the inverse Laplace transform.
10.4. The Inverse Fourier Transform and the Causal Wiener Filter
Our interest is condiotions on f that assure causality, i.e.,
! 1
f () exp(2ki) d = 0
k = 1, 2, . . . .
0

Suppose f is a rational function of exp(2i), i.e.,

f () =

p(exp(2i))
q(exp(2i))
112

(10.21)

where p and q are polynomials. The change of variable z = exp(2i) will reduce (10.21) to a
residue problem. In particular, as dz = 2izd we nd
! 1
!
p(exp(2i))
p(z) k1
1
dz
exp(2ki) d =
z
2i C1 q(z)
0 q(exp(2i))
!
mj
h 

rj,m
dz
=
2i C1 (z j )m z k+1
j=1 m=1
where the last equality follows from the partial fraction expansion of p/q. We now note that if
|j | < 1 then the residues at z = 0 and z = j are equal and opposite
"
"
!
"
dm1 k1 ""
1
1 dk
1
m k1
m "
(z ) z
dz =
z
+
(z

)
"
"
2i C1
(m 1)! dz m1
k! dz k
z=
z=0
"
"
"
"
(m 1 + k)!
(m

1
+
k)!
m1 mk "
k
mk "
=
z
+
(1)
(1) (z )
"
"
(m 1)!k!
k!(m 1)!
z=
z=0
(m 1 + k)!
{(1)m1 + (1)m }mk = 0.
=
k!(m 1)!
This provides the clue to revising the IIR lter (9.39) a
() = cxy ()/
cxx(). The obvious guess of
simply keeping the good part is too naive. Instead we establish the
Proposition 10.6. If f () > 0 and integrable and is a rational function of exp(2i) then f may
be factored as
f () = L(exp(2i))L(exp(2i))
(10.22)
where all poles of L lie strictly inside the unit disk.
Proof: To begin

$n

j=1 (exp(2i)

j )

j=1 (exp(2i)

j )

f () = c $d

where c = 0 and no j nor j is zero. Also, no j has magnitude 1, for this would make f
nonintegrable. And no j has magnitude 1, for f never vanishes.
As f is real we equate conjugates and nd
$n
$n
j=1 (exp(2i) j )
j=1 (exp(2i) j )
= c $d
c $d
j=1 (exp(2i) j )
j=1 (exp(2i) j )
$n
j=1 (1/j exp(2i))j
= c exp(2i(d n)) $d
j=1 (1/j exp(2i))j
and so for pole j we nd that 1/j is also a pole. As such we may partition the poles into
those lying within the unit circle {+
j : j = 1, . . . , d/2} and their reections across the unit circle
+

{1/j : j = 1, . . . , d/2}. Similar, we partition the zeros into those lying within the unit circle
+
{+
j : j = 1, . . . , n/2} and their reections across the unit circle {1/j : j = 1, . . . , n/2}. With this
we may dene
$n/2
+
j=1 (exp(2i) j )
,
L() $d/2
+
(exp(2i)

)
j
j=1
113

note that its zeros and poles lie strictly inside the unit circle and compute
$n/2
+
+
j=1 (exp(2i) j )(exp(2i) j )
L()L() = $d/2
+
+
j=1 (exp(2i) j )(exp(2i) j )
$n/2
+
+
+
j=1 (exp(2i) j )(1/j exp(2i))j
= exp(i(d n)) $d/2
+
+
+
j=1 (exp(2i) j )(1/j exp(2i))j
$n/2 +
j=1 j f ()
(nd)/2
.
= exp(i(d n))(1)
$d/2 +
c

j=1 j
As both the left hand side and f are nonnegative we can take magnitudes of both sides and conclude
that
" $n/2
"
+ "
"
" j=1 j "
L()L() = f () where = " $d/2 + "
"c
j "
j=1

On setting L = L/ we conclude (10.22). End of Proof.

We assume that cxx is rational and exploit its factorization in the the associated error spectrum,
recall (9.38), takes the form
()
cxy () a
()
cxy () + |a()|2L(exp(2i))L(exp(2i))
c () = cyy () a
cxy ()|2/cxx ()
= cyy () + |a()L(exp(2i)) cxy /L(exp(2i))|2 |
Now, as we wish to determine the best causal a we expect a
to have all of its poles within C1 .
As the poles of L are also there then the best we can do is to match the causal part of scaled
cross-spectrum. More precisely, we suppose that cxy is also rational then use partial fractions to
write


%
%
cxy
cxy
cxy
+
=
L(exp(2i))
L(exp(2i)) +
L(exp(2i))
as a sum of terms with poles within and without (respectively) of C1 . With this notation we may
solve the causal Wiener lter for rational spectra via
%

cxy
1
.
(10.23)
a() =
L(exp(2i)) L(exp(2i))
+

10.5. Further Applications of the Second Residue Theorem

We show how to structure an integrand so that the Second Residue Theorem returns the number
of zeros of a pair of functions within a chosen region. This in turn leads to a beautiful comparison
theorem that permits us to equate the number of zeros of a hard function with that of an easy
function. This in turn permits us to prove the Fundamental Theorem of Algebra, i.e., the statement
that every nth order polynomial has n zeros. And nally, by a very similar argument we prove a
useful perturbation result. Namely, we show that the zeros of a polynomial dont move drastically
when one perturbs the polynomial coecients. We begin with the zero counter.
114

Proposition 10.7. The Argument Principle. If r = f /g is the ratio of two dierentiable

functions on and in the simple closed curve C and neither f nor g have zeros on C then
! 
1
r (z)
dz = Z(f, C) Z(g, C),
(10.24)
2i C r(z)
where Z(f, C) is the number (counting multiplicity) of zeros of f in C.
Proof: From r(z) = f (z)/g(z) comes
g(z)f  (z) f (z)g  (z)
r  (z)
=
r(z)
f (z)g(z)
and so each pole of r  /r is a zero of either f or g. We take these up separately.
If is a zero of f of order k then r(z) = (z )k q(z) where q() = 0. It follows that r  (z) =
k(z )k1q(z) + (z )k q  (z) and so
r  (z)
k
q  (z)
=
+
.
r(z)
z
q(z)
As the latter term is well behaved at z = it follows that res(r /r, ) = k.
If is a zero of g of order m then r(z) = (z )m p(z) where p() = 0. It follows that
r  (z) = m(z )m1 p(z) + (z )m p (z) and so
r  (z)
m
p (z)
=
+
.
r(z)
z
p(z)
As the latter term is well behaved at z = it follows that res(r  /r, ) = m.
Combining these two residue calculations, the Second Residue Theorem delivers (10.24). End of
Proof.
From this we establish the very useful comparison principle.
Proposition 10.8. Rouch
es Theorem. If f and g are two complex dierentiable functions on
and in the simple closed curve C and
|f (z) g(z)| < |g(z)|

zC

(10.25)

then f and g have the same number of zeros in C.

Proof: We dene r f /g and deduce from (10.25) that r has neither zeros nor poles on C. As
such we may read from the Argument Principle that
! 
1
r (z)
dz = Z(f, C) Z(g, C),
2i C r(z)
and so it remains to show that this integral is zero. To wit, we note that
!
r  (z)
1
dz = Z(f + tg, C) Z(g, C)
F (t) =
2i C r(z) + t
is both integer valued and continuous in t. This implies that F (t) is constant. This constant reveals
itself on noting that
|C| max{|r (z)| : z C}
|F (t)|
.
2 t max{|r(z)| : z C}
115

This implies that F (t) 0 as t . Hence, 0 = F (0) = Z(f, C) Z(g, C) as claimed. End of
Proof.
This has many subtle applications, perhaps the simplest being
Proposition 10.9. Fundamental Theorem of Algebra. If f (z) is a polynomial of degree n
the f has precisely n zeros.
Proof: From f (z) = f0 + f1 z + + fn z n we construct g(z) = fn z n and C = C(0, R) where
1 
|fj |.
R=1+
|fn | j=0
n1

It now follows from the triangle inequality that for z C,

|f (z) g(z)| = |f0 + f1 z + + fn1 z

n1

|R

n1

n1


|fj | < |fn |Rn = |fn z n | = |g(z)|.

j=0

It now follows from Rouches Theorem that f and g have the same number of zeros in C. As g has
n zeros there then so too does f . End of Proof.
By a very similar argument we can show that the roots of polynomial are continuous functions
of its coecients. We will use this result to build a quantitative perturbation theory in Chapter 12.
Proposition 10.10. Suppose f (, z) = f0 () + f1 ()z + f2 ()z 2 + + fn ()z n where each fj is a
continuous complex function of the complex parameter in some ball about = 0. If is a zero of
order k of z  f (0, z) then there exists an > 0 and 0 > 0 such that z  f (, z) has precisely k
zeros in C(, ) when || < 0 .
Proof: Pick > 0 so that no other zeros of z  f (0, z) lie in C = C(, ) and record
F = min |f (0, z)| and R = 1 + max |z|.
zC

zC

As each fj is continuous we may choose an 0 small enough to guarantee that

max |fj () fj (0)| <
j

F
,
(n + 1)Rn

|| < 0 .

This permits us to establish the bound

|f (, z) f (0, z)|

|fj () fj (0)||z j | < F |f (0, z)|,

j=0

and so conclude from Rouches Theorem that z  f (, z) has the same number of zeros in C as
z  f (0, z) for each || < 0 . End of Proof.
10.6. Exercises
1. Compute the integral of z 2 along the parabolic segment z(t) = t + it2 as t ranges from 0 to 1.
116

2. Evaluate each of the integrals below and state which result you are using, e.g., The bare
handed calculation (10.2), Cauchys Theorem, The Cauchy Integral Formula, The Second
Residue Theorem, and show all of your work.
!
!
!
cos(z)
cos(z)
cos(z)
dz,
dz,
dz,
C(2,1) z 2
C(2,1) z(z 2)
C(2,1) z(z + 2)
!
!
!
cos(z)
cos(z)
z cos(z)
dz,
dz.
dz,
3
3
z
C(0,2) z + z
C(0,2)
C(0,2) z 1
3. Compute the Rj,k per (10.6) for the B in (9.45). Conrm that they agree with those appearing
in (9.50).
4. Choose C in the Cauchy Integral Formula to be the circle of radius centered about a dened
in (10.1) and deduce from (10.11) the beautiful Mean Value Theorem
! 2
1
f (a) =
f (a + exp(it)) dt.
(10.26)
2 0
Conrm this for f (z) = z m , where m is a positive integer, by computing both sides of (10.26)
by hand for arbitrary a and .
5. Use (10.16) to compute the inverse Laplace transform of 1/(s2 + 2s + 2).
6. Use the result of the previous exercise to solve, via the Laplace transform, the dierential
equation
x (t) + x(t) = exp(t) sin t,
x(0) = 0.
Hint: Take the Laplace transform of each side.
7. Evaluate all expressions in (10.17) in Matlab s symbolic toolbox via syms, diff and subs
and conrm that the nal result jibes with (8.16).
8. Return to 6.6 and argue how one deduces (8.30) from (8.29). Evaluate (8.30) and graph it
with ezplot and contrast it with Fig. 6.3.
9. Let us check the limit we declared in going from (10.19) to (10.20). First show that
| exp( exp(i)t)| = exp(t cos ).
Next show (perhaps graphically) that
cos 1 2/

when /2 .

Now conrm each step in

"
"!
! 3/2
"
" 3/2
"
"
r( exp(i)) exp( exp(i)t) exp(i) d" max |r( exp(i))|
| exp( exp(i)t)| d
"

"
" /2
/2
!
exp(t cos ) d
= max |r( exp(i))|2

/2
!
exp(t(1 2/)) d
max |r( exp(i))|2

/2

= max |r( exp(i))|(/t)(1 exp(t)),

117

and nally argue why

max |r( exp(i))| 0
as .

118

11. The Eigenvalue Problem

Harking back to our discussion of dynamics, in Chapter 8 we labeled the complex number an
eigenvalue of B if I B was not invertible. In order to nd such one has only to nd those s
for which (sI B)1 is not dened. To take a concrete example we note that if

1 1 0
B = 0 1 0
(11.1)
0 0 2
then the GaussJordan method delivers
(sI B)1

(s 1)(s 2)
s2
0
1

0
(s 1)(s 2)
0
=
(s 1)2 (s 2)
0
0
(s 1)2

(11.2)

and so 1 = 1 and 2 = 2 are the two eigenvalues of B. Now, to say that j I B is not invertible
is to say that its columns are linearly dependent, or, equivalently, that the null space N (j I B)
contains more than just the zero vector. We call N (j I B) the jth eigenspace and call each of
its nonzero members a jth eigenvector. With respect to the B in (11.1) we note that


1
0
0 and 0
(11.3)
0
1
respectively span N (1I B) and N (2 I B). That B R33 but possesses only 2 linearly
independent eigenvectors suggests that matrices can not necessarily be judged by the number of
their eigenvectors. After a little probing one might surmise that Bs condition is related to the fact
that 1 is a double pole of (sI B)1 . In order to esh out that remark and nd a proper replacement
for the missing eigenvector we must take a much closer look at the resolvent. We conduct that in
the rst three section, arriving at the Spectral Representation, the most signicant and general
Proposition of the second half of the text. In the remaining two sections we provide alternate
interpretations, with examples, of this general result and nally return to dynamics and oer four
distinct pathways to constructing and understanding the matrix exponential, exp(Bt).
11.1. The Resolvent
One means by which to come to grips with (sI B)1 is to treat it as the matrix analog of the
scalar function
1
.
(11.4)
sb
This function is a scaled version of the even simpler function 1/(1 z). This latter function satises
(recall the n-term geometric series (9.11))
zn
1
2
n1
+
= 1+z +z ++z
1z
1z

(11.5)

for each positive integer n. Furthermore, if |z| < 1 then z n 0 as n and so (11.5) becomes,
in the limit,


1
zn ,
=
1z
n=0
119

the full geometric series. Returning to (11.4) we write

1/s
1
b
bn1 bn 1
1
=
= + 2 ++ n + n
,
sb
1 b/s
s s
s
s sb
and hence, so long as |s| > |b| we nd,
1
1
=
sb
s n=0

 n
b
.
s

This same line of reasoning may be applied in the matrix case. That is,
(sI B)1 = s1 (I B/s)1 =

1 B
B n1 B n
+ 2 + + n + n (sI B)1 ,
s s
s
s

(11.6)

and hence, so long as s is larger than any element of B, e.g., if |s| > BF where the latter is
dened in (1.17), we nd


(B/s)n .
(11.7)
(sI B)1 = s1
n=0

Although (11.7) is indeed a formula for the resolvent you may, regarding computation, not nd
it any more attractive than the Gauss-Jordan method. We view (11.7) however as an analytical
rather than computational tool. More precisely, it facilitates the computation of integrals of R(s).
For example, if C is the circle of radius centered at the origin and > B then
!
!


1
n
(sI B) ds =
B
s1n ds = 2iI.
(11.8)
C

n=0

Lets check this on the concrete resolvent in (11.2). For > 2 we nd indeed that
# ds #

ds
0
!
2
1
0
0
C s1
C
(s1)

# ds
1
1
0
(sI B)1 ds =
0
= 0 1 0 .
C s1
#
2i C
2i
ds
0 0 1
0
0
C s2

(11.9)

This result is essential to our study of the eigenvalue problem. As are the two resolvent identities.
Regarding the rst we deduce from the simple observation
(s2 I B)1 (s1 I B)1 = (s2 I B)1 (s1 I B s2 I + B)(s1 I B)1
that
R(s2 ) R(s1 ) = (s1 s2 )R(s2 )R(s1 ).

(11.10)

The second identity is simply a rewriting of

(sI B)(sI B)1 = (sI B)1 (sI B) = I,
namely,
BR(s) = R(s)B = sR(s) I.
You may wish to conrm that our concrete resolvent, (11.2), indeed obeys these identities.
120

(11.11)

The GaussJordan method informs us that R(s) will be a matrix of rational functions of s,
with a common denominator. In keeping with the notation of the previous chapters we assume the
denominator to have the h distinct roots, {j }hj=1 with associated orders {pj }hj=1 , and recall that
(10.6) produced
R(s) =

pj
h 

j=1 k=1

Rj,k
(s j )k

where Rj,k

1
=
2i

!
R(z)(z j )k1 dz.

(11.12)

Cj

In the previous 2 chapters we took a fairly pedestrian, element-wise approach and evaluated these
as matrices of integrals. For example, the resolvent in (11.2) has coecients

# ds #

ds
0
2
1
0
0
C1 s1
C
(s1)
# 1 ds
1
= 0 1 0
0
(11.13)
R1,1 =
C1 s1
# 0 ds
2i
0 0 0
0
0
C s2
1

and similarly
R1,2

0 1 0
= 0 0 0
0 0 0

and R2,1

0 0 0
= 0 0 0 .
0 0 1

As noted already, these matrices enjoy some amazing properties, e.g.,

2
R1,1
= R1,1 ,

2
R2,1
= R2,1 ,

2
and R1,2
= 0.

R1,1 R2,1 = 0,

(11.14)

To establish that such structure inheres to the coecients in the partial fraction expansion of the
resolvent of every matrix we must step back from an element-wise focus on the Rj,k and instead
manipulate them as integrals of matrices. To begin, lets show that each Rj,1 is a projection.
k

Ck

z
j

Cj

Figure 11.1. The curves that gure in Propositions 11.13.

2
Proposition 11.1. Rj,1
= Rj,1 .

Proof: Recall that the Cj appearing in (11.12) is any circle about j that neither touches nor
encircles any other root. Suppose, as in Figure 11.1, that Cj and Cj are two such circles and Cj
encloses Cj . Now, by the curve replacement lemma,
!
!
1
1
R(z) dz =
R(w) dw
Rj,1 =
2i Cj
2i Cj
121

and so
2
Rj,1

1
=
(2i)2

R(z) dz
Cj

1
=
(2i)2

Cj

!
Cj

Cj

R(w) dw

R(z)R(w) dw dz

R(z) R(w)
dw dz
wz
Cj Cj
%
!
!
!
!
1
1
1
dw dz
dz dw
R(z)
R(w)
=
(2i)2
Cj
Cj w z
Cj
Cj w z
!
1
=
R(z) dz = Rj,1 .
2i Cj
1
=
(2i)2

We used the rst resolvent identity, (11.10), in moving from the second to the third line. In moving
from the fourth to the fth we used only
!
Cj

1
dw = 2i and
wz

!
Cj

1
dz = 0.
wz

(11.15)

The latter integrates to zero because Cj does not encircle w. End of Proof.
Recalling Denition 6.1 that matrices that equal their squares are projections we adopt the
abbreviation
Pj Rj,1.
With respect to the proof that Pj Pk = 0 when j = k, the calculation runs along the same lines.
The dierence comes in (11.15) where, regarding Figure 11.1, as Cj lies completely outside of Ck ,
both integrals are zero. Hence,
Proposition 11.2. If j = k then Pj Pk = 0.
Along the same lines we dene
Dj Rj,2
and prove that Dj is nilpotent.
p

Proposition 11.3. If 1 k pj 1 then Djk = Rj,k+1. Dj j = 0.

122

Proof: For k and  greater than or equal to one and Cj and Cj as in Figure 11.1,
!
!
1
k
R(z)(z j ) dz
R(w)(w j ) dw
Rj,k+1Rj,+1 =
2
(2i) Cj
Cj
! !
1
=
R(z)R(w)(z j )k (w j ) dw dz
(2i)2 Cj Cj
! !
R(z) R(w)
1
(z j )k (w j ) dw dz
=
2
(2i) Cj Cj
wz
!
!
(w j )
1
k
dw dz
=
R(z)(z

)
j
(2i)2 Cj
wz
Cj
!
!
1
(z j )k


dz dw
R(w)(w j )
(2i)2 Cj
wz
Cj
!
1
=
R(z)(z j )k+ dz = Rj,k++1.
2i Cj
because

!
Cj

(w j )
dw = 2i(z j )
wz

!
and
Cj

(z j )k
dz = 0.
wz

(11.16)

2
With k =  = 1 we have shown Rj,2
= Rj,3 , i.e., Dj2 = Rj,3 . Similarly, with k = 1 and  = 2 we nd
Rj,2 Rj,3 = Rj,4, i.e., Dj3 = Rj,4, and so on. Finally, at k = pj we nd
!
1
pj
Dj = Rj,pj +1 =
R(z)(z j )pj dz = 0
2i Cj

by Cauchys Theorem. End of Proof.

Of course this last result would be trivial if in fact Dj = 0. Note that if pj > 1 then
!
pj 1
= Rj,pj =
R(z)(z j )pj 1 dz = 0
Dj
Cj

for the integrand then has a term proportional to 1/(z j ), which we know, by (10.2), leaves a
nonzero residue.
With this we have now arrived at a much richer specication of the generic expansion (11.12),
namely

%
pj 1
h


1
1
R(z) =
Pj +
Dk ,
(11.17)
k+1 j
z

(z

)
j
j
j=1
k=1

along with verication of a number of the properties enjoyed by the Pj and Dj .

11.2. The Spectral Representation
With just a little bit more work we shall arrive at a similar expansion for B itself. We begin by
applying the second resolvent identity, (11.11), to Pj . More precisely, we note that (11.11) implies
123

that

!
1
BPj = Pj B =
(zR(z) I) dz
2i Cj
!
1
zR(z) dz
=
2i Cj
!
!
j
1
R(z)(z j ) dz +
R(z) dz
=
2i Cj
2i Cj

(11.18)

= Dj + j P j ,
where the second equality is due to Cauchys Theorem and the third arises from adding and subtracting j R(z). Summing (11.18) over j we nd
B

Pj =

j=1

j Pj +

j=1

Dj .

(11.19)

j=1

We can go one step further, namely the evaluation of the rst sum. This stems from (11.8) where
we integrated R(s) over a circle C where > B. The connection to the Pj is made by the residue
theorem. More precisely,
!
h

R(z) dz = 2i
Pj .
C

j=1

Comparing this to (11.8) we nd

h


Pj = I,

(11.20)

j=1

and so (11.19) takes the form

B=

j Pj +

j=1

Dj .

(11.21)

j=1

It is this formula that we refer to as the Spectral Representation of B. To the numerous

connections between the Pj and Dj we wish to add one more. We rst write (11.18) as
(B j I)Pj = Dj

(11.22)

and then raise each side to the kth power to arrive at

(B j I)k Pj = Djk ,

(11.23)

where weve used the fact that Pj2 = Pj and BPj = Pj B. With k = pj in (11.23) we arrive at the
lovely
(B j I)pj Pj = 0.
(11.24)
For this reason we call the range of Pj the jth generalized eigenspace, call each of its nonzero
members a jth generalized eigenvector. The completion of a basis of eigenvectors to a basis of
generalized eigenvectors will follow from the following nesting property.
Proposition 11.4. N ((j I B)k ) R(Pj ) for k = 1, . . . , pj , and N ((B j I)pj ) = R(Pj ).
124

Proof: The key is the second resolvent identity, (11.11). For starting with k = 1 we nd that
if e N (B j ) then Be = j e and (11.11) reveals that R(s)Be = j R(s)e = sR(s)e e, which
upon simple rearrangement brings
1
R(s)e =
e.
(11.25)
s j
and so

1
Pj e =
2i

1
R(s)e ds =
2i
Cj

!
Cj

1
e ds = e.
s j

Regarding k = 2 we note that if x N ((B j I) ) then Bx = j x + e for some e N (B j I).

The second resolvent identity applied to x now reveals R(s)Bx = j R(s)x + R(s)e = sR(s)x x,
and upon rearrangement and recalling (11.25)
2

R(s)x =

1
1
e+
x.
2
(s j )
s j

(11.26)

Upon integrating this around Cj we nd indeed that Pj x = x. If y N ((B j I)3 ) then By =

j y + x for some x N ((B j I)2 ). The second resolvent identity applied to y now reveals
R(s)By = j R(s)y + R(s)x = sR(s)y y, and upon rearrangement and recalling (11.26)
R(s)y =

1
1
1
e+
x+
y,
3
2
(s j )
(s j )
s j

and integrating this around Cj brings Pj y = y. The pattern is now clear. Finally, if x R(Pj )
then (11.24) reveals that (B j I)pj x = 0, i.e., x N ((B j I)pj ). End of Proof.
This result suggests that the pole order, pj , does not tell the full story. The paucity of eigenvectors
associated with j is instead signied by the dierence between the geometric multiplicity,
nj dim N (B j I)

(11.27)

mj dim R(Pj ).

(11.28)

and the algebraic multiplicity

The previous proposition establishes that nj mj .
With regard to the example, (11.1), with which we began the chapter we note that n1 = 1 < 2 =
m1 and that the two eigenvectors in (11.3) may be completed by the second column of the associated
P1 (see (11.13)). We will see that there is a canonical means of completing the eigenvectors.
However, prior to developing that we look at the much easier, and in fact typical case, where each
mj = nj .
11.3. Diagonalization of a Semisimple Matrix
If pj = 1 then we call j semisimple. If each pj = 1 we call B semisimple. Our rst observation
is that each nilpotent vanishes in the semisimple case, i.e.,
B=

j Pj .

(11.29)

j=1

Our rst objective is to construct a concrete alternative to this beautiful but perhaps overly concise
representation.
125

As each pj = 1 it follows from Proposition 11.4 that the algebraic and geometric multiplicities
coincide, i.e., mj = nj . It then follows from (11.20) that these multiplicities sum to the ambient
dimension. i.e.,
h

nj = n.
(11.30)
j=1

We then denote by Ej = [ej,1 ej,2 ej,nj ] a matrix composed of basis vectors of R(Pj ). We note
that
Bej,k = j ej,k ,
and so
BE = E where E = [E1 E2 Eh ]

(11.31)

and is the diagonal matrix of eigenvalues,

= diag(1 ones(n1 , 1) 2 ones(n2 , 1) h ones(nh , 1)).
It follows from (11.30) that E is square and has linearly independent columns. As such, it is
invertible and so we have established
Proposition 11.5. If B is semisimple then there exists an invertible matrix, E, of eigenvectors of B,
and a diagonal matrix, , of eigenvalues of B (repeated according to their geometric multiplicities),
such that
B = EE 1 and = E 1 BE.
(11.32)
In this sense we say that E diagonalizes B. Let us work out a few examples and a number of
striking consequences. The resolvent of the rotation matrix


0 1
B=
(11.33)
1 0
has the partial fraction expansion
(sI B)

1
=
si





1
1/2 i/2
1/2 i/2
+
i/2 1/2
s + i i/2 1/2

We recognize that the two eigenvalues are simple poles of the resolvent, and note by inspection that
R(P1 ) and R(P2 ) are spanned by
 
 
1
1
and e2 =
,
e1 =
i
i
respectively. We lay these into E = [e1 , e2 ] and conrm that


i 0
1
E BE = =
0 i
as claimed. For our second example, the resolvent

B= 2
0
126

of the Clement matrix

1 0
0 2
1 0

(11.34)

has the partial fraction expansion

1 1 1
1 0 1
1 1 1
1/4
1/2
1/4
2 2 2 +
0 0 0 +
2 2 2
(sI B)1 =
s+2
s
s

2
1 1 1
1 0 1
1 1 1
and we read o the eigenvectors from the rst columns of the respective projections



1
1
1
e1 = 2 , e2 = 0 , and e3 = 2 .
1
1
1
We lay these into E = [e1 , e2 , e3 ] and conrm that

2 0 0
E 1 BE = = 0 0 0 .
0 0 2
Regarding functional consequences of our two representations
B = EE 1 =

j Pj ,

(11.35)

j=1

we note, regarding the rst and using only E 1 E = I, that

B 2 = BB = EE 1 EE 1 = E2 E 1 ,
and so, taking higher powers
B k = Ek E 1 .

(11.36)

This is worth boxing because B k is in general expensive to compute and dicult to interpret,
while k is the diagonal matrix of the kth powers of the eigenvalues of B. Regarding the second
representation in (11.35) we note, using Pi Pj = ij Pj that
 h
 h

h



2
j Pj
i Pi =
2j Pj ,
B =
j=1

i=1

j=1

and so, for higher powers,

k

B =

kj Pj .

(11.37)

j=1

As each of these representations reduces powers of the matrix to powers of (scalar) eigenvalues we
may take familiar sums of these scalar powers and arrive at new functions of the original matrix.
In particular, recalling the power series denition, (9.12),


1
(Bt)k .
exp(Bt)
k!
k=0

127

(11.38)

In light of (11.36) this becomes


1
E(t)k E 1 = E
exp(Bt) =
k!
k=1


(t)k /k! E 1 = E exp(t)E 1

(11.39)

k=1

where exp(t) is the diagonal matrix

exp(t) = diag(exp(1 t)ones(n1 , 1) exp(2 t)ones(n2 , 1) exp(h t)ones(nh , 1)).
In like fashion we draw from (11.37) the representation
exp(Bt) =

exp(j t)Pj .

(11.40)

j=1

As the matrix exponential is of such fundamental importance to the dynamics of linear systems we
pause to develop (in fact complete) yet a third representation. It begins from the partial fraction
expansion of the resolvent of a semisimple matrix
(zI B)

h

j=1

Pj
.
z j

(11.41)

Recalling our discussion of the Backward Euler Method in 8.3 we now evaluate
lim (I (t/k)B)k .

To nd this we note that (zI B)1 = (I B/z)1 /z and so (11.41) may be written
(I B/z)

h
h


zPj
Pj
=
=
z j
1 j /z
j=1
j=1

(11.42)

If we now set z = k/t, where k is a positive integer, and use Pi Pj = ij Pj we arrive rst at
(I (t/k)B)

h

j=1

Pj
(1 (t/k)j )k

(11.43)

and so, in the limit nd

lim (I (t/k)B)k =

exp(j t)Pj = exp(Bt).

(11.44)

j=1

Although each of these representations have stemmed naturally from their scalar formulations we
should still check that they do indeed resolve the dynamics problem. We carry this out for the
representation (11.40).
 d

d 
d
exp(Bt) =
exp(j t)Pj =
exp(j t)Pj =
j exp(j t)Pj
dt
dt j=1
dt
j=1
j=1
h

h

i=1

i Pi

exp(j t)Pj = B exp(Bt).

j=1

128

For the rotation matrix, (11.33), representation (11.40) yields




 

1/2 i/2
1/2 i/2
cos(t) sin(t)
exp(Bt) = exp(it)
+ exp(it)
=
.
i/2 1/2
i/2 1/2
sin(t) cos(t)
While for the Clement matrix, (11.34), we nd

1 1 1
1 0 1
1 1 1
exp(2t)
1
exp(2t)
2 2 2 + 0 0 0 +
2 2 2
exp(Bt) =
4
2
4
1 1 1
1 0 1
1 1 1

1 0 1
cosh(2t) sinh(2t) cosh(2t)
1
1

0 0 0 +
2 sinh(2t) 2 cosh(2t) 2 sinh(2t)
=
2 1 0 1
2 cosh(2t) sinh(2t) cosh(2t)

(11.45)

(11.46)

11.4. The Schur Form and the QR Algorithm

In general, B is nonsemisimple and there is at least one pj > 1, and so there are too few
eigenvectors with which to construct a diagonalizing similarity transformation. In this case however
there still exist triangularizing similarity transformations. The diagonal of the resulting triangular
matrix will indeed be comprised of the eigenvalues of B. There are two standard ways of developing
this triangularization; the Schur form is simple to construct but typically delivers a full upper
triangle while the Jordan form is dicult to construct but connes the upper triangle to a single
superdiagonal. We start with the former.
Proposition 11.6. For B Cnn there exists a unitary Q Cnn and an upper triangular
U Cnn such that
Q AQ = U.
(11.47)
Proof: The proof is by induction on n. For n = 1 the proof is trivial since B is a scalar in this
case.
Now assume that for any Bn1 C(n1)(n1) there exists a unitary Qn1 C(n1)(n1) such
that Qn1 Bn1 Qn1 is upper triangular.
Now suppose Bx1 = 1 x1 . Suppose x1  = 1 and denote by {x2 , . . . , xn } an orthonormal basis
for the orthogonal complement of x1 . The matrix X = [x1 , x2 , . . . , xn ] is therefore unitary and


1 y

X BX =
0 Bn1
where y Cn1 and Bn1 C(n1)(n1) . From the induction hypothesis there exists a unitary
Qn1 C(n1)(n1) such that Qn1 Bn1 Qn1 is upper triangular. Choose


1
0
Q=X
0 Qn1
and note that Q is unitary and





1
0
1
0

Q BQ =
X BX
0 Qn1
0 Qn1





1
0
1 y
1
0
=
0 Qn1
0 Bn1
0 Qn1


1
y Qn1
=

0 Qn1 Bn1 Qn1

129

where the last matrix is an upper triangular matrix because Qn1 Bn1 Qn1 is upper triangular.
End of Proof.
Regarding examples, the Schur form is typically achieved by an ecient implementation of a
very simple iteration. Namely, compute the QR factorization of B, reverse the factors and repeat.
In symbols,
B1 = B,

[Q1 , R1 ] = qr(B1 ),

B2 = R1 Q1 ,

[Q2 , R2 ] = qr(B2 ),

...

(11.48)

and we repeat until Bk is triangular. It is far from easy to see why these iterates should be come
triangular. It is however easy to see that they are each similar to one another, for
Bk+1 = Rk Qk = QTk Qk Rk Qk = QTk Bk Qk .
Lets run through these steps for

16 12 8
4
4 2 7 6
.
B=
2
4 16 8
1 2 3 6

Its resolvent is
(sI B)1 =
where P1 = I and

(11.49)

1
1
P1 +
D1
s 10
(s 10)2

6 12 8
4
4 8 7 6

D1 =
2
4
6
8
1 2 3 4

and so 1 = 10 is the sole eigenvalue and p1 = 2. For practical purposes we have terminated (11.48)
when the Eigenerror, max |diag(Bk ) 10|, was less than 0.01. This required 1000 iterations, see
Figure 11.2, and resulted in the Schur form

10.0100 0.0298 10.4400 20.0570

0.0000 10.0000 3.4933
7.1777

B1000 =
0.0000 0.0000 10.0000 0.0205 .
0.0000 0.0000 0.0000
9.9900
1

10

Eigenerror

10

10

10

200

400

600

iteration

130

800

1000

Figure 11.2. The convergence of the naive qr algorithm, (11.48), to a Schur form of (11.49).
11.5. The Jordan Canonical Form
To the insucient number of eigenvectors we add carefully constructed generalized eigenvectors
from R(Pj ). As this space is lled out, recall Prop. 11.4, by null spaces of powers of (B j I),
we will construct Jordan bases for the R(Pj ) along the lines used in the Jordan decomposition of
nilpotents in 4.4.
As a warm up lets restrict ourselves to matrices B with a single distinct nonsemisimple eigenvalue,
i.e., h = 1 and p1 > 1. We rst record the dimensions
dj = dim N ((B 1 I)j ),

j = 1, . . . , p1 ,

and proceed to
cp

Step 1: construct a basis {vp11 , vp21 , , vp11 } for N ((B 1 I)p1 ) mod N ((B 1 I)p1 1 ), where
vpk1 is the coordinate vector for the kth pivot column of (B 1 I)p1 1 .
Step 2: Note that (B 1 I)vpj 1 N ((B 1 I)p1 1 ) and that {(B 1 I)vpj 1 : j = 1, . . . , cp1 } is linc 1
}.
early independent mod N ((B1 I)p1 2 ). We complete this to a basis by appending {vp11 1 , . . . , vpp111
Step 3: Repeat Step 2, with p1 replaced by p1 1.
On completion we arrange the basis vectors by increasing block size
X = {v11 , . . . , v1c1 , (B 1 I)v21 , v21 , . . . , (B 1 I)v2c2 , v2c2 , . . . ,
(B 1 I)i1 vi1 , . . . , vi1 , . . . , (B 1 I)i1 vici , , . . . , vici , . . .},
and observe the chain/block structure inherent in the columns of X. Namley, for the 1-blocks
Bxj = 1 xj ,

j = 1, . . . , c1 ,

and for 2-blocks,

Bxj = 1 xj

and Bxj+1 = 1 xj+1 + xj ,

j = c1 + 1, . . . , c1 + c2

and for 3-blocks,

Bxj = 1 xj ,

Bxj+1 = 1 xj+1 + xj ,

Bxj+2 = 1 xj+2 + xj+1 ,

j = c1 + 2c2 , . . . , c1 + 2c2 + c3

and so on. It follows that BX = XJ where J is the block diagonal matrix begining with c1 zeros,
then c2 blocks of size 2, then c3 blocks of size 3 up through cm blocks of size m. Each block has 1
along its diagonal and ones along its superdiagonal. The chain numbers, cj , are determined by the
null space dimensions, dj , precisely as in (4.14).
Lets take the concrete example of (11.49). The associated null spaces have dimensions d1 = 2
and d2 = 4 and so we expect to build 2 chains of length two. Regarding the pivot columns of
(B 1 I) we note, either directly or via (11.22), that (B 1 I) = D1 . As columns 1 and 3 are
pivot columns of D1 it follows from Step 1 in that


1
0

0
and v 2 = 0
v21 =
2
0
1
0
0
131

comprise a basis for N ((B 1 I)2 ) mod N (B 1 I). We then build

6 1 8 0
4 0 7 0

X = {(B 1 I)v21 , v21 , (B 1 I)v22 , v22 } =

2 0 6 1
1 0 3 0
and nd that

(11.50)

10 1 0 0
0 10 0 0

X 1 BX = J =
0 0 10 1 ,
0 0 0 10

as predicted.
In the general case we merely repeat this procedure for each eigenvalue. The basis vectors for
the distinct j and k are independent from another because Pj Pk = 0.
Proposition 11.6. If B Cnn then B is similar to a Jordan matrix J Cnn . In particular,
there exists an X (comprised of generalized eigenvectors of B) for which
X 1 BX = J.

(11.51)

The structure of J is determined by the eigenvalues of B, denoted 1 , 2 , . . . , h , their associated

orders as poles of the resolvent of B, denoted p1 , p2 , . . . , ph , and nally by their associated nullities
dj,k = dim N ((B k )j ) for k = 1, . . . , h and j = 1, . . . , pk . In particular, for each k = 1, . . . , h,
there will be d1,k Jordan blocks with k on the diagonal. Among these there will be cj,k blocks of size
j, where j = 1, . . . , pk and c:,k = Spk d:,k where Spk is the hanging chain matrix of (4.14). The sum
of the sizes of all Jordan blocks associated with k is mk = dim R(Pk ), the algebraic multiplicity of
k .
Let us construct the Jordan form of

3
1

B = 1

1
1
1

2
4
1
3
3
1
1

3 5 7 4 9
4 6 7 4 10

5 5 6 4 8

5 11 11 7 15 .

6 10 10 8 17
3 6 7 7 9
2 4 5 3 4

(11.52)

Its resolvent is
(sI B)1 =

1
1
1
1
1
1
P1 +
P2 +
P3 +
D2 +
D3 +
D32 ,
2
2
3
s1
s2
s3
(s 2)
(s 3)
(s 3)

where the eigenprojections, eigennilpotents

1 2 3 5 7 4
0
0
0
0 0 0

0
0
0 0 0
0

P1 = 1 2 3 5 7 4

1 2 3 5 7 4
0
0
0
0 0 0
0
0
0
0 0 0

and their nonzero powers are

9
2 2 3
5 7 4 9
0 0 1 2 2 1 3
0

0
0 0 0 1 1 1 2

3 4 2 5
9 , P2 = 1 1 2

2 3 2 4
9
1 1 1

0 0 1 2 2 1 3
0
0
0 0 0 1 1 1 2
132

and

0
1

D2 = 0

0
1
1

and

0
0

D3 = 0

0
0
0

0
0
0
0
0
0
0

0
1
1
0
0
1
1

0
2
2
0
0
2
2

0
3
3
0
0
3
3

0
4
4
0
0
4
4

0
1
0
1
1
0
0

0
1
1
1
2
1
0

0
1
1
1
2
1
0

0
1
1
1
2
1
0

0
2
2
0
0
2
2

0
5

0 ,

0
5
5

0
2

3
1
0

0
0

P3 = 0

0
0
0

0
1
0
1
1
0
0

0
0

and D32 = 0

0
0
0

0
1
1
1
2
1
0

0
0
0
0
0
0
0

0
2
1
3
3
2
1

0
0
0
0
0
0
0

0
2
1
3
3
2
1

0
1
1
2
2
2
1

0
3

5
3
1

0 0 0 0
1 1 1 1

0 0 0 0

1 1 1 1 ,

1 1 1 1
0 0 0 0
0 0 0 0

corresponding to the three eigenvalues, j = j, and their respective pole orders, pj = j. The Jordan
Form is fully determined by the sequence of dimensions
d1,1 = dim N (B I) = 1,
d1,2 = dim N (B 2I) = 2,

d2,2 = dim N ((B 2I)2 ) = 3,

d1,3 = dim N (B 3I) = 1,

d2,3 = dim N ((B 3I)2 ) = 2,

d3,3 = dim N ((B 3I)3 ) = 3.

In particular,
d1,1 = 1 species that there is one Jordan block associated with 1 . The size of the block is
p1 = 1.
d1,2 = 2 species that there are two Jordan blocks associated with 2 . Their sizes are d2,2 d1,2 = 1
and p2 = 2.
d1,3 = 1 species that there is one Jordan block associated with 3 . The size of the block is
p3 = 3.
As a result B is similar to

1 0 0 0 0 0 0
0 2 0 0 0 0 0

0 0 2 1 0 0 0

J = 0 0 0 2 0 0 0
(11.53)
0 0 0 0 3 1 0

0 0 0 0 0 3 1
0 0 0 0 0 0 3
and the tranformation matrix is X = [X1 , X2 , X3 ] where
1
X1 = v1,1
,

1
1
1
1
1
1
X2 = {v1,2
, (B2 I)v2,2
, v2,2
} and X3 = {(B3 I)2 v3,3
, (B3 I)v3,3
, v3,3
} (11.54)

133

1
Here v1,1
is a basis for N (B 1 I),

1
v1,1

1
0

0

= 1 ,

1
0
0

1
1
v2,2
is a basis for N ((B 2 I)2 ) mod N (B 2 I), and v3,3
is a basis for N ((B 3 I)3 ) mod
1
= P2 e2 where e2 is the
N ((B 3 I)2 ). Recalling Step 1 and (B 2 I)P2 = D2 we obtain v2,2
coordinate vector of the pivot column of D2 . As column 1 is the pivot column of D2 we nd


2
0
0
1


0
1

1
1
v2,2 = 1 and (B 2 I)v2,2 = 0 .


1
0
0
1
0
1

These two lie, by construction in N (B 2 I)2 . We recognize that

1
0

1

1
= 1
v1,2

0
0
1
completes the basis of N (B 2 I)2 .
1
= P3 e3 where e3 is the coordinate vector of the
Similarly, as (B 3 I)2 P3 = D32 we obtain v3,3
2
pivot column of D3 . As column 4 is the pivot column of D32 we nd



0
0
0
2
1
1



1
1
0

1
1
1
= 3 , (B 3 I)v3,3
= 1 and (B 3 I)2 v3,3
= 1 .
v3,3



3
2
1
2
1
0
1
0
0
These vectors arranged per (11.54) indeed deliver X 1 BX = J for the B and J of (11.52) and
(11.53).
As each matrix is similar to a triangular matrix with its eigenvalues, repeated by their algebraic
multiplicity, along the diagonal, we may use the product formulas for the determinant and trace to
establish
134

Corollary 11.7. In the language of Prop. 11.6,

det(B zI) =

h
&
(j z)mj

and tr(B) =

j=1

mj j .

(11.55)

j=1

The rst statement in (11.55) nally brings us to a concise formulation of the characteristic
polynomial,
h
&
(j z)mj
(11.56)
B (s) det(B zI) =
j=1

rst encountered in 8.2 during our introduction to dynamics.

Next let us return to the exponential. In the previous section we constructed three equivalent
representations. We here oer a fourth, motivated by the Cauchy Integral Formula of (10.11),
!
1
exp(Bt)
(zI B)1 exp(zt) dz
(11.57)
2i C
where C is a closed curve that encircles all of the eigenvalues of B. On recalling (10.15) we see
however that (11.57) is not really new, just a nal conrmation of our hunch, (8.22), that the
matrix exponential is the inverse Laplace transform of the resolvent. This formulation becomes
more concrete if we substitute the partial fraction expansion of the resolvent and perform, via the
second residue theorem, the requisite integrals. Namely


mj 1
h !
 exp(zt)
exp(zt)
1 
Pj +
D k dz
exp(Bt) =
2i j=1 Cj z j
(z j )k+1 j
k=1


m
1
j
h


tk k
D .
exp(j t) Pj +
=
k! j
j=1
k=1
With this representation we return to the example (11.49) and compute

1 + 6t 12t
8t
4t
4t 1 8t 7t
6t

exp(Bt) = exp(10t)(I + tD1 ) = exp(10t)

2t
4t
1 + 6t
8t
t
2t
3t 1 4t
We may use this to uncover exp(Jt) via

X exp(Bt)X 1

1
0
= exp(Jt) = exp(10t)
0
0

t
1
0
0

0
0
1
0

0
0

t
1

where X, the basis constructed in (11.50).

We next move onto to example (11.52),
exp(Bt) = exp(t)P1 + exp(2t)P2 + exp(3t)P3 + t exp(2t)D2 + t exp(3t)D3 + (t2 /2) exp(3t)D32
135

This is a bit too large to see, but we may still uncover its action via
X exp(BT )X 1 = exp(Jt)

exp(t)
0
0
0
0
0
0

0
exp(2t)
0
0
0
0
0

0
exp(2t) t exp(2t)
0
0
0

0
0
exp(2t)
0
0
0
= 0

0
0
0
0
exp(3t) t exp(3t) t2 exp(3t)/2

0
0
0
0
0
exp(3t)
t exp(3t)
0
0
0
0
0
0
exp(3t)

(11.58)

11.6. Exercises
1. Argue as in Proposition 11.1 that if j = k then Dj Pk = Pj Dk = 0.
2. Argue from (11.22) and Pj2 = Pj that Dj Pj = Pj Dj = Dj .
3. Let us consider a semisimple matrix with a multiple eigenvalue,

4 0 0
B = 1 2 3
2 4 10
(i) Find the partial fraction expansion of its resolvent. (ii) From the projections in (i) extract
three linearly independent eigenvectors and use these to diagonalize B.
4. Starting from the representation (11.57) use the second resolvent identity to conrm that
(exp(Bt)) = B exp(Bt).
5. Note that B has the same spectral representation as B (except for a change in sign in the
eigenvalues). Construct exp(Bt) and show that it is the inverse of exp(Bt).
6. Show that exp(B T t) = (exp(Bt))T .
7. Regarding (11.45), note that exp(Bt)T exp(Bt) = I. Show that this is a direct consequence of
B T = B.
8. Show that exp((A + B)t) = exp(At) exp(Bt) if and only if AB = BA.
9. We write a Jordan Block as J = I + D where D is nilpotent, say D m = 0 but D m1 = 0.
(i) Use (11.39) to express exp(Dt) as a nite sum. (ii) Use the previous exercise to develop
exp(Jt) = exp(It) exp(Dt) and compare your ndings to (11.58).
10. Use Corollary 11.7 to conclude that
det exp(B) = exp(tr(B)).
11. We would now like to argue, by example, that the
every circulant matrix. We call this matrix

2 8 6
4 2 8
B=
6 4 2
8 6 4
136

Fourier Matrix of Exer. 7.7 diagonalizes

4
6

8
2

circulant because each column is a shifted version of its predecessor. First compare the results
of eig(B) and F4 B(:, 1) and then conrm that
B = F4 diag(F4 B(:, 1))F4/4.
Why must we divide by 4? Now check the analogous formula on a circulant matrix in R55 of
your choice. Submit a marked-up diary of your computations.
12. Let us return exercise 6.7 and study the eigenvalues of B as functions of the damping d when
each mass and stiness is 1. In this case



2 1 0
0
I
B=
where S = 1 2 1 .
S dS
0 1 2
(i) Write and execute a Matlab program that plots, as below, the 6 eigenvalues of B as d
ranges from 0 to 1.1 in increments of 0.005.
2

1.5

imaginary

0.5

0.5

1.5

2
2.5

1.5

0.5

0.5

real

Figure 9.1. Trajectories of eigenvalues of the damped chain as the damping increased.
(ii) Argue that if [u; v]T is an eigenvector of B with eigenvalue then v = u and Su dSv =
v. Substitute the former into the latter and deduce that
Su =

2
u.
1 + d

(iii) Conrm,
from
Exercise
7.10,
that
the
eigenvalues
of
S
are

=
2
+
2, 2 = 2 and
1

3 = 2 2 and hence that the six eigenvalues of B are the roots of the 3 quadratics

dj (dj )2 4j
2
.
+ dj + j = 0, i.e., j =
2
Deduce from the projections in Exer. 7.10 the 6 associated eigenvectors of B.
(iv) Now argue that when d obeys (dj )2 = 4j that a complex pair of eigenvalues of B collide
on the real line and give rise to a nonsemisimple eigenvalue. Describe Figure 9.1 in light of
your analysis.

137

12. The Symmetric Eigenvalue Problem

In the previous chapter we arrived at the Spectral Representation of a general matrix, and
interpreted it as diagonal decomposition in the case that the matrix was semisimple. That condition
was fairly technical and far from easy to check and so we devote an entire chapter to a very large,
very important class of semisimple matrices, namely, symmetric matrices.
We show that if B = B T Rnn then each eigenvalue, j , is real, each eigenprojection, Pj , is
symmetric (and therefore orthogonal) and each eigennilpotent, Dj , vanishes. We saw a concrete
example of this in Exer. 9.11.
We next show how to construct an orthonormal basis for each R(Pj ), how to characterize the j
as extrema of Rayleigh quotients and how to estimate the j via powers of B. We then study how
eigenvalues move when the underlying matrix is perturbed and we close with an application to the
optimal damping of mechanical networks.
12.1. The Spectral Representation
We establish the three key properties, one at a time.
Proposition 12.1. If B is real and B = B T then the eigenvalues of B are real.
Proof: We suppose that and x comprise an eigenpair of B, i.e., Bx = x and show that = .
On conjugating each side of this eigen equation we nd
Bx = x and Bx = x

(12.1)

where we have used the fact that B is real. We now take the inner product of the rst with x and
the second with x and nd
xT Bx = x2

and xT Bx = x2 .

(12.2)

The left hand side of the rst is a scalar and so equal to its transpose
xT Bx = (xT Bx)T = xT B T x = xT Bx

(12.3)

where in the last step we used B = B T . Combining (12.2) and (12.3) we see
x2 = x2 .
Finally, as x > 0 we may conclude that = . End of Proof.
We next establish
Proposition 12.2. If B is real and B = B T then each eigenprojection, Pj , and each eigennilpotent,
Dj , is real and symmetric.
Proof: As the transpose of the inverse is the inverse of the transpose we learn that
{(sI B)1 }T = {(sI B)T }1 = (sI B)1
i.e., the resolvent of a symmetric matrix is symmetric. Hence

T
!
!
1
1
T
1
Pj =
(sI B) ds
=
{(sI B)1 }T ds
2i Cj
2i Cj
!
1
(sI B)1 ds = Pj .
=
2i Cj
138

By the same token, we nd that each DjT = Dj . The elements of Pj and Dj are real because they
are residues of real functions evaluated at real poles. End of Proof.
The next result will banish the nilpotent component.
Proposition 12.3. The zero matrix is the only real symmetric nilpotent matrix.
Proof: Suppose that D Rnn and D = D T and D m = 0 for some positive integer m. We show
that D m1 = 0 by showing that every vector lies in its null space. To wit, if x Rn then
D m1 x2 = xT (D m1 )T D m1 x
= xT D m1 D m1 x
= xT D m2 D m x
= 0.
As D m1 x = 0 for every x it follows (recall Exercise 4.3) that D m1 = 0. Continuing in this fashion
we nd D m2 = 0 and so, eventually, D = 0. End of Proof.
We have now established the key result of this chapter.
Proposition 12.4. If B is real and symmetric then
B=

j Pj

(12.4)

j=1

where the j are real and the Pj are real orthogonal projections that sum to the identity and whose
pairwise products vanish.
The bulk of the symmetric matrices constructed in Chapters 2 and 3 were also positive denite,
i.e., they obeyed
x Rn .
xT Bx > 0,
It follows that the eigenvalues of such a matrix are positive and this in turn permits us to construct
its square root,
Proposition 12.5. If B Rnn is symmetric and positive denite with eigenvalues, j , and
eigenprojections, Pj , then
h


B 1/2
j Pj
j=1

is symmetric and positive denite and obeys B 1/2 B 1/2 = B.

In applications one often is confronted with the so-called generalized eigenproblem
Su = Mu

(12.5)

where S and M are symmetric and M is positive denite. For example, the dynamics of the chain
Figure 3.1, with uniform stiness, k1 = k2 = k3 = k4 = 1, but nonuniform mass, mj = j requires
the solution of (12.5) with

2 1 0
1 0 0
S = 1 2 1 and M = 0 2 0 .
0 0 3
0 1 2
139

The eigenvalues in the this case are the poles of the generalized resolvent (S M)1 or the zeros
of the associated det(S M). In this case

4 + 10
4 10
1 =
, 2 = 1, 3 =
,
3
3
and the jth eigenvector spans N (S j M). Rather than producing an associated generalized
spectral representation based on this generalized resolvent we instead note that Prop. 12.5 permits
us to write (12.5) in standard form. Namely, from
Su = M 1/2 M 1/2 u
we multiply on the left by M 1/2 and set q = M 1/2 u and arrive at the standard symmetric eigenproblem,
(12.6)
Bq = q where B = M 1/2 SM 1/2 .
For the example above

2 1/ 2
0
B = 1/ 2
1 1/ 6 .
0
1/ 6
2/3

12.2. Orthonormal Diagonalization of a Symmetric Matrix

It follows, as in 11.3, that each real symmetric matrix, B, has a full set of eigenvectors that may
be used to diagonalize B, as in (11.32). As our Pj are in fact orthogonal projections we may now
argue that we may diagonalize B with an orthonormal transformation. If x R(Pj ) and y R(Pk )
then
xT y = (Pj x)T Pk y = xT PjT Pk y = xT Pj Pk y = 0.
and hence the individual eigenbases
Ej [ej,1 ej,2 . . . ej,nj ]
are orthogonal to one another. In the simple case where each nj = 1 we will have a full set of
orthogonal eigenvectors. For example, we return to Exer. 9.11 and note that eigenvectors may be
read from the columns of the associated projections. In particular

1
1

0
e1 = 2 ,
and e3 =
e2 =
2
1
1
1
are eigenvectors of

2 1 0
B = 1 2 1
0 1 2

associated with the eigenvalues 1 = 2 + 2, 2 = 2 and 3 = 2 2. The

ej are clearly orthogonal
to one another, and upon simple normalization, i.e., q1 = e1 /2, q2 = e2 / 2 and q3 = e3 /2 we note
that the matrix

1/2
1/
2
1/2

Q = [q1 , q2 , q3 ] = 1/ 2
0 1/ 2
1/2
1/ 2 1/2
140

enjoys both the lovely property QT Q = I and

0
2+ 2 0
2
0 ,
QT BQ = = 0
0
0 2 2

the diagonal matrix of eigenvalues. A matrix Q for which QT Q = I is called orthonormal. An

orthonormal square matrix is easily inverted, namely Q1 = QT . We have in fact proven, and
illustrated, that every symmetric real matrix with simple eigenvalues may be diagonalized by an
orthonormal matrix of eigenvectors.
So let us move toward the general case with the loop matrix

2 1 0 1
1 2 1 0

B=
(12.7)
0 1 2 1
1 0 1 2
and recognize in its resolvent
1
1
1
P2 +
P3 ,
(sI B)1 = P1 +
s
s2
s4
where

1
1
1
P1 =

4 1
1

1
1
1
1

1
1
1
1

1
1
,
1
1

1
0 1 0
1 0
1
0 1

P2 =
1
0
2 1 0
0 1 0
1

1 1 1 1
1 1 1 1 1

and P3 =
4 1 1 1 1
1 1 1 1

and eigenvalue 1 = 0 with a 1dimensional eigenspace, an eigenvalue 2 = 2 with a 2dimensional

eigenspace, and an eigenvalue 3 = 4 with a 1dimensional eigenspace. One easy way of seeing the
dimensions is via nj = mj = tr(Pj ). In any event, we proceed as above to read eigenvectors from
the rst nj columns of Pj . In particular, E = [E1 , E2 , E3 ] where

1
1
0
1
1
0

1 .
,
=
and
E
=
E1 =
E
2
3
1
1 0
1
0 1
1
1
As expected the EjT Ek = 0 when j = k. As a perhaps lucky bonus we recognize that the two
columns of E2 are also orthogonal to one another. As a result the normalized collection

Q = [E1 /2, E2 / 2, E3 /2]

is an orthonormal diagonalizer of the B in (12.7).
It remains only to show that we can always engineer such luck.
By constructing an orthogonal basis {qj,k : 1 k nj } for each R(Pj ), collecting the basis
vectors in Qj = [qj,1 qj,2 qj,nj ] and assembling the Qj into a single square orthonormal matrix
Q = [Q1 Qh ], we arrive at
B = QQT

and QT BQ =
141

(12.8)

where is the diagonal matrix of eigenvalues of B, ordered as in the construction of Q, and repeated
by their multiplicity.
Finally, let us return to the generalized eigenproblem, (12.5). Recall that and u are an eigenpair
for (S, M) if and only if and q = M 1/2 u are an eigenpair for B = M 1/2 SM 1/2 . The orthonormality of Q = (q1 , q2 , . . . , qn ) then implies that the matrix of (S, M) eigenvectors, U = (u1 , u2 , . . . , un ),
obeys
I = QT Q = (M 1/2 U T )M 1/2 U = U T MU,
(12.9)
i.e., U is orthonormal with weight matrix M.
12.3. Perturbation Theory
It is often the case in practice that one can solve the base instantiation of a problem but
wishes to know how the solution changes if one perturbs the problem. In our setting we ask how
the eigenvalues change when we change the underlying matrix. More precisely, given symmetric B
and C and a real we attempt to expand the eigenvalues of B + C in a Taylor series in . To
lessen the notation (and with an eye to future applications) we develop the theory for the greatest
eigenvalue
1 (B + C) = 1 (B) + d1 + 2 d2 /2 +
where our calculus suggests that dk should denote the kth derivative of 1 at B in the direction C.
First some examples. If C = I and Bq1 = 1 q1 then (B + I)q1 = (1 + )q1 and so
1 (B + I) = 1 (B) +
and so clearly d1 = 1 and the rest of dk = 0. Similarly, if C = B then
1 (B + B) = (1 + )1 (B) = 1 (B) + 1 (B)
and so d1 = 1 (B) and again the remaining dk = 0. We could generalize this to those C that are
multiples of I or B, but this still remains to sparse a sampling of the direction space. Our third
example exposes the key obstacle. The eigenvalues of




1 0
1 0
(12.10)
+
0 1
0 1
are 1 + and 1 which appear as banal as our two previous examples until we order them. For
if 1 is the largest and 2 the smallest we nd
1 (B + C) = 1 + || and 2 (B + C) = 1 ||.

(12.11)

Hence, a diagonal perturbation of a diagonal matrix can produce eigenvalues that do not vary
smoothly with . The culprit here is the multiplicity of 1 (I), for as it is 2 we nd that it splits
when I is perturbed. Although there are means to accomodate such splitting we will assume here
that 1 of the unperturbed matrix is simple.
To resolve the general case we return to our source and develop the resolvent
R(s, ) = (sI (B + C))1
142

in a Taylor series in . Namely,

R(s, ) = ((sI B) C(sI B)1 (sI B))1
= (sI B)1 (I C(sI B)1 )1
= R(s, 0)(I CR(s, 0))1


= R(s, 0)
j (CR(s, 0))j
j=0

= R(s, 0) + R(s, 0)CR(s, 0) + O(2).

Regarding the associated eigenprojection we note that 1 (B + C) is a pole of s  R(s, ). and a
zero of the characteristic polynomial, s  B (s, ) = det(sI (B + C)). This is a polynomial in
s with coecients that are polynomials in . It follows from Prop. 10.9 that there exists a small
circle, C1 , about 1 (B) that both includes 1 (B + C) and excludes j (B + C) for j > 1 and all
in some interval about 0. As a result, the perturbed eigenprojection is
!
1
R(s, ) ds
P1 (B + C) =
2i C1
!
1
=
(R(s, 0) + R(s, 0)CR(s, 0) + O(2 )) ds
2i C1
!
1
R(s, 0)CR(s, 0) ds + O(2 ).
= P1 (B) +
2i C1
We now turn to the partial fraction expansion of R(s, 0) to achieve
1
2i

1
R(s, 0)CR(s, 0) ds =
2i
C1
=

1
2i

 Pk (B)
Pj (B)
C
ds
s j (B)
s k (B)
h

C1 j=1

k=1

 Pk (B)
 Pj (B)
P1 (B)
P1 (B)
C
+
C
ds
s 1 (B)
s k (B) j=2 s j (B) s 1 (B)
h

C1

= P1 (B)C

k=2

h

k=2


Pk (B)
Pk (B)
+
CP1 (B).
1 (B) k (B)
1 (B) k (B)
h

k=2

It follows that
P1 (B + C) = P1 (B) + (P1 (B)CS1 + S1 CP1 (B)) + O(2)
where
S1

h

k=2

(12.12)

Pk (B)
,
1 (B) k (B)

is the socalled reduced resolvent. In order to derive the perturbation series for 1 (B + C) we take
the trace of each side of
(B + C)P1 (B + C) = 1 (B + C)P1 (B + C).
Namely
tr((B + C)P1(B + C)) = 1 (B + C)tr(P1 (B + C)) = 1 (B + C).
143

(12.13)

where we have used the fact that 1 (B + C) is simple and that the trace of a projection is its rank.
We now develop the left side of (12.13).
(B + C)P1 (B + C) = (B + C)(P1 (B) + (P1 (B)CS1 + S1 CP1 (B)) + O(2))
= 1 (B)P1 (B) + (CP1 (B) + B(P1 (B)CS1 + S1 CP1 (B))) + O(2)

(12.14)

We note that P1 (B)S1 = S1 P1 (B) = 0 will cause the S1 terms in (12.14) to vanish upon taking the
trace. In particular
tr(BP1 (B)CS1 ) = tr(BP1 (B)S1 C) = 0 and tr(BS1 CP1 (B)) = tr(BS1 P1 (B)C) = 0.
Hence, the trace of (12.14) reveals
1 (B + C) = tr((B + C)P1 (B + C)) = 1 (B) + tr(CP1 (B)) + O(2).

(12.15)

or, on recalling that P1 (B) = q1 q1T is just projection onto the rst eigenvector, that
Proposition 12.6. If B and C are symmetric and 1 (B), the greatest eigenvalue of B, is simple
then
1 (B + C) = 1 (B) + q1T Cq1 + O(2 ).
(12.16)
where q1 = q1 (B) is the associated eigenvector of B.
As a very simple example, you might wish to conrm that the largest eigenvalue of




2 1
2 0
+
1 2
0 0
behaves like 3 + + O(2). We will develop much richer examples in sections and exercises to come.
One such example will require an understanding the generalized perturbation problem
(S + X)u = Mu
where M is symmetric and positive denite. As with (12.5) we may return this to the standard
perturbation problem
(B + C)q = q
via q = M 1/2 u, B = M 1/2 SM 1/2 and C = M 1/2 XM 1/2 . In this case the key term in the
perturbation expansion is
q1T Cq1 = (M 1/2 u1)T M 1/2 XM 1/2 M 1/2 u1 = uT1 Xu1 .
As a result,
Proposition 12.7. If S, M and C are symmetric and M is positive denite and 1 (S, M), the
greatest eigenvalue of (S, M), is simple then
1 (S + X, M) = 1 (S, M) + uT1 Xu1 + O(2 ).
where u1 = u1 (S, M) is the associated eigenvector of (S, M).
12.4. Rayleighs Principle and the Power Method
144

(12.17)

As the eigenvalues of a symmetric matrix, B Rnn , are real we may order them from high to
low,
1 2 n .
In this section we will derive two extremely useful characterizations of the largest eigenvalue. The
rst was discovered by Lord Rayleigh in his research on sound and vibration. To begin, we denote
the associated orthonormal eigenvectors of B by
q1 , q2 , . . . , qn
and note that each x Rn enjoys the expansion
x = (xT q1 )q1 + (xT q2 )q2 + + (xT qn )qn .

(12.18)

Applying B to each side we nd

Bx = (xT q1 )1 q1 + (xT q2 )2 q2 + + (xT qn )n qn .

(12.19)

Now taking the inner product of (12.18) and (12.19) we nd

xT Bx = (xT q1 )2 1 + (xT q2 )2 2 + + (xT qn )2 n
1 {(xT q1 )2 + (xT q2 )2 + + (xT qn )2 }
= 1 xT x.
That is, xT Bx 1 xT x for every x Rn . This, together with the fact that q1T Bq1 = 1 q1T q1
establishes
Proposition 12.8. Rayleighs Principle. If B is symmetric then its largest eigenvalue is
xT Bx
1 = max T
x=0 x x
and the maximum is attained on the line through the associated eigenvector, q1 .
As Rayleighs Principle identies only the principle direction it is often presumed that the
maximum is taken over unit vectors, i.e., Rayleighs Principle is written
1 = max xT Bx.
xT x=1

(12.20)

As unit vectors in the plane are specially easy to write, namely x = x() = (cos(), sin())T , we
may visualize the Rayleigh quotient for 2by2 matrices. For example, if


2 1
B=
(12.21)
1 2
then x()T Bx() = 2 sin(2) clearly assumes its maximum of 3 at = /4 and its minimum
of 1 when = 3/4. Conrm that 3 and 1 are the eigenvalues of B and that their associated
eigendirections are /4 and 3/4.
We note by the same reasoning that
Proposition 12.9. Rayleighs Generalized Principle. If S is symmetric and M is symmetric
and positive denite then the largest eigenvalue of (S, M) is
1 = max uT Su
uT M u=1

145

and the maximum is attained on the line through the associated eigenvector, u1 .
For our next characterization we return to (12.19) and record higher powers of B onto x
B k x = (xT q1 )k1 q1 + (xT q2 )k2 q2 + + (xT qn )kn qn


xT q2 k2
xT qn kn
T
k
= (x q1 )1 q1 + T
q2 + + T
qn .
x q1 k1
x q1 k1

(12.22)

And so, using sign(t) t/|t|,

Bkx
= sign(k1 xT q1 )q1 + O((2 /1 )k )
k
B x
and note that the latter term goes to zero with increasing k so long as 1 is strictly greater than
2 . If, in addition, we assume that 1 > 0, then the rst term does not depend on k and we arrive
at
Proposition 12.10. The Power Method. If B is symmetric and its greatest eigenvalue is simple
and positive and the initial guess, x, is not orthogonal to q1 then
Bkx
= sign(xT q1 )q1
k B k x
lim

By way of illustration, we record B k x/B k x for the B of (12.21), a random x, and k = 1

through 7,

 
 
 
 
 
 

0.70866
0.71176
0.72098
0.74783
0.82030
0.95709
0.97404
.
,
,
,
,
,
,
0.70555
0.70242
0.69296
0.66389
0.57194
0.28980
0.22635

This is indeed approaching q1 = [1, 1]/ 2. As with the Rayleigh quotient it is a simple matter
to arrange things as to converge on the eigenvector associated with the smallest eigenvalue. To see
this we recall that 1/n is the largest eigenvalue of B 1 and hence B k x/B k x ought to converge
to qn . The associated method is called Inverse Iteration.
12.5. Return to Dynamics
We revisit our work of dynamics of mechanical systems. We close up loose ends, but this is not
mere reconciliation, we go considerably beyond. We consider the initial value problem for u(t) Rn
Mu (t) + Du (t) + Su(t) = 0,
u(0) =f, u (0) = g,

t > 0,

(12.23)

where  d/dt and M, D, and S are each symmetric and positive denite. It will be convenient
to consider an equivalent rst order system. In particular, with V (t) denoting the 2n x 1 vector
[u(t), u (t)] we nd that (12.23) becomes

 

f
0
I

V (t) = A(D)V (t), V (0) =
(12.24)
, A(D) =
g
M 1 S M 1 D
We suppose the mass, M, and stiness, S, to be prescribed and we venture to produce a damping
D for which the energy induced by the initial displacement, f , and velocity, g, is dissipated as
146

eciently as possible. The energy we have in mind is simply the sum of the kinetic and potential
instantaneous energies, namely
E(t) u(t)T Mu (t) + u(t)T Su(t).
A more natural measure of damping eciency is the total energy
!
E(t) dt.
0

Let us try to make their dependence on D more explicit. As V (t) = exp(A(D)t)V (0) it follows that


S 0
T
T
.
E(t) = V (0) exp(A(D) t)F exp(A(D)t)V (0) where F =
0 M
As a result, the total energy is
!
E(t) dt = V (0)T E(D)V (0),

!
where E(D) =

exp(A(D)T t)F exp(A(D)t) dt.

(12.25)

One obvious defect with this measure of damping eciency is its dependence on the initial state,
V (0) = [f, g]. We remedy this by maximizing the above over all initial states of unit energy. That
is, we consider the greatest total energy
1 (D)

max

(0)T F V

(0)=1

V (0)T E(D)V (0).

(12.26)

One recognizes in (12.26) Rayleighs Generalized Principle (Prop. 12.9) for the greatest eigenvalue
of the pair (E(D), F ). That is, 1 (D) is the greatest eigenvalue of the generalized problem
E(D)V = F V.

(12.27)

With E(D) as derived in (12.25) this remains a formidable eigenproblem. Over the next few pages we
will restrict the damping matrix to natural subspaces in which E(D) may be expressed as products
of the system matrices, M, K, D, and their inverses.
The most restricted class is that of so-called friction damping where we suppose the damping
to be proportional to the mass. The key to this entire section is the following energy equation.
Proposition 12.11. If D = 2aM then


d
1
T

T
E(t) =
u(t) Mu (t) + au(t) Mu(t) + E(t) .
dt
2a

(12.28)

Proof: As u satises (12.23) we nd

E  (t) = 2(Mu (t) + Su(t))T u = 4au (t)T Mu (t).
Continuing to use (12.23),
E(t) = u(t)T Mu (t) + u(t)T Su(t)
= u(t)T Mu (t) u(t)T Mu (t) 2au(t)T Mu (t)
= u(t)T Mu (t) {(u(t)T Mu (t)) u (t)T Mu (t)} a(u(t)T Mu(t))
1
= (u(t)T Mu (t)) a(u(t)T Mu(t)) E  (t),
2a
147

(12.29)

where in the nal line we have used (12.29). End of Proof.

Integrating E, this proposition yields

s=t
! t
1
T

T
E(s) ds = u(s) Mu (s) + au(s) Mu(s) + E(s)
2a
0
s=0
1
1
= f T Mg + af T Mf + E(0) u(t)T Mu (t) au(t)T Mu(t) E(t).
2a
2a
Now if a > 0 then u(t), u (t), and E(t) tend to zero as t . As a result,
Corollary 12.12. If D = 2aM then the total energy is
!
E(0)
E(t) dt =
+ f T M(af + g).
2a
0
If we now reconcile (12.30) with (12.25) we nd that


aM + S/(2a) M/2
E(2aM) =
.
M/2
M/(2a)

(12.30)

(12.31)

Hence, if V = [u, v] is an eigenvector of (E(2aM), F ) with eigenvalue then

aMu + Su/(2a) + Mv/2 = Su and Mu/2 + Mv/(2a) = Mv.
The latter requires that u = (2 1/a)v and this in turn reduces the former to
(4 2 4 /a + 1/a2 )Sv = (4a 1)Mv.
This states that v is an eigenvector of (S, M) and
4 2

4a 1
= j
4 /a + 1/a2

where j is an associated eigenvalue of (S, M). This quadratic in has the two roots

a a2 + j
1
.
+
2a
2j
1 (D), the largest of these roots, is attained at j = n for the least of the eigenvalues of (S, M). On
combining this with the previous corollary we nd
Corollary 12.13. If D = 2aM then the associated greatest total energy is

a + a2 + n
1
1 (2aM) =
+
.
2a
2n

(12.32)

We have graphed this function in Figure 12.1 for the simple choice n = 1. We note that 1
approaches innity for both small and large a. For later purposes we record the magnitude, a, for
which the greatest total energy is least,
'

51
,
(12.33)
a
= n
2
148

as well as the associated leading eigenvector of (E(2aM), F ),


V1 = qn (21 (2aM) 1/a, 1) = qn ((a + a
2 + n )/n , 1).

(12.34)

Greatest Total Energy, 1(2aM)

6
5.5
5
4.5
4
3.5
3
2.5
2
1.5

damping size, a

Figure 12.1. The greatest total energy for a system with friction damping D = 2aM.
This graph enjoys the nice property that it always lies below any chord connecting two of its
points. We call functions with such graphs convex, the precise denition being.
Denition 12.14. A subset C of a vector space is said to be convex if for each pair of points in C
the connecting chord also lies in C. That is, for each pair, c1 and c2 in C, the point tc1 +(1t)c2 C
for each 0 t 1. A real valued function, f , on a convex set, C, is said to be convex if, for each
pair, c1 and c2 in C, the graph of f lies below the chord, i.e.,
f (tc1 + (1 t)c2 ) tf (c1 ) + (1 t)f (c2 )

0 t 1.

For example, every subspace is convex. The intersection of convex sets is convex, and the set of
symmetric positive denite matrices is convex. Regarding functions, every linear function is convex
and the maximum of a family of convex functions is convex. These statements make nice exercises,
though the last is best proven here.
Proposition 12.15. If the function v  H(u, v) is convex for each u in some set U and
h(v) max H(u, v),
uU

then h is convex.
Proof: Given v1 and v2 and t [0, 1] we note that
H(u, tv1 + (1 t)v2 ) tH(u, v1 ) + (1 t)H(u, v2 ),

u U.

From the fact that the maximum of a sum is less than the sum of the maximums it follows that
max H(u, tv1 + (1 t)v2 ) t max H(u, v1 ) + (1 t) max H(u, v2),
uU

uU

uU

i.e., h(tv1 + (1 t)v2 ) th(v1 ) + (1 t)h(v2 ). End of Proof.

Our goal in the remainder of the section is to show that 1 is in fact convex over a larger class of
damping matrices and that (12.33) remains the best damping. We look rst for a more convenient
characterization of E(D). Recalling the energy equation, Prop. 12.11., we search for an X for which
E(t) =

d
V (t)T XV (t).
dt
149

(12.35)

Representing E directly in terms of V and computing the derivative on the right side requires of X
that
V (t)T F V (t) = V  (t)T XV (t) V (t)T XV  (t)
= V (t)T (XA(D) + AT (D)X)V (t).
Evaluating this expression at t = 0 and using the fact that the initial state, V (0), is perfectly
arbitrary we nd that X must satisfy the Liapunov equation
AT (D)X + XA(D) = F.

(12.36)

We will argue in the exercises that this equation possesses a unique symmetric positive denite
solution, that we naturally refer to as E(D). We note however that (12.36) may be solved explicitly
when D is drawn from the Caughy class, C, of symmetric positive denite matrices for which
S 1 DM 1 = M 1 DS 1 .
In particular, if D C then
E(D) =

1

1
D + SD 1 M
M
2
2
1
1
M
MD
M
2

(12.37)

(12.38)

is the desired solution of (12.36). We note that 2aM C and now proceed to study the greatest
eigenvalue of generalized perturbation problem (E(2aM + C), F ) for C C. From
(2aM + C)1 = ((I + C(2aM)1 )2aM)1
= (2aM)1 (I + C(2aM)1 )1


1
= (2aM)
()k (C(2aM)1 )k
= (2aM)

k=0

(2aM)1 C(2aM)1 + O(2)

we develop
E(2aM + C) = E0 + E1 + O()2


C/2 SM 1 C/(4a2 )
0
= E(2aM) +
+ O(2).
0
C/(4a2 )

(12.39)

We now have all the pieces required to establish that 2

aM is a critical point of 1 . That is,
Proposition 12.16. If n , the least eigenvalue of the undamped system, (S, M), is simple then
1 (2aM + C) = 1 (2aM) + O(2 )

(12.40)

for each C C.
Proof: If n is simple eigenvalue of (S, M) then 1 (2aM) is a simple eigenvalue of (E(2aM), F )
and we may deduce from Prop. 12.7 that
1 (2aM + C) = 1 (2aM) + V1T E1 V1 + O(2 )
150

where V1 was derived in (12.34) and E1 in (12.39). We rst compute

  a+a2 + 

1
2
n
C/2

SM
C/(4
a
)
0
qn
n
E1 V1 =
2
0
C/(4a )
qn

 a+a2 +  1

n
C 4a12 SM 1 C qn
n
2
.
=
4a12 Cqn


and then
V1T E1 V1

a
+

a
2 + n
n

2 

1
n
2
2 4a

1
2
4a

%
qnT Cqn = 0,

because the braced term vanishes thanks to (12.33). End of Proof.

The convexity of D  1 (D) will follow from Prop. 12.15 and our various maximum principles.
Proposition 12.17. The greatest total energy, D  1 (D), is convex for D C.
Proof: In light of Prop. 12.15 and the variational characterization, (12.26), of 1 (D), it will suce
to show that D  V T E(D)V is convex for each V R2n . We split V into its two ndimensional
components, V = [f g], and nd
V T E(D)V = 12 f T Df + f T SD 1Mf + f T Mg + g T MD 1 Mg.

(12.41)

As a nite sum of convex functions is convex it suces to show that each of the summands in (12.41)
is convex. The rst term is linear in D while the third is independent of D and so both are convex
in D. With respect to the second term we may deduce (Exer. 12.6) from D C that SD 1M is
symmetric and positive denite. As such, it follows from our earlier energy considerations, namely
Prop. 3.4, that
(12.42)
f T SD 1 Mf = maxn f T x xT M 1 DS 1 x.
xR

As 2f x x M DS x is convex in D, it follows from Prop. 12.15 and (12.42) that so too is

D  f T SD 1 Mf . In a similar fashion, from
T

g T MD 1 Mg = maxn 2xT Mg xT Dx,

xR

one deduces the convexity of D  g T MD 1 Mg. End of Proof.

It remains only to conrm that a critical point of a convex function is indeed a global minimizer.
=
Proposition 12.18. The greatest total energy, D  1 (D), attains its minimum over C at D
2aM.
then, as 1 is convex,
Proof: If D2 C and 1 (D2 ) < 1 (D)
+ (D2 D))
= 1 (D2 + (1 )D)
1 (D2 ) + (1 )1 (D)

1 (D
for arbitrary . Hence, for > 0,
+ (D2 D))
1 (D)

1 (D

1 (D2 ) 1 (D).

As 0 we have argued in (12.40) that the left side goes to zero, in contradiction to 1 (D2 ) < 1 (D).
End of Proof.
151

Let us apply this nding to the optimal damping of the chain in Figure 3.1. We assume uniform
masses, m, and stinesses, k, and seek that damping matrix that produces the least greatest total
energy. The respective mass, damping, and stiness matrices of the form

d11 d12 d13

1 0 0
2 1 0
M = m 0 1 0 , D = d12 d22 d23 , S = k 1 2 1 .
0 0 1
d13 d23 d33
0 1 2
This D has six degrees of freedom and one can associate these with 6 dashpots at and between the 3
masses in Figure 3.1. We rst restrict our choices to the Caughy Class by enforcing DS 1 = S 1 D.
This restricts us to the three dimensional class of damping matrices of the form

a2 a1 a3
a3
a1
a2 .
(12.43)
D = a2
a1 a3 a2
a3

The eigenvalues of this D are 2a3 a1 and a1 a2 2 hence D C so long as its parameters, a1 , a2
and a3 obey

2a3 > a1 > 2|a2 |.

We note that the least eigenvalue of (S, M) is simple and
3 =

k
(2 2).
m

Invoking Prop. 12.18 we nd that D  1 (D) attains its minimum over C at D = 2aM. In terms
of the parametrization (12.43) this requires


(12.44)
a1 = a3 = 2km(2 2)( 5 1) and a2 = 0.
12.6. Exercises
1. The stiness matrix associated with the unstable swing of gure 2.2 is

1 0 1 0
0 1 0 0

S=
1 0 1 0 .
0 0 0 1
(i) Find the three distinct eigenvalues, 1 = 1, 2 = 2, 3 = 0, along with their associated
eigenvectors e1,1 , e1,2 , e2,1 , e3,1 , and projection matrices, P1 , P2 , P3 . What are the respective
geometric multiplicities?
(ii) Use the folk theorem that states in order to transform a matrix it suces to transform its
eigenvalues to arrive at a guess for S 1/2 . Show that your guess indeed satises S 1/2 S 1/2 = S.
Does your S 1/2 have anything in common with the element-wise square root of S?
(iii) Show that R(P3 ) = N (S).
(iv) Assemble
1
1
S + = P1 + P2
1
2
152

and check your result against pinv(S) in Matlab .

(v) Use S + to solve Sx = f where f = [0 1 0 2]T and carefully draw before and after pictures
of the unloaded and loaded swing.
(vi) It can be very useful to sketch each of the eigenvectors in this fashion. In fact, a movie
is the way to go. Please run the Matlab truss demo by typing truss and view all 12 of the
movies. Please sketch the 4 eigenvectors of (i) by showing how they deform the swing.
2. The Cayley-Hamilton Theorem. Use Proposition 12.4 to show that if B is real and symmetric
and p(z) is a polynomial then
h

p(B) =
p(j )Pj .
(12.45)
j=1

Conrm this in the case that B is the matrix in Exercise 1 and p(z) = z 2 + 1. Deduce from
(12.45) that if p is the characteristic polynomial of B, i.e., p(z) = det(zI B), then p(B) = 0.
Conrm this, via the Matlab command, poly, on the S matrix in Exercise 1.
3. Argue that the least eigenvalue of a symmetric matrix obeys the minimum principle
n = min
x=0

xT Bx
xT x

(12.46)

and that the minimum is attained at qn .

4. The stiness of a ber net is typically equated with the least eigenvalue of its stiness matrix.
Let B be the stiness matrix constructed in Exercise 2 of 2.5, and let C denote the stiness
matrix of the same net, except that the stiness of ber 4 is twice that used in constructing
B. Show that xT Bx xT Cx for all x R4 and then deduce from (12.46) that the stiness of
the C network exceeds the stiness of the B network.
5. The Power Method. Submit a Matlab diary of your application of the Power Method to the
S matrix in Exercise 1.
6. Use (12.37) to show that (12.38) is symmetric and and positive denite and a solution to
(12.36).
7. Prove that the Caughy class, C, is convex.
8. One general approach to the Lyapunov Equation,
AX + XA = S

(12.47),

is to begin with the Schur decomposition, A = U T U. For then U T UX + XU T U = S is

simplied via left multiplication by U and right multiplication by U , to
T B + BT = C,

where B = UXU

and C = USU .

(12.48)

This triagular system can be solved by forward substitution. To wit, xamine the (1,1) element
and nd
t1,1 b1,1 + b1,1 t1,1 = c1,1
and next the (1,2) element and nd
t1,1 b1,2 + b1,1 t1,2 + b1,2 t2,2 = c1,2 .
153

Generalize and deduce that so long as no two eigenvalues of A are the negative conjugates of
one another then (12.47) possesses a unique solution X SPD.
9. Prove that the maximum of a sum is less than or equal to the sum of the maximums. More
precisely, given to real valued functions, f and g, on some set U, prove that
max(f (u) + g(u)) max f (u) + max g(u).
uU

uU

154

uU

13. The Singular Value Decomposition

13.1. Introduction
The singular value decomposition is, in a sense, the spectral representation of a rectangular
matrix. Of course if A is m-by-n and m = n then it does not make sense to speak of the eigenvalues
of A. We may, however, rely on the previous section to give us relevant spectral representations of
the two symmetric matrices
AT A and AAT .
That these two matrices together may indeed tell us everything about A can be gleaned from
N (AT A) = N (A),
R(AT A) = R(AT ),

N (AAT ) = N (AT ),
and R(AAT ) = R(A).

You have proven the rst of these in a previous exercise. The proof of the second is identical. The
row and column space results follow from the rst two via orthogonality.
On the spectral side, we shall now see that the eigenvalues of AAT and AT A are nonnegative
and that their nonzero eigenvalues coincide. Let us rst conrm this on the A matrix associated
with the unstable swing (see gure 2.2)

0 1 0 0
A = 1 0 1 0 .
(13.1)
0 0 0 1
The respective products are

1 0 0
AAT = 0 2 0
0 0 1

0
and AT A =
1
0

0 1 0
1 0 0
.
0 1 0
0 0 1

Analysis of the rst is particularly simple. Its null space is clearly just the zero vector while 1 = 2
and 2 = 1 are its eigenvalues. Their geometric multiplicities are n1 = 1 and n2 = 2. In AT A we
recognize the S matrix from exercise 12.1 and recall that its eigenvalues are 1 = 2, 2 = 1, and
3 = 0 with multiplicities n1 = 1, n2 = 2, and n3 = 1. Hence, at least for this A, the eigenvalues of
AAT and AT A are nonnegative and their nonzero eigenvalues coincide. In addition, the geometric
multiplicities of the nonzero eigenvalues sum to 3, the rank of A.
Proposition 13.1. The eigenvalues of AAT and AT A are nonnegative. Their nonzero eigenvalues,
including geometric multiplicities, coincide. The geometric multiplicities of the nonzero eigenvalues
sum to the rank of A.
Proof: If AT Ax = x then xT AT Ax = xT x, i.e., Ax2 = x2 and so 0. A similar
argument works for AAT .
nj
Now suppose that j > 0 and that {xj,k }k=1
constitutes an orthogonal basis for the eigenspace
R(Pj ). Starting from
AT Axj,k = j xj,k
(13.2)
we nd, on multiplying through (from the left) by A that
AAT Axj,k = j Axj,k ,
155

i.e., j is an eigenvalue of AAT with eigenvector

Axj,k , so long as Axj,k = 0. It follows from the

rst paragraph of this proof that Axj,k  = j , which, by hypothesis, is nonzero. Hence,
Axj,k
yj,k ,
j

1 k nj

(13.3)

is a collection of unit eigenvectors of AAT associated with j . Let us now show that these vectors
are orthonormal for xed j.
T
yj,i
yj,k =

1 T T
x A Axj,k = xTj,i xj,k = 0.
j j,i

We have now demonstrated that if j > 0 is an eigenvalue of AT A of geometric multiplicity nj

then it is an eigenvalue of AAT of geometric multiplicity at least nj . Reversing the argument, i.e.,
generating eigenvectors of AT A from those of AAT we nd that the geometric multiplicities must
indeed coincide.
Regarding the rank statement, we discern from (13.2) that if j > 0 then xj,k R(AT A). The
union of these vectors indeed constitutes a basis for R(AT A), for anything orthogonal to each of
these xj,k necessarily lies in the eigenspace corresponding to a zero eigenvalue, i.e., in N (AT A). As
R(AT A) = R(AT ) it follows that dim R(AT A) = r and hence the nj , for j > 0, sum to r. End of
Proof.
Let us now gather together some of the separate pieces of the proof. For starters, we order the
eigenvalues of AT A from high to low,
1 > 2 > > h
and write
AT A = Xn X T

(13.4)

where
X = [X1 Xh ],

where Xj = [xj,1 xj,nj ]

and n is the n-by-n diagonal matrix with 1 in the rst n1 slots, 2 in the next n2 slots, etc.
Similarly
(13.5)
AAT = Y m Y T
where
Y = [Y1 Yh ],

where Yj = [yj,1 yj,nj ].

and m is the m-by-m diagonal matrix with 1 in the rst n1 slots, 2 in the next n2 slots, etc. The
yj,k were dened in (13.3) under the assumption that j > 0. If j = 0 let Yj denote an orthonormal
basis for N (AAT ). Finally, call

j = j
and let denote the m-by-n matrix diagonal matrix with 1 in the rst n1 slots and 2 in the next
n2 slots, etc. Notice that
(13.6)
T = n and T = m .
Now recognize that (13.3) may be written
Axj,k = j yj,k
156

and that this is simply the column by column rendition of

AX = Y .
As XX T = I we may multiply through (from the right) by X T and arrive at the singular value
decomposition of A,
A = Y X T .
(13.7)
Let us conrm this on the A matrix

2 0 0
0 1 0
4 =
0 0 1
0 0 0
and

Hence,

in (13.1). We have

1 0
0
0

2 0
0
and X = 1 0

1
0 0
0
2
0
0
0
2

2 0 0
3 = 0 1 0
0 0 1

1
0

1
0

0 1 0
and Y = 1 0 0 .
0 0 1

2 0 0 0
= 0 1 0 0
0 0 1 0

and so A = Y X T says that A should coincide with

1/ 2
0 1 0
2 0 0 0
1 0 0 0 1 0 0 0
0

0 0 1
0 0 1 0
1/ 2

(13.8)

0 1/ 2
1
0
0
0
0 1/ 2

0
0
.
1
0

This indeed agrees with A. It also agrees (up to sign changes in the columns of X) with what one
receives upon typing [Y,SIG,X]=svd(A) in Matlab .
You now ask what we get for our troubles. I express the rst dividend as a proposition that
looks to me like a quantitative version of the fundamental theorem of linear algebra.
Proposition 13.2. If Y X T is the singular value decomposition of A then
(i) The rank of A, call it r, is the number of nonzero elements in .
(ii) The rst r columns of X constitute an orthonormal basis for R(AT ). The n r last columns
of X constitute an orthonormal basis for N (A).
(iii) The rst r columns of Y constitute an orthonormal basis for R(A). The m r last columns of
Y constitute an orthonormal basis for N (AT ).
Let us now solve Ax = b with the help of the pseudoinverse of A. You know the right thing
to do, namely reciprocate all of the nonzero singular values. Because m is not necessarily n we must
also be careful with dimensions. To be precise, let + denote the n-by-m matrix whose rst n1
diagonal elements are 1/1 , whose next n2 diagonal elements are 1/2 and so on. In the case that
h = 0, set the nal nh diagonal elements of + to zero. Now, one denes the pseudo-inverse of
A to be
A+ X+ Y T .
157

Taking the A of (13.1) we nd

1/ 2

0
+ =
0
0

0
1
0
0

0
0

1
0

and so

1/ 2
0

A+ =
1/ 2
0

0 1/2
1
0
=
0 1/2
0
0

0
1
0
0

0 1/ 2
1/ 2
0
0
0

0 1/ 2 0
0
1
0

0
1
0
0

0
0 1 0

0
1 0 0
1
0 0 1
0

0
0

0
1

in agreement with what appears from pinv(A). Let us now investigate the sense in which A+ is the
inverse of A. Suppose that b Rm and that we wish to solve Ax = b. We suspect that A+ b should
be a good candidate. Observe now that
(AT A)A+ b = Xn X T X+ Y T b
+

because X T X = I

= Xn Y b
= XT + Y T b
T

by (13.6)

because T + = T

= X Y b
= AT b

by (13.4)

by (13.7),

that is, A+ b satises the least-squares problem AT Ax = AT b.

13.2. The SVD in Image Compression
Most applications of the SVD are manifestations of the folk theorem, The singular vectors associated with the largest singular values capture the essence of the matrix. This is most easily seen
when applied to gray scale images. For example, the jpeg associated with the image in the top
left of gure 11.2 is 262-by-165 matrix. Such a matrix-image is read, displayed and decomposed by
M = imread(jb.jpg); imagesc(M); colormap(gray);
[Y,S,X] = svd(double(M));
The singular values lie on the diagonal of S and are arranged in decreasing order. We see their
rapid decline in Figure 13.1.
158

10

singular value

10

10

10

10

20

40

60

80

100

120

140

160

index

Figure 13.1. The singular values of John Brown.

We now experiment with quantitative versions of the folk theorem. In particular, we examine the
result of keeping but the rst k singular vectors and values. That is we construct
Ak = Y(:,1:k)*S(1:k,1:k)*X(:,1:k);
for decreasing values of k.

Figure 13.2. The results of imagesc(Ak) for, starting at the top left, k=165, 64, 32 and moving
right, and then starting at the bottom left and moving right, k=24,20,16.
159

13.3. Low Rank Approximation

In this section we establish a precise version of last sections folk theorem. To prepare the way we
note that if B = B T Rmm and we list its eigenvalues, including multiplicities, {1 , 2 , . . . , m }
then, from Corollary 11.7 it follows that
m


tr(B) =

j (B).

(13.9)

i=1

This together with our trace formulation of the Frobenius norm of A Rmn , recall (1.18), yields
A2F

= tr(AA ) =

i (AA ) =

i=1

i2 ,

i=1

i.e., the Frobenius norm of a matrix is the square root of the sum of the squares of its singular
values. We will also need the fact that if Q is square and QT Q = I then
QA2F = tr(QAAT QT ) = tr(AAT QT Q) = tr(AAT ) = A2F .

(13.10)

The same argument reveals that AQF = AF . We may now establish
Proposition 13.3. Given an m-by-n matrix A (with SVD A = Y X T ) and a whole number
k min{m, n} then the best (in terms of Frobenius distance) rank k approximation of A is
Ak = Y (:, 1 : k)(1 : k, 1 : k)X(:, 1 : k)T .
The square of the associated approximation error is
A Ak 2F =

min

rank(B)=k

A B2F =

j2 .

j>k

Proof: If B is m-by-n then

A B2F = Y X T B2F = Y ( Y T BX)X T 2F =  Y T BX2F .
If B is to be chosen, among matrices of rank k, to minimize this distance then, as is diagonal, so
too must Y T BX. If we denote this diagonal matrix by S then
Y T BX = S

implies B = Y SX T ,

and so
A B2F =


(j sj )2 .
j

As the rank of B is k it follows that the best choice of the sj is sj = j for j = 1 : k and sj = 0
there after.
End of Proof.
13.4. Exercises
1. Suppose that A is m-by-n and b is m-by-1. Set x+ = A+ b and suppose x satises AT Ax = AT b.
Prove that x+  x. (Hint: decompose x = xR + xN into its row space and null space
+
+
+
components. Likewise x+ = x+
R + xN . Now argue that xR = xR and xN = 0 and recognize that
you are almost home.)
160

2. Experiment with compressing the bike image below (also under Resources on our Owlspace
page). Submit labeled gures corresponding to several low rank approximations. Note:
bike.jpg is really a color le, so after saving it to your directory and entering Matlab
you might say M = imread(bike.jpg) and then M = M(:,:,1) prior to imagesc(M) and
colormap(gray).

161

14. Reduced Dynamical Systems

We saw in the last chapter how to build low rank approximations of matrices using the Singular
Value Decomposition, with applications to image compression. We devote this chapter to building
reduced models of dynamical systems from variations of the power method, with applications to
neuronal modeling.
14.1. The Full Model
We consider distributed current injection into passive RC cable of Chapter 8. We suppose it has
length  and radius a and we choose a compartment length dx and arrive at n = /dx compartments.
The cells axial resistivity is R. Its membrane capacitance and conductance are Cm and GL .
GL

GL

Cm

u1

Cm

v1

Cm

u2

v2

u3

v3

G
Cm

un

vn

Figure 14.1. The full RC cable.

If uj is the synaptic current injected at node j and vj is the transmembrane potential there then
(arguing as in Chapter 8) v obeys the dierential equation
Cv  (t) + Gv(t) = u(t),

(14.1)

where

a2
Sn ,
Rdx
In is the n-by-n identity matrix and Sn is the n-by-n seconddierence matrix

1 1 0 0 0
1 2 1 0 0

Sn =

0 0 1 2 1
0 0 0 1 1
C = (2adxCm )In ,

G = (2adxGL )In

(14.2)

We divide through by 2adxCm and arrive at

v  (t) = Bv(t) + g(t)
where

a
Sn (GL /Cm )In and g = u/(2adxCm ).
(14.3)
2RCm dx2
We will construct a reduced model that faithfully reproduces the response at a specied compartment (the putative spike initiation zone), assumed without loss to be the rst,
B=

y(t) = q T v(t),

where q T = (1 0 0 0),

to an arbitrary stimulus, u. On applying the Laplace Transform to each side of (14.1) we nd

sLv = BLv + Lg
162

and so the output

Ly = q T Lv = q T (sI B)1 Lg H(s)Lg

may be expressed in terms of the transfer function

H(s) q T (sI B)1 .
14.2. The Reduced Model
We choose a reduced dimension k and an n k orthonormal reducer X, suppose v = X v and
note that v is governed by the reduced system
v (t) = X T BX v(t) + X T g(t),
y(t) = q T X v(t),
We choose X so that the associated transfer function,

H(s)
= q T X(sIk X T BX)1 X T ,
is close to H. We will see that it suces to set
X(:, 1) = B 1 q/B 1 q,
X(:, j + 1) = B 1 X(:, j)/B 1 X(:, j),

j = 1, . . . , k 1,

(14.4)

followed by orthonormalization,
X = orth(X).
We note that (14.4) is nothing more than the Power Method applied to B 1 , and so as j increases the
associated column of X approaches the (constant) eigenvector associated with the least eigenvalue

of B. We apply this algorithm and then return to prove that it indeed makes H close to H.

Soma Potential (mV)

15

10

full, N=100
reduced, k=3

0
0

10

20

30

40

50

60

t (ms)

Figure 14.2. Response of the rst compartment in a 100 compartment cell and the corresponding
3 compartment reduced cell to identical current injections distributed randomly in space and time.
cabred.m
Now, in order to understand why this method works so well we examine the associated transfer
functions. In particular, we examine their respective Taylor series about s = 0. To begin, we note
163

that the rst resolvent identity, (11.10), permits us to evaluate

R(s + h) R(s)
(h)R(s + h)R(s)
= lim
= R2 (s).
h0
h0
h
h

R (s) = lim
and so

R (s) = 2R(s)R (s) = 2R3 (s),

R (s) = 6R2 (s)R (s) = 6R4 (s),

and in general
dj R(s)
= (1)j j!Rj+1 (s) and so
dsj
and so
H(s) = q T R(s) =

sj Mj ,

dj R(0)
= j!(B 1 )j+1
dsj

where Mj = q T (B 1 )j+1

j=0

are the associated moments. The reduced transfer function is analogously found to be

H(s)
=

j = q T X((X T BX)1 )j+1 X T .

where M

j,
sj M

(14.5)

j=0

j = Mj for 0 j < k then H(s)

= H(s) + O(sk ) and so we now show that the reducer built
If M
according to (14.4) indeed matches the rst k moments.
j = Mj for 0 j < k.
Proposition 14.1. If (B 1 )j+1 q R(X) for 0 j < k then M
T , beginning with j = 0 where
Proof: We will show that MjT = M
j
M0T = B 1 q

T = X(X T BX)1 X T q.
and M
0

We note that P = X(X T BX)1 X T B obeys P 2 = P and P X = X and proceed to develop

T = X(X T BX)1 X T q
M
0
= X(X T BX)1 X T BB 1 q
= P B 1q = B 1 q = M0T ,
where the penultimate equality follows from the assumption that B 1 q R(X) and the fact that
P acts like the identity on X. When j = 1 we nd
T = X(X T BX)1 (X T BX)1 X T q
M
1
= X(X T BX)1 X T BB 1 X(X T BX)1 X T BB 1 q
= P B 1 P B 1 q = (B 1 )2 q = M1T
where the penultimate equality follows from the assumption that (B 1 )2 q R(X). The remaining
moment equalities follow in identical fashion.
End of Proof.
164

14.3. Exercises
1. Show that the n-by-n second dierence Sn in (14.2) obeys
x Sn x =
T

n1


(xj xj+1 )2

j=1

for every x Rn . Why does it follow that the eigenvalues of Sn are nonpositive? Why does
it then follow that eigenvalues of the B matrix in (14.3) are negative? If we then label the
eigenvalues of B as
n (B) n1 (B) 2 (B) 1 (B) < 0,
which eigenvector of B does (14.4) approach as j increases?
2. Please derive the Taylor expansion displayed in (14.5).
3. Please conrm that X(X T BX)1 X T B is indeed projection onto R(X).

165