Matrix Computations

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS
TIMO EIROLA
Mat-1.3651 Lecture notes
0.75
0.5
0.25
-0.75
-0.5
-0.25
0.25
0.5
0.75
-0.25
-0.5
-0.75
Aalto University School of Science timo.eirola@aalto.fi

April 26, 2013
Contents 1. Introduction 1.1. Notation and basic properties 2. Linear system Ax = b , elimination, LU decomposition. 2.1. LU decomposition 2.2. Conditioning 2.3. Partial pivoting 2.4. Full pivoting 2.5. The LDL decomposition and the Choleskydecomposition 3. Projections 3.1. Oblique projections 4. Least squares problems, QRdecomposition 4.1. Householder transformation 4.2. Givens rotations 4.3. Uniqueness of Q R 4.4. Pseudoinverse 4.5. Other unitary decompositions 5. Eigendecompositions 5.1. Schur decomposition 5.2. The Sylvester equation 5.3. Jordan form 6. Numerical computation of eigenvalues 6.1. Sensitivity of eigenvalues 6.2. Computation of eigenvalues, QRiteration 1 3 8 8 11 13 15 15 17 18 20 21 23 25 26 26 28 29 32 35 38 38 44
ii
EIROLA
7. Eigenvalues of a Hermitian matrix, singular value decomposition 7.1. Eigenvalues of a Hermitian matrix 7.2. Inertia 7.3. Singular value decomposition 8. Computation of eigenvalues of a Hermitian matrix 8.1. Hermitian QR iteration 8.2. Jacobi iteration 8.3. Divide and conquer method 8.4. Computation of the singular value decomposition 9. Krylov subspace iterations for eigenvalue problems 9.1. Lanczos iteration for a Hermitian operator 9.2. Bi-orthogonal Lanczos 9.3. Arnoldi iteration 10. Classical iterations for linear systems 10.1. Jacobi and GaussSeidel iterations 10.2. SOR iteration (overrelaxation) 10.3. SSOR (Symmetric SOR) 10.4. Tshebysheviteration 11. The conjugate gradient method 11.1. Minimization in the Krylov subspace 11.2. Convergence of the conjugate gradient method 11.3. Preconditioned conjugate gradients 12. Krylov subspace iterations for nonsymmetric systems 12.1. Generalized minimal residual method (GMRes)
51 51 55 56 60 60 61 62 63 66 68 72 74 80 82 84 89 91 96 96 100 101 104 104
iii
12.2. Biconjugate gradient method 12.3. Quasiminimal residual method (QMR) 12.4. Biconjugate gradient squared 12.5. On convergence 13. Preconditioning 13.1. ILU preconditioners 13.2. Polynomial preconditioning 13.3. Direct sparse approximations of the inverse 13.4. The multigrid idea References
107 108 111 113 114 115 117 118 118 124
1. Introduction These lecture notes consider techniques that are used for large systems of linear equations (1.1) (1.2) Ax = b Ax = x . and eigenvalue problems of large matrices We will consider the following issues: sparsity: the number of nonzero entries in the matrix; how algorithms get faster, when this is small linearity: these techniques make essential use of the linearity of the system complexity: the number of oating point operations needed to obtain the solution round-o errors: how they aect the solution a priori / a posteriori error analysis The methods can be divided in two classes which have the following characteristics: Direct methods matrix A is given manipulate the matrix work in Cn or Rn dimension n is visible exact solution all eigenvalues Krylov methods only the operator v Av is available work in the subspace span{b, Ab, . . . , Ak1 b} work in Cn , Rn , or Hilbert space dimension not visible approximate solution only interesting eigenvalues
In applications large matrices are typically sparse so that multiplication of vectors with them is computationally a cheap operation. The iterative methods for linear equations and eigenvalue problems exploit this property. In these methods the main computational units are matrixvector products, inner products of vectors, and formation of linear combinations of vectors. These are very dierent from the techniques of massaging the matrix that classical direct methods for these problems do (though iteratively for eigenvalue problems). In these notes we rst give a short review of direct techniques LU and QR decompositions for linear equations in chapters 2 and 4, respectively. Then in chapter 5 the basic eigendecompositions are reviewed. Chapter 6 is devoted to sensitivity issues of eigenvalues and the standard QRiteration for computing them. In
0Version:
April 26, 2013
EIROLA
chapter 7 we consider the theoretical properties of eigenvalues of Hermitian matrices and in chapter 8 some classical iterations to compute them. Chapter 9 starts the main topic of these notes the Krylov subspace iterations. In these methods approximations for (1.1) and (1.2) are searched in the increasing sequence of subspaces spanned by vectors (1.3) b, Ab, A2 b, A3 b, . . . . First we study the Krylov subspace iterations for eigenvalue problems. Then in chapter 10 we consider the classical iterations for linear systems. The powerful conjugate gradient iteration for Hermitian systems is studied in chapter 11 and nally starting in chapter 12 the modern Krylov subspace iterations for unsymmetric linear problems are derived and analyzed. Finally basic preconditioning techniques are introduced in chapter 13
1.1. Notation and basic properties. The entries of a matrix A Cmn are denoted by aij = (A)ij and if the vectors aj Cm form the columns of A we have a11 . . . a1n . . . A= . . . = a1 e1 + a2 e2 + + an en , am1 . . . amn
where the standard basis vectors ej are the columns of the identity matrix I . Matrix A denes a linear operator mapping v Cn to Av Cm . (1) Matrix A Cmn is a square matrix, if m = n . (2) Matrix A on upper(lower)triangular, if aij = 0 for i > j ( i < j ). (3) A is a diagonal matrix, if it is both upper and lower triangular, i.e., if aij = 0 , for i = j . (4) The product AB Cmn of matrices A Cmp and B Cpn is dened by (AB )ij = p k =1 aik bkj . Matrix product is associative: A(BC ) = (AB )C , but not commutative: generally AB = BA . (5) The determinant of a square matrix is denoted by det(A) . The determinant of a triangular matrix is the product of its diagonal elements and det(AB ) = det(A) det(B ) . (6) Identity matrix I satises AI = IA = A for all matrices A . (7) A is called nonsingular, if det(A) = 0 . Then it has an inverse A1 . This satises AA1 = A1 A = I . (8) The product of two lower triangular matrices is lower triangular. The inverse of a nonsingular lower triangular matrix is lower triangular. Similarly for upper triangular matrices. (9) The transpose of A Cmn is denoted by AT Cnm . It is dened by (AT )ij = (A)ji . Matrix A = AT is called the adjoint of A . (10) A Rnn is symmetric, if AT = A .
EIROLA
(11) A Rnn is skew symmetric, if AT = A . (12) A Cnn is Hermitian, if A = A . (13) A Rnn is orthogonal, if AT = A1 . (14) A Cnn is unitary, if A = A1 . (15) The matrix product satises (AB )T = B T AT and for invertible matrices (AB )1 = B 1 A1 . (16) Let A Cmn . The nullspace (or kernel) of A is N (A) = {x Cn | Ax = 0} , and the range is R(A) = {Ax Cm | x Cn } . The following contains the main properties of invertible matrices Theorem 1.1. Let A Cnn . Then the following are equivalent (1) A1 exists. (2) det(A) = 0 . (3) The system Ax = b has a unique solution for all b . (4) The columns of A are linearly independent. (5) The rows of A are linearly independent. (6) N (A) = {0} .
(7) R(A) = Cn .
The dimension of a space is the maximum number of linearly idependent vectors in it. The fundamental theorem of linear algebra (the dimension theorem) says that for every matrix A Cmn holds dim(R(A)) + dim(N (A)) = n . Let V and W be subspaces of a nite dimensional vector space U . By V + W we mean the set v + w v V , w W . This is also a subspace and we have dim(V + W ) = dim(V ) + dim(W ) dim(V W ) .
We write U = V W if every vector u U can be written in a unique way as u = v + w , where v V and w W . This is equivalent to V +W =U and V W = {0 } . In this case dim(U ) = dim(V ) + dim(W ) . is called the direct sum. The set of eigenvalues of A Cnn is denoted by (A) . It is called the spectrum of A and is dened by (A) = C N (I A) = {0} . When is an eigenvalue, the space N (I A) is called the eigenspace. The generalized eigenspace is given by Then the following holds: Cn = (A) E . E = x (I A)n x = 0 .
The p norm of a vector x Cn ( 1 p < ) is dened by x

p
n j =1
|xj |p
1 p
and x = maxj |xj | . The 2norm is called Euclidean and is denoted x 2 . For a matrix A C mn we set A A
p p,q
x =
= max Ax
x
p =1
= A
p,p
and A = A A
F
. Further, the Frobenius-norm is dened as

1
= (tr(A A)) 2 =
m i=1
n j =1
|ai,j |2
1 2
The standard inner product of vectors x, y Cn is denoted by x, y =

j
xj y j .
The following H older inequality is left as an exercise (consulting literature is allowed). Lemma 1.2. If
1 1 p
1 q
= 1 , then | x, y | x
The special case p = q = 2 of this is the Cauchy (SchwartzBunyakovsky) inequality: | x, y | x y .

1Also
the case p = 1 , q = is included.
EIROLA
For S Cn we denote by S the set of vectors orthogonal to all vectors of S : If V is a subspace of Cn , then V is called the complementary subspace and we have Cn = V V . The codimension is dened as the dimension of the complementary subspace, i.e., codim(V ) = dim(V ) = n dim(V ) . S = v Cn s, v = 0 s S .
The geometric form of the fundamental theorem of linear algebra is that any matrix A Cmn satises N (A ) = R (A ) . This is seen as follows: A x = 0 y , x = 0 y R(A ) x R(A ) . v , Ax = 0 v A v , x = 0 v
If (v 1 , v 2 , . . . ) is a sequence of linearly independent vectors then the GramSchmidt process constructs an orthonormal sequence (q 1 , q 2 , . . . ) in the following way: q1 = v1/ v1 (1.4) qk wk = v k ,
k 1 j =1
vk , qj q j , .
= wk / wk
k = 2, 3, . . .
This sequence satises a) (q 1 , q 2 , . . . ) is orthonormal. b) span(q 1 , . . . , q k ) = span(v 1 , . . . , v k ) for all k 1 . By: |x| (respectively |A| ) we mean the vector (matrix) formed from the absolute values: |x1 | |A1,1 | |A1,2 | . . . |A1,n | |x2 | |A2,1 | |A2,2 | . . . |A2,n | , . . |x| = | A | = . . . . . . . . . . . |xn | |Am,1 | |Am,2 | . . . |Am,n | Notation x y means xj yj for all j ( A B Ai,j Bi,j i, j ).
turns out to be handy.
Since most programming environments can use architecture optimized low level basic linear algebra routines (BLAS), it is advisable in algorithms to use operations of type matrix-matrix, matrix-vector, vector-vector whenever possible. In writing such algortihms the matlabnotation for parts of matrices: Ai,k . . . Ai,l . . . A(i : j, k : l) = . . . Aj,k . . . Aj,l
The rounding operation of oating point numbers is denoted by f l . We assume constantly that for given oating point numbers x, y the computer performs the operation z = x y , where {+, , , /} , such that rst it computes the exact value, and then the result is rounded to the closest oating point number, i.e., the result is z = f l(x y ) . We idealise the oating point arithmetic further by assuming that the exponential part of the oating point numbers is unbounded, i.e., under or overows do not happen. Thus we have the following set of oating point numbers in binary form with some xed s N : Then f l : R F and we have: the result of a oating point operation on the computer satises Here = 2s is the machine constant, typically 1016 . z = f l (x y ) = x y + e where |e| |x y | . F = 0.d1 , d2 . . . ds 2k di {0, 1} , k Z .
Problem 1.1. If A and B are nn oating point matrices, then which inequality of the form |E | ? you get for the matrix product: AB = AB + E ? The elementary permutation matrix is denoted by (j, k ) . Multiplication by this changes the j :th and the k :th component: y = (j, k ) x has yk = xj , yj = xk , yi = xi , i = j, k .
EIROLA
2. Linear system Ax = b , elimination, LU decomposition. A linear system Ly = b , where L is a nonsingular lower triangular matrix, is easy to solve with recursive substitution. This holds also for a system with an upper triangular matrix. A basic strategy for solving the linear system Ax = b starts with the attempt to write A as a product A = LU of a lower triangular matrix L and an upper triangular matrix U . Then our system is equivalent to two easy ones: Ly = b Ax = b LU x = b . Ux = y 2.1. LU decomposition. This is obtained using elimination as follows: Assume, that 1 0 0 a2,1 a 1 0 1,1 a3 ,1 0 1 a1,1 . . .
1 an, 1,1
i.e., M 1 A = A1 , where the part below the diagonal of the rst column zero. If (the pivot element) a1,1 = 0 , then we can continue: 1 0 0 . . . 0 a a1,2 . . . a1,n a1,1 a1,2 a1,3 1,1 1 0 . . . 0 0 0 a1,1 . . . a1,n1 0 a1,1 a1,2 2,1 0 a 1 . . . 0 0 a . . . a 0 0 2,1 2,n1 a1,1 = . . . . . . .. . . . . . . . . . . an1,1 0 a . . . a 0 0 n1,1 n1,n1 0 a1,1 0 . . . 1
a1,1 = 0 . Then . . . 0 a1,1 . . . 0 a2,1 . . . 0 a ,1 3 . .. . . . an,1 0 0 ... 1
a1,2 a1,3 a2,2 a2,3 a3,2 a3,3 . . . . . . an,2 an,3
. . . a1,n a1,1 a1,2 a1,3 . . . a1,n . . . a2,n 0 . . . a3,n = 0 . A . . . . . . . . an,n 0
of A1 is
. . . a1,n . . . a1,n1 , A
i.e., M 2 A1 = M 2 M 1 A = A2 . If the resulting square matrix with decreasing dimension never has zero pivot element (upper left corner) we get: M n 1 . . . M 2 M 1 A = U , where U is an upper triangular matrix. The matrices M k are lower triangular, so that their product and inverses are also such. In order to form the decomposition A = LU we need to compute the product
1 1 1 L = M 1 M 2 . . . M n 1 . 0Version:
April 26, 2013
Now each of the Gauss transforms M k is of the form . . . 0 k tk = k+1 , . . .

0
k n
M k = I tk eT k ,
0 . . . 1 . ek = 0 . . .
0
T
uv . Lemma 2.1. If v T u = 1 then I uv T is invertible and (I uv T )1 = I + 1 vT u
Proof. (I +
uv T )(I 1vT u
uv T ) = I uv T +
1 uv T 1v T u
1 uv T uv T 1v T u
=I .
T 1 Since eT = I + tk eT k tk = 0 , we now get (I tk ek ) k . Generally: j k = T ej tk = 0 , so that T T L = (I + t1 eT 1 )(I + t2 e2 ) . . . (I + tn1 en1 )
T T T = (I + t1 eT 1 + t2 e2 )(I + t3 e3 ) . . . (I + tn1 en1 )
=I+
2,1 a 1,1 a a3,1 a = 1.,1 . .
tk eT k
k
0 1
a2,1 a1,1
0 0 1 . . .
an2,1 a1,1
. . .
an,1 a1,1
an1,1 a1,1
... 0 . . . 0 . . . 0 . .. . ... 1
In other words, L is formed easily by collecting the nonzero parts of the tk vectors of the Gauss transforms. Problem 2.1. What is the complexity of the LU decomposition, i.e., how many 2 3 oating point operations are needed? (Answer: 3 n .) Problem 2.2. Show, that det(Ak1 (1 : k, 1 : k )) = det(A(1 : k, 1 : k )) , for k = 2, ..., n . In other words the pivot elements never become zero, if and only if all the upper left determinants of A are = 0 .
10
EIROLA
The following programmed algorithm starts from f l(A) , stores L (except the ones on the diagonal) below the diagonal and U on the diagonal and above it: for k =1 : n 1 do t=
1 A(k,k )
A(k + 1 : n, k )
A(k + 1 : n, k + 1 : n) = A(k + 1 : n, k + 1 : n) t A(k, k + 1 : n) A(k + 1 : n, k ) = t end Theorem 2.2. Let A = f l(A) . If the pivot elements are nonzero, then the computer produces L, U such that L U = A + H , where |H | 2(n 1) (|A| + |L| |U |) + O (2) . U = A , L = 1 , H = 0 = true.
(n1)(n1)matrices.
Proof. By induction: n = 1 :
Assume, that the theorem holds for all A=
Let
wT Rn n , v B
B R(n1)(n1) .
At the rst step of the algorithm:

1 v) = t = f l( 1
v+f
|f | |F |
|v | ,
A := A1 (2 : n, 2 : n) = B tw T + F
2(|B | + |t||w|T ) + O (2) .

1
After this the algorithm continues with the matrix A . By induction hypotheses 1 the result is L1 U 1 = A + H 1 , where |H 1 | so that LU = 1 0 t L1 wT 0 0 = A+ =A+H . f H 1 + F 0 U1 2(n 2)(|A | + |L1 ||U 1 |) + O (2) ,
1
11
We get |A | |H 1 + F | | f | |H |
1
(1 + 2)(|B | + |t||w|T ) + O (2)
2(n 2)(|B | + |t||w|T + |L1 ||U 1 |) + 2(|B | + |t||w|T ) + O (2) 2(n 1)(|B | + |t||w|T + |L1 ||U 1 |) + O (2) |v | 2(n 1) 1 0 | | |w |T + |v | |B | |t| |L1 |
= 2(n 1)(|A| + |L||U |) + O (2) , i.e., the theorem holds for

nnmatrices.
| | |w |T 0 |U 1 |
+ O ( 2 )
2.2. Conditioning. Some problems are dicult to solve. The concept conditioning describes us how accurately a value can in principle be computed. Function f : Rn Rm is said to be well conditioned if for a small change d at x does not cause a big change f (x + d) f (x) . For this we dene the absolute condition number = lim sup
0 d
f (x + d) f (x) . d
fi (x) xj
If f is dierentiable and J = D f (x) is the Jacobian matrix J i,j = = J . The (more usual) relative condition number is dened as = lim sup
0 d
, then
f (x + d) f (x) / f (x) . d / x x / f (x) .
Again, if f is dierentiable, then = J
When is of reasonable size, then f is well conditioned at x , and if is huge, f is ill conditioned. Example 2.1. f (x) = f is well conditioned. x , x > 0 . Now f (x) =
1 2 x
and =
1 2 x
1 2
. Hence
Hence f can be ill conditioned, if x1 x2 . In practice we see this so that if x1 and x2 are expressed in d digits and the rst s digits of them are the same, then we get only d s correct digits, when computing x1 x2 .
Example 2.2. f (x) = x1 x2 , J = [1 1] , J = 2 2 x2 1 + x2 = . |x1 x2 |
2 so that
12
EIROLA
Conditioning of a linear system Ax = b . Here the solution x depends on A and b . Consider rst x = f (b) = A1 b . For this we get A1 b A1 Ax A A1 . f = = 1 x A b And considering the map F : A x = A1 b we rst get = A1 A1 + O ( 2 ) . (A + )1 A1 = (A + )1 (I (A + )A1 ) = (A + )1 A1
Hence D F (A)() = A1 A1 b and F = A1 x
D F (A) = A1 A = A A 1
A1 b .
so that
A 1 b
Thus for these maps we get the relative condition number A = A A1 (an upper bound for f ). This is called the condition number of matrix A , and the system is well conditioned if this is not large. If the computations are done with a good (stable) algorithm with d decimal digits, then we typically loose log10 digits in the accuracy of the solution. This is due to the conditioning of the problem and cannot be cured. But we can do worse with a bad algorithm. Since A A1 A A1 = I = 1
1 i+ j 1
we nd that the condition number of a matrix is always at least one. Example 2.3. The Hilbert matrix consisting of elements aij = be very ill-conditioned. Consider the following in matlab x = 1.0000 1.0000 0.9999 1.0019 0.9863 1.0602 0.8324 1.3033 0.6445 1.2603 0.8918 1.0195 is known to
>> >> >> >> >>
n=12; A=hilb(n); x0=ones(n,1); b=A*x0; cond(A)
ans = 1.6976e+16 >> x=A\b
13
Here the correct solution is the vector x0 consisting of ones. The machine constant in these computations is =2.2204e-16 so that is already larger than one. For a system that has reasonable condition number we want to nd a method that gives us accurate solutions. This should be possible, since small changes in the matrix or the right hand side do not change the solution very much. And this is true. 2.3. Partial pivoting. From theorem 2.2 we see that the elimination produces L and U such that |A LU | C (|A| + |L| |U |) with a reasonable size C . The right hand side can however be big, if L and/or U contain large elements and then the error is not necessarily small. Example 2.4. Consider A = U= 1 0 1 1/ 1 1 0 for a small . Then both L = and 1 1 1/ 1
contain some large elements.
Now both L and U are reasonable and the system is easily solvable also with nite precision. The idea of the previous example is called partial pivoting:
Instead of equation Ax = b consider an equivalent equation Ax = b , where 1 1 b A= and b = 2 , i.e., just write the two equations in reverse order. For 1 b1 this A we get 1 1 1 0 1 1 = . 1 1 0 1
Find the element with largest absolute value of the rst column of A . Let it be ak,1 . Using the permutation matrix 1 = (1, k ) we change it to the place 1,1 and eliminate: ak,1 ak,2 . . . ak,n a2,1 a2,2 . . . a2,n a k,1 ak,2 . . . ak,n . . . . . . 0 . . . . M 1 1 A = M 1 = . A a1,1 a1,2 . . . a1,n . . . . . . . . 0 . . . an,1 an,2 . . . an,n
14
EIROLA
Next we nd the element with largest absolute value in the rst column of A , etc. We get M n1 n1 . . . M 2 2 M 1 1 A = U ,
T where M k = I tk eT k and |tk | so that
T [0 . . . 0 1 . . . 1] . Now j > k = eT k j = ek ,
T T j M k = j (I tk eT k ) = j j tk ek j = (I j tk ek ) j .
Denote: tk = n1 . . . k+1tk , M k = I tk eT k . Then n1 . . . k+1 M k = M k n1 . . . k+1 and U = M n 1 n 1 M n 2 n 2 . . . M 1 1 A = = M n 1 M n 2 n 1 n 2 M n 3 . . . M 1 1 A = = M n1 M n2 M n3 . . . M 2 n1 n2 . . . 2 M 1 1 A = = M n1 . . . M 2 M 1 n1 . . . 2 1 A We got: A = L U , where = n1 . . . 2 1 and as before L = M1
1
M2
. . . M n 1
T = I + t1 eT 1 + + tn1 en1 .
Now all the elements of L have absolute values at most one. Problem 2.3. Program the algorithm solving the system A x = b using partial pivoting. Problem 2.4. Show: if det(A) = 0 , then, using partial pivoting, in exact arithmetic all pivots are nonzero. Problem 2.5. Prove a version of Theorem 2.2 now for the LU decomposition with partial pivoting. Problem 2.6. A is strictly diagonal dominant, if |ai,i | > j =i |ai,j | for all i = 1, . . . , n . Show: if AT is strictly diagonal dominant, then above = I , i.e., pivoting is not needed. Problem 2.7. Using partial pivoting we get L n , but the worst case may produce U 2n A . Consider the following example: , when i = j or j = n, 1 . ai,j = 1 , when i > j, 0 , else
15
The example of problem 2.7 tells us that for large systems (e.g. 2100 1030 ) there may be cases where partial pivoting does not work. However, these seem to be very rare (never encountered in real problems) and partial pivoting is the most popular elimination method used. 2.4. Full pivoting. A reasonable growth rate can be guaranteed only for full pivoting. There one nds in all of A the element aj,k with largest absolute value. Then this is brought to place 1,1 by multiplication with permutations from the left and the right: 1 A Q1 = (1, j ) A (1, k ) . Then eliminate: M 1 1 A Q1 = A1 and continuing this way we get: M n1 n1 . . . M 2 2 M 1 1 A Q1 Q2 . . . Qn1 = U , from which, as before A Q = L U , where and Q are permutation matrices. One can show the following Theorem 2.3 (Wilkinson). With exact arithmetic the full pivoting algorithm produces: 1 1 1 1 U n(2 3 2 4 3 . . . n n1 ) 2 A . The coecient on the right hand side grows roughly like n3 . Thus for n = 10000 it is of the order 1012 . For larger matrices we may need other types of methods. Problem 2.8. What is the complexity of full pivoting if a comparison is as expensive as a oating point operation?
2.5. The LDL decomposition and the Choleskydecomposition. Let A be a regular Hermitian matrix having a decomposition A = LU . Write U = D L , where D is a diagonal matrix and L a lower triangular matrix, with ones on the diagonal. Then
A = LD L = LDL
, i.e.,
. D (L1 L) = L1 LD
The left hand side of the latter equation is an upper triangular matrix and on the is a diagonal right there is a lower triangular matrix. This is possible only, if L1 LD matrix, and then L1 L also has to be diagonal. Both L1 and L have ones on the diagonal (for L1 consider forming the inverse by determinants), so that L1 L = I , i.e., L=L A = LDL . and =D . D
We obtained the LDL decomposition of a Hermitian matrix:
16
EIROLA
Further, if A positive denite the diagonal of D is positive, so that it can be writ 1 ten: D = (D 2 )2 = diag( d1 , . . . , dn )2 . Then we get the Choleskydecomposition of A : 1 1 A = LD 2 D 2 L = GG , 1 where G = LD 2 is a lower triangular matrix with positive diagonal. This decomposition can be computed directly (with n3 /3 f loperations) as follows: for k = 1 : n do A(k, k ) = A(k, k )
A(k + 1 : n, k ) = A(k + 1 : n, k )/A(k, k ) for j = k + 1 : n do end end Here we use only the lower triangular part of A . The upper part is not needed and the result is stored in the place of the lower triangular part. With Hermitian matrices we want to preserve Hermitianity in pivoting. This is done using symmetric pivoting: AT . With such we get the diagonal element with largest absolute value to the place 1,1 . A(j : n, j ) = A(j : n, j ) A(j : n, k )A(j, k )
17
3. Projections A nonzero linear map P is a projection if P 2 = P . Then (assuming P = I ) I P is also a projection: (I P )2 = (I P ) (I P )P = I P P + P 2 = I P . If V Cnk has linearly independent columns, then there exists W Cnk such that W V = I . For example1, W = (V ) = V (V V )1 will do. Lemma 3.1. If W V = I then P = V W is a projection. Proof. P 2 = V W V W = V W = P . A projection P is called orthogonal, if (I P )x R(P ) for all x . Theorem 3.2. For a projection P the following are equivalent (i) P is orthogonal (ii) P = P (iii) x 2 = (I P )x (iv) P 2 = 1 (v) R(P ) N (P ) .
+ Px
Proof. (i) (ii) : Assume P is orthogonal. Then, for all x, y , Hence P (I P ) = 0 , i.e., P = P P and P = (P P ) = P P = P . If P = P , then, for all x, y , i.e., x P x R(P ) . x
2
0 = x P x, P y = P (I P )x, y .
x P x, P y = P (I P ) x, y = (P P 2 ) x, y = 0 ,
(i) = (iii) : If P is orthogonal, then
= (I P )x + P x , (I P )x + P x = (I P )x
2
= (I P )x, (I P )x + (I P )x, P x + P x, (I P )x + P x, P x + Px
2
(iii) = (iv) is clear.

given A Cmn the matrix A = lim0 (A A + I )1 A is called the pseudoinverse (see section 4.4) of A . If A has full column rank, then A = (A A)1 A . In particular, if A is invertible, then A = A1 .
1For
18
EIROLA
(iv) = (v) : Assume P 2 = 1 . If x R(P ) and y N (P ) are non-orthogonal unit vectors, let u = x x, y y . Then P u = P x = x = 1 , but u
2
= x x, y y , x x, y y P
2
= x, x x, y y , x x, y x, y + | x, y |2 y , y = 1 | x, y |2 < 1 , 1/ u > 1 , a contradiction. Hence R(P ) N (P ) .
i.e.,
(v) = (i) : Clear from R(I P ) N (P ) .
Example 3.1. If V Cnk has orthonormal columns, then V V = I and P = V V is Hermitian projection, in particular, orthogonal. 3.1. Oblique projections. Problem 3.1. Show that for any projection N (P ) = R (I P ) and Cn = N ( P ) R ( P ) .
For a subspace E denote by P E the orthogonal projection onto E . Theorem 3.3. Let E and F be subspaces such that (3.1) Then dim E = dim F , and for any basis v 1 , . . . , v d of E there exists a basis w 1 , . . . , wd of F such that (3.2) v i , wj = i,j , i, j = 1, . . . , d .
P E PF 2
<1.
Such basis are called biorthogonal. Proof. First note that E F = {0} since if 0 = y E F then P F y = 0 and we get the following contradiction:
y = P Ey = P Ey P F y P E P F 2
y < y
E F = {0} further implies
Similarly we obtain dim F dim E .
dim E n dim F = n (n dim F ) = dim F .
Let v 1 , . . . , v d be a basis of E . Set Ej = span({v 1 , . . . , v d }\{v j }) . Since dim Ej = n d + 1 and dim F = d we have F Ej = {0} . So, for each j we can take 0 = y j F Ej . If v j , y j = 0 then y j E which is impossible. Hence, for every j we can set wj = y j / v j , y j . Clearly these satisfy (3.2).
19
d If j =1 j w j = 0 , then taking inner product with v i implies i = 0 . Thus w j s are linearly independent and there are d = dim F of them, hence they form a basis for F .
Problem 3.2. Let V , W Cnd be such that W V Rdd is invertible. Show that E = R(V ) and F = R(W ) satisfy the assumption of the previous theorem. Theorem 3.4. (3.1) is equivalent to E F = F E = {0} . Proof. It was already shown in the proof of the previous theorem that (3.1) implies E F = F E = {0 } . Assume now that (3.1) does not hold. We will show that there exists a nonzero vector either in E F or in F E . Let x be a unit vector such that u v 2 1 , where u = P E x, v = PF x. Since u, u x = 0 and x = u + (x u) we get Hence Then 2u x
2
u 1 x = 2
= u + u x, u + u x = u
1 2
. Similarly,
v1 x = 2
1 2
1 . Set a = u 2 x, b =
+ ux
= x
=1.
1 2
xv.
1= a + b a+b = uv 1 . Hence a + b = a + b and consequently a = b with > 0 . Since a and 1 1 b have same length we get = 1 , i.e. u 2 x= 2 x v . Thus u = x v and v = x u . But u E , x v F , v F , x u E . Since both u and v cannot be zero, either E F or F E is dierent from {0} .
Problem 3.3. Let E and H be subspaces of Cn such that Cn = E H . Construct a projection P such that R(P ) = E and R(I P ) = H . Hint: set F = H , take a basis v 1 , . . . , v d of E , and use theorems 3.4 and 3.3. Problem 3.4. Is P unique in the previous problem?
20
EIROLA
4. Least squares problems, QRdecomposition When the mnsystem A x = b has more equations than unknowns ( m > n ), it usually does not have a solution. A least squares solution means a vector x , for which A x b 2 is smallest, i.e., we look for the orthogonal projection of b onto R (A ) .
b bAx Ax R(A)
In other words we look for x such that A x b R(A) . Since R(A) = N (A ) , we see that x has to solve A (A x b) = 0 , i.e., (4.1) A A x = A b . The numerical solution of (4.1) is usually found by forming the QRdecomposition of A , i.e., writing A = QR , where the matrix Q Cmn has orthonormal columns and R Cnn is an upper triangular matrix. Then Q Q = I and if R is invertible (the columns of A are linearly independent), then (4.1) becomes R R x = R Q b , i.e., R x = Q b , which is easy to solve with backward substitutions. Problem 4.1. Show that R is invertible if and only if the columns of A are linearly independent. QRdecomposition can be computed by performing the GramSchmidt process on the columns of A . A better (more stable) way is to apply certain simple unitary transforms to A . The following result gives the justication for this: Problem 4.2. If U Cnn is unitary: U U = I , then Ux
2
= x
UA
= A
UA
= A
When computing U A , where U Cnn is unitary, one has in the computer A = A + E and U = U + F , where usually E 2 Cn A 2 . F 2 cn , and cn Cn . In the computer the result then is: U A = U A + H = (U + F )(A + E )+ H = U A + F A + U E + F E + H = U A + E ,
21
where the multiplication error satises Then E

2
cn A
+ O (2) and cn Cn .
2
+ E
2 2
+ H
+ O ( 2 ) ( C n + cn + cn ) A
+ O ( 2 ) .
Since the coecient of E not amplied.
is one, the errors accumulate only additively, they are
4.1. Householder transformation. Let v Cn \ {0} be arbitrary. Then the matrix v v H =I 2 v v is Hermitian and unitary: H = H , v v v v v v 2 =I . H H =H =I 4 +4 v v (v v )2 It is called a Householder transformation. H is a mapping that mirrors vectors in the plane perpendicular to v . In the gure v x Pv x = v v v is the orthogonal projection of x to the line spanned by v . And H x = x 2Pv x.
Pvx v
v Hx
Often we need a transformation H such that a given vector x is turned, by the multiplication with H in the direction of e1 . Since Hx = x 2 x, v v, v, v
we have Hx span(e1 ) = v span(x, e1 ) (unless x span(e1 ) ). The trial v = x + e1 gives x, v = x, x + x, e1 so that and v , v = x, x + x, e1 + x, e1 + ||2 , Hx = (1 2 x, x + x, e1 x, x + x, e1 + x, e1 + ||
2
)x 2
x, v e1 . v, v =
, e1 Choosing = | x x 2 ( = x 2 , if x, e1 = 0 ) we get: x, e1 x,e1 | 1 | x, e1 | x 2 , giving x, v = 2 v , v and H x = e1 .
22
EIROLA
In practice one never forms the matrix H , only the vector v is stored and for example the multiplication HA is done as: HA = A i.e., a rankone matrix is added to A . QRdecomposition with Householder transformations. Assume A Cmn and let a1 be the rst column of A . Find a vector v 1 such that H 1 a1 = r1,1 e1 , v1 v where H 1 = I 2 1 . Then v1 v1 r1,1 r1,2 . . . r1,n 0 . H1 A = . . A 1 . 0
2 v ,v
v (v A) ,
v2 v Let a1 be the rst column of A1 and let H 2 = I 2 2 be such that H 2 a1 = v2v2 1 0 v v 2 2 = . Then r2,2 e1 Cn1 . Set v 2 = [0 v 2 ] and H 2 = I 2 0 H2 v2v2 r1,1 r1,2 r1,3 . . . r1,n 0 r2,2 r2,3 . . . r2,n . 0 0 H 2H 1A = . . . . . . A
2
Continuing this way we get:
When solving a least squares problem minx A x b one does not need to form Q or store the vectors v k if at each step also the right hand side is multiplied by H k resulting in b = H n . . . H 2 H 1 b since by unitariness of the H k s Axb
2
r1,1 r1,2 0 r2,2 . . . 0 H n . . . H 2H 1A = 0 0 0 . . . . . . 0 0
. . . r1,n . . . r2,n . .. . . . R . . . rn,n = 0 ... 0 . . . ... 0
= H n . . . H 2 H 1 (A x b )
R x b1 b2
,
2
23
so that the best x is given by the solution of the system R x = b1 , and min A x b If we want to form the product Qn = H 1 H 2 . . . H n = I 2 v1 v 1 v v 1 1 I 2 v2 v vn v 2 n . . . I 2 v v v v 2 n 2 n
2
= b2
it is better to make it in the form Qk = I + W k V k , k = 1, . . . , n as follows: set: W 1 = w 1 = v2v1 v 1 , V 1 = v 1 . Then Q1 = H 1 = I + W 1 V 1 and

1
Qk = Qk1 H k = (I + W k1 V k 1 )(I 2
vk v k )= v v k k = I + W k 1 V k 1 + w k v k = I + W k V k ,
where wk = v2vk Qk1 v k and W k = [W k1 w k ] , V k = [V k1 v k ] .

k
Notice that also here v k is multiplied by the unitary matrix Qk1 and then scaled so that the computation is stable. Also this representation of Qn is very economic when n m . Problem 4.3. Show how to use Householder transformations to form the decomposition A = QL , where Q has orthonormal columns and L is a lower triangular matrix? Problem 4.4. Assume we need to solve the nn system A x = b . Compute the work load ( n3 terms) for decompositions A = QR and A = L U (with full pivoting) assuming in the latter, that work(comparison) = k work(oating point operation). For which value of k the work loads are the same? 4.2. Givens rotations. Let U= , a C2 \ {0} and b = a b , = , m= m m |a|2 + |b|2 .
where
Then U is unitary, det(U ) = 1 and U the vector a b in the direction of vector
a m = . In other words, U rotates b 0 1 . 0
24
EIROLA
From these we can build unitary transformations that zeroes suitable elements of a xj xk vector/matrix: if = , = , m = |xj |2 + |xk |2 , then setting: m m
j 1 ... 1 1 k j
Gj,k =
..
1 . .. 1
we get a unitary transformation satisfying (Gj,k x)j = m , (Gj,k x)k = 0 , (Gj,k x)l = xl , l = j, k .
Naturally, in practice multiplication of a vector v with Gj,k is done as w = v followed by wj v =U j . wk vk Similarly, B = Gj,k A is obtained by copying the other rows to B and computing the j :th and k :th elements of each column as above.
25
QRdecomposition # 0 # A1,2 = G1,2 A = # . . .
where An,m =
R . Again, if we are solving a least squares problem A x = b we 0 need not form Q or store the rotations if at each step we multiply also the right hand side by Gj,k .
using Givens rotations can be done as follows: # # ... # # # # ... 0 # # ... # # . . . # 0 # # ... # # . . . # A1,3 = 1,2 # # . . . # # # # . . . G A = 1,3 . . . . . . . . . . . . . . . . . . # # # ... # # # # ... # # # ... # # # ... # 0 # # ... 0 # # . . . # 2,3 0 0 # ... 0 # # . . . # A = A1,m = 1,m 1,m1 0 # # . . . # 0 # # ... G2,3 A = G1,m A = . . . . . . . . . . . . . . . . . . . . . 0 # # ... 0 # # ... # # # # ... # # # ... # 0 # # . . . # 0 # # ... 0 0 # ... n,m 2,m 0 0 # . . . # A = A = . n,m1 2,m1 0 0 # . . . # Gn,m A = 0 0 0 .. G2,m A = . . . . . . . . . . . . . . . . . . . . . 0 0 # ... # 0 0 0 ...
# # # # . . .
# # # # # . . .
# # # # # . . . 0
4.3. Uniqueness of Q R . If A Cmn , rank(A) = n , then the Q R decomposition of A , where the diagonal elements of R are positive is unique: If A = Q R = Q R are two such decompositions, then U = Q Q = R R1 is upper triangular and we get U U R = Q Q Q Q R = Q Q Q Q R = Q Q R = Q Q R = R , so that U U = I , i.e., U is a unitary upper triangular matrix. This is possible only if it is a diagonal matrix, and |uj,j | = 1 j . Since R and R1 have positive diagonal elements so does U and we get uj,j = 1 j . Hence U = I , R = R , and Q = Q. The use of Householder transformations does not produce positive diagonal for R r1,1 n,n we get unitary D and from A = Q R we . . . , |r but setting D = diag |r1 rn,n | ,1 | R has positive diagonal. get A = Q R , where Q = Q D and the matrix R = D

26
EIROLA
4.4. Pseudoinverse. The (MoorePenrose) pseudoinverse of a matrix A Cmn is dened as (A A + I )1 A . A = lim +

0
Problem 4.5. Show, that the limit above always exists. Hint: it suces to show that lim0+ (A A + I )1 A v exists for every v Cn . Consider rst the cases v N (A ) and v R(A) . You may want to show rst that for every matrix holds R(AA ) = R(A) . Problem 4.6. Assume A has rank n ,. Show that then the pseudoinverse is A = (A A)1 A . This has the properties A A = I and AA x = x for all In this full rank case the pseudoinverse is obtained easily from the QR-decomposition of A : A = (R Q QR)1 R Q = R1 Q . Problem 4.7. What A in the case rank(A) = n m ? How can you use the QR-decomposition in this case? Remark 4.1. In the general case the pseudoinverse can be computed from the singular value decomposition of A (see section 7.3). Remark 4.2. Another way to dene A is to require a) A A and AA are both Hermitian, b) AA A = A , c) A AA = A . 4.5. Other unitary decompositions. Problem 4.8. Let A Cnn . Show how, using Householder transformations, one nds unitary U and V such that B = U AV is a bidiagonal matrix: d1 f1 d2 f2 . . . .. .. B = U AV = d f
n 1
x R (A ) .
dn
n 1
This will be the starting point for computing the singular value decomposition of matrix A .
27
Problem 4.9. Using Householder transformations show, that arbitrary square matrix A Cnn is unitarily similar to a Hessenbergmatrix M (that is mj,k = 0 for j k + 2 ), i.e., # # # ... # # # 0 U AU = M = 0
# # # ... # # # # # ... # # # 0 # ... # # #
..
0 0 0 ... # # # 0 0 0 ... 0 # #
This will be practical in the eigenvalue computations.
28
EIROLA
5. Eigendecompositions The eigenvalues of a square matrix A Cnn are the complex numbers for which there exists a nonzero vector x such that A x = x . Clearly, such a satises det(I A) = 0 and also the latter is sucient for the existence of a nonzero solution for x A x = 0 . Hence the eigenvalues are the roots 1 , . . . , n of the characteristic polynomial: det(z I A) = z n tr(A) z n1 + + (1)n det(A) = (z 1 ) . . . (z n ) , from which one nds 1 + 2 + + n = tr(A) = a1,1 + a2,2 + + an,n and 1 2 . . . n = det(A) . Assume A and B are similar, denoted by A B , in other words there exists a regular S such that B = SAS 1 . Then det(z I B ) = det(S (z I A)S 1 ) = det(S ) det(z I A) det(S )1 = det(z I A) , so that A and B have the same characteristic polynomials and tr and det are similarity invariants: tr(B ) = tr(A) , The corresponding eigenvectors satisfy: Problem 5.1. Denote by Aj,k the matrix that is left when the j :th row and the j,j 1 )= k :th column are deleted from A . Set: F (A) = n j =1 det(A ) . Show: F (SAS F (A) . If B = SAS 1 , then B k = SAS 1 SAS 1 . . . SAS 1 = SAk S 1 and for any polynomial p(z ) = n z n + n1 z n1 + + 1 z + 0 holds p ( B ) = n B n + n 1 B n 1 + + 1 B + 0 I = S p ( A ) S 1 .
B k =0 1 k!
det(B ) = det(A) .
Ax = x SAS 1 Sx = Sx BSx = Sx .
The same holds for converging power series of matrices. For example: e = B k = S eA S 1 .
The set of eigenvalues (A) = C det(I A) = 0 A.

0Version:
is called the spectrum of
April 26, 2013
29
Problem 5.2. Let A be a block upper triangular matrix: A = C pp , A2,2 C (np)(np) . Show: (A) = (A1,1 ) (A2,2 ) .
A1,1 A1,2 , A1,1 0 A2,2
5.1. Schur decomposition. Every matrix is unitarily similar to a triangular matrix. This is the content of the Schur decomposition theorem, which is an important analytic and numerical tool. Let us show rst a general lemma. Lemma 5.1. Assume the matrix X Cnp has linearly independent columns and AX = XB , A Cnn , B Cpp . Then R(X ) is A invariant ( x R(X ) = Ax R(X ) ) and (B ) (A) . Further, there exists an invertible S and a unitary Q such that S 1 AS = B A1,2 0 A2,2 and Q AQ = T 1,1 T 1,2 0 T 2,2 ,
where T 1,1 B . In particular, (T 1,1 ) = (B ) . Proof. Let Y Cn(np) be such that S = X Y AX AY = XB AY , so that S 1 AS = I B S 1 AY 0 = is invertible. Then AS =
B A1,2 0 A2,2
R , where 0 R is an upper triangular matrix. Then Q = H 1 H 2 . . . H p = Q1 Q2 is unitary and Q1 = XR1 , Q 2 X = Q2 Q1 R = 0 . Hence the matrix T = Q AQ satises: Take Householder matrices H 1 , . . . , H p such that H p . . . H 2 H 1 X =
1 1 T 1,1 = Q = Q = RBR1 1 AQ1 = Q1 AXR 1 XBR 1 T 2,1 = Q =0. 2 AQ1 = Q2 XBR
Theorem 5.2 (Schur decomposition). For every A Cnn there exists a unitary Q such that T = Q AQ is upper triangular. Here the diagonal elements of T (the eigenvalues of A ) can be in any order. Proof. By induction: the case n = 1 is clear. Let be an eigenvalue of the matrix A Cnn and x = 0 a correspoding eigenvector. By the previous lemma with X = x and B = there exists a unitary U such that U AU = wT 0 C .
30
EIROLA
By the induction hypotheses there exists a unitary U such that U C U is an upper 1 . Then triangular matrix. Set: Q = U U T = Q AQ =
1 U
U AU
1 U
wT U 0 U CU
is an upper triangular matrix. Problem 5.3. Let pA (z ) = det(z I A) = z n + cn1 z n1 + + c1 z + c0 be the characteristic polynomial of matrix A . Prove the CayleyHamilton theorem:
w Hint: For a triangular A = [ 0 B ] use pA (A) = (A I ) pB (A) and induction.
pA (A) = An + cn1 An1 + + c1 A + c0 I = 0 .
Problem 5.4 (Real Schur decomposition). Show, that for every A Rnn there exists an orthogonal matrix V Rnn (orthogonal: V T V = I ) such that T = V T AV is a block upper triangular matrix, where the diagonal blocks of T are either scalars corresponding to the real eigenvalues of A , or 2 2matrices, which have a pair of complex eigenvalues. Hint: if = + i is a complex eigenvalue of A then there exist vectors u, v Rn such that A u v = u v . a b Orthonormalise (Gram-Schmidt): u v = q 1 q 2 = Q1 , form an orthog0 c onal Q = Q1 Q2 and multiply: QT AQ . Denition 5.1. Matrix A is normal, if A A = AA , i.e., if [A, A ] = 0 , where the commutator is dened as [A, B ] = AB BA . Theorem 5.3. A is normal, if and only if it is unitarily similar to a diagonal matrix. Proof. If D is a diagonal matrix and Q unitary such that A = QDQ , then QDQ = QDDQ = QDQ QDQ = AA . A A = QDQ On the other hand, if A is normal, then let Q be unitary such that T = Q AQ is upper triangular. Then T T = Q A QQ AQ = Q AA Q = Q AQQ A Q = T T , so that T is normal. We claim that, that a normal upper triangular matrix is diagonal. By induction: the case n = 1 is clear. Let T Cnn be a normal upper triangular matrix. Then (T T )1,1 = |t1,1 |2 = (T T )1,1 = |t1,1 |2 + |t1,2 |2 + + |t1,n |2 ,
31
so that t1,2 = = t1,n = 0 . Hence T =
t1,1 0 , where T is a (n1)(n1) 0 T normal upper triangular matrix. By the induction hypotheses T and hence also T is a diagonal matrix. Problem 5.5. Show that A Cnn is normal if and only if A
2 F
n j =1
|j |2 .
Theorem 5.4. For given A Cnn there exist vectors uj , v j Cn , j = 1, . . . , n , such that (5.1) z C . Further, the numbers j = uj , v j are the eigenvalues of A . Proof. Take the Schur decomposition A = QT Q and write the upper triangular matrix as t1 t 2 = e1 t T = 1 + e2 t2 + + en tn . . . . t n
I z T = I z (e1 t 1 + e2 t2 + + en tn ) I z A = (I z un v n ) . . . (I z u2 v 2 )(I z u1 v 1 ) ,
Since the rst j 1 components of tj are zero, we have t j ej k = 0 , j 2, k 1 . Hence

= (I z en t n )(I z en1 tn1 ) . . . (I z e1 t1 )
and I z A = Q(I z T )Q
= Q (I z e n t n )Q Q(I z en1 tn1 )Q . . . Q(I z e1 t1 )Q
= (I z Qen t n Q ) . . . (I z Qe1 t1 Q ) .
Setting uj = Qej , v j = Qtj we get (5.1). Since the numbers

v j uj = T j Q Qej = tj ej
are the diagonal elements of T , they are the eigenvalues of A . Corollary 5.5. From (5.1) we can further write (I z A)1 = I + whenever z v j uj = 1 for all j . Proof. Use lemma 2.1.
z 1z v 1 u1
u1 v 1 ... I +
z 1z v n un
un v n
32
EIROLA
5.2. The Sylvester equation. In order to make similarity transformations from the Schur decomposition for obtaining even simpler similar matrices we rst study the Sylvester equation, which is also very useful for example in the control theory. Consider the problem: for given A Cnn , B Cmm , and C Cnm nd an X Cnm such that (5.2) AX XB = C .
The following theorem gives the solvability condition for (5.2). Theorem 5.6. The linear transformation : Cnm Cnm (X ) = AX XB is invertible if and only if (A) (B ) = . Proof. It is sucient to show that the kernel of is trivial. Let Q1 and Q2 be unitary such that L = Q 1 A Q1 and R = Q2 BQ2 are upper triangular matrices. Then AX XB = 0 Q1 AQ1 Q1 XQ2 Q1 XQ2 Q2 BQ2 = 0 LY Y R = 0 , where Y = Q 1 XQ2 . Now (L) (R) = , so that lj,j = rk,k j, k and equation (X ) = 0 implies l1,1 0 l2,1 l2,2 . . . . . . ln,1 ln,2 0 y1,1 0 y2,1 . . . . . . . . . ln,n yn,1 y1,1 y2,1 . . . ... ... .. . y1,2 . . . y1,m y2,2 . . . y2,m . . .. . . . . . yn,2 . . . yn,m r1,1 r1,2 . . . r1,m y1,2 . . . y1,m y2,2 . . . y2,m 0 r2,2 . . . r2,m =0, . . . . .. .. . . . . . . . . . . 0 0 . . . rm,m yn,2 . . . yn,m
yn,1
33
which recursively gives (l1,1 r1,1 )y1,1 = 01,1 = y1,1 = 0 (l2,2 r1,1 )y2,1 = 02,1 = y2,1 = 0 . . . ... . . . ... ... = (l1,1 r2,2 )y1,2 = 01,2 = y1,2 = 0
= (l1,1 rm,m )y1,m = 01,m = y1,m = 0 = (l2,2 r2,2 )y2,2 = 02,2
= (l2,2 rm,m )y2,m = 02,m = y2,m = 0 = (ln,n r2,2 )yn,2 = 0n,2 = yn,2 = 0
= y2,2 = 0
(ln,n r1,1 )yn,1 = 0n,1 = yn,1 = 0
= (ln,n rm,m )yn,m = 0n,m = yn,m = 0
i.e., Y = 0 and X = Q1 Y Q 2 = 0 . Hence the kernel of is {0} , so that is invertible. For necessity, see problem 5.7. Remark 5.1. The proof above shows how one can (and usually should) solve the Sylvester equation: obtain the Schur decompositions of the coecient matrices and on the right hand side set C = Q 1 CQ2 . We get the matrix Y by a recursion similar to that above, where the zeros on the right hand side are replaced by elements of C and the already computed entries of Y . Then we get X by multiplication. This is much cheaper than solving an nm nm system of linear equations. We leave the details to an exercise. From the proof above we see that for getting elements of Y we need to multiply by the entries of L and R and divide by numbers li,i rj,j , i.e., the dierences of the eigenvalues of A and B . Hence the conditioning for solving the Sylvester equation is proportional to ( A + B ) max | |1 .
(A) , (B )
Problem 5.6. A more elegant proof (but no algorithm) for the previous theorem can be obtained as follows. Show: AX = XB = Ak X = XB k and p(A) X = X p(B ) for every polynomial p . Take p to be the characteristic polynomial of A . Show, that pA (B ) is invertible. Problem 5.7. Show, that the eigenvalues of are: i j , i (A) , j (B ) .
Problem 5.8. Assume AX = XM and A Y = Y N , where (M ) (N ) = . Show that Y X = 0 . In particular, if Ax = x , A y = y , and = , then y x.
34
EIROLA
By the Schur decomposition we know that every matrix is unitarily similar to a triangular matrix. Now we want to make further similarity tranformations to get every matrix in a more simple form. We will need the following. T 1,1 T 1,2 Lemma 5.7. Let T = 0 T 2,2 p q p q there exists Z C such that I Z 0 I
1 1
p q be such that (T 1,1 ) (T 2,2 ) = . Then I Z T 1,1 0 = 0 I 0 T 2,2
T 1,1 T 1,2 0 T 2,2
Proof. Since is
I Z 0 I
I Z , the matrix product on the left hand side above 0 I
T 1,1 T 1,1 Z ZT 2,2 + T 1,2 , 0 T 2,2 so that a suitable Z is found as the solution of equation T 1,1 Z ZT 2,2 = T 1,2 , which exists by the previous theorem. Let A Cnn be a given matrix with spectrum (A) = {1 , . . . , p } . Take the Schur decomposition T 1,1 T 1,2 . . . T 1,p 0 T 2,2 . . . T 2,p , T = Q AQ = . .. . . . 0 0 T p,p where the diagonal of each Tj,j consists of j s. By the previous lemma we get: Corollary 5.8. Every A Cnn is similar to a block diagonal matrix diag(T 1,1 , T 2,2 , . . . T p,p ) , where the upper triangular matrices T j,j have constant diagonals. Since the conditioning of the Sylvester equations are inversely proportional to the separation of the eigenvalues of T i,i and T j,j this block diagonal form is computable only when the eigenvalues of A are well separated. If A Cnn has n distinct eigenvalues, this form is a diagonal matrix. Problem 5.9. Assume that an eigenvalue j is a multiple root of the characteristic polynomial and that the number of linearly independent eigenvectors corresponding to j equals to its multiplicity. Show that the corresponding T j,j is diagonal.
35
5.3. Jordan form. If j is a multiple root of the characteristic polynomial, then the number of linearly independent eigenvectors may be less than the multiplicity of j . Here we discuss, to how simple form the triangular block T j,j can be transformed in this case. A Jordan block is a matrix of the form 1 1 .. .. Ck k . J k () = . . 1 Block diagonal matrices consisting of such diagonal blocks J = diag(J k1 (1 ), J k2 (2 ), . . . , J km (m )) are called Jordan matrices. Here the j s need not be dierent. Theorem 5.9 (Jordan form). For every matrix A Cnn there exists an invertible matrix S Cnn such that S 1 AS is a Jordan matrix. Proof. Let V be an invertible matrix that transforms A to the block diagonal form of the previous corollary. Write the diagonal blocks in the form T j,j = j I + N j , when N j is a strictly upper triangular matrix (the diagonal is also zero). By 1 Proposition 5.10 below, for each N j there exists an S j such that J j = S j N j Sj is a Jordan matrix, so that A gets the Jordan form: [V diag(S 1 , . . . , S p )]1 AV diag(S 1 , . . . , S p ) = diag(1 I + J 1 , . . . , p I + J p ) . Problem 5.10. Show: J k (0)k = 0 , J k (0) ej = ej 1 , j = 2, . . . , k , J k (0)T J k (0) = 0 0 0 I k 1 ,
Denition 5.2. Matrix N is said to be nilpotent if N p = 0 for some p 1 . For example, strictly upper triangular matrices are nilpotent. Problem 5.11. Show that a matrix is nilpotent if and only if zero is the only eigenvalue of it. Proposition 5.10. Let N Cnn be nilpotent. Then there exists an invertible S Cnn and n1 n2 . . . nm 1 , j nj = n , such that S 1 N S = diag(J n1 (0), . . . , J nm (0)) . Problem 5.12. Assume N p = 0 . Show that (I N )1 = I + N + N 2 + + N p1 .
[I J k (0)T J k (0)] x = (xT e1 ) e1 .
36
EIROLA
Proof. By induction: for N = 0 there is nothing to prove and also the case n = 1 is clear. For 0 = N Cnn let p N , u Cn be such that N p = 0 , N p1 u = 0 . Also take w such that 0 = N p1 u, w = u, (N )p1 w . Set E= x Clearly N E E . The set W = {w , N w , . . . , (N )p1w } is linearly independent since if 1 w + 2 N w + . . . p (N )p1 w = 0 then multiplication by (N )p1 gives 1 = 0 after which multiplication by (N )p2 gives 2 = 0 and so on. Hence, E = W and it has dimension n p . Set U = [N p1 u, . . . , N u, u] and let the columns of V Cn(np) span E . To show that [U V ] is nonsingular assume [U V ] [ ] = 0 . If = 0 , then also = 0 , so let j {1, . . . , p} be largest such that j = 0 . We have U = V E j p k so that multiplying u E by N j 1 and using N E E we get k =1 k N j N p1 u E , i.e., N p1 u w , a contradiction. Now we have N U = U J p (0) and N V = V M for some M C(np)(np) . Hence N U V = U V i.e., N
J p (0) 0 0 M
N j x, w = 0 , j 0
J p (0) 0 0 M
. By the induction hypotheses, M is similar to some Jordan

J p (0) 0 0 JM
matrix J M and hence N has size p
. Clearly, the largest Jordan block of J M
Remark 5.2. The previous proof shows, that, if N is a real nilpotent matrix, then it can be put to a Jordan form with a real similarity transform. Problem 5.13. The minimal polynomial mA of a square matrix A is dened to be the monic (the coecient of the highest degree term is one) polynomial of lowest degree that annihilates A , i.e., satises p(A) = 0 . Show that this is unique. For each eigenvalue j , j = 1, . . . , m of A let kj be the dimension of the largest Jordan block correspoding to j . Show: mA (z ) = (z 1 )k1 (z 2 )k2 . . . (z m )km . Problem 5.14. Show, that a matrix is diagonalisable (similar to a diagonal matrix) if and only if the roots of the minimal polynomial are simple. Problem 5.15. Show, that every square matrix A satises A AT . Hint: show that .1 .1 .. .. = J k ()T . J ( ) k 1 1
1 1
Problem 5.16. The Schur decomposition can be computed as stably as one computes eigenvectors since these are used to form the Householder transformations, which operate stably. Jordan form is much more dicult to compute. Generally the
37
numerical computation of a Jordan form is an illposed task unless the eigenvalues are simple. This is because the Jordan form is not a continuous function of the matrix. Think and give examples of which steps in the derivation of the Jordan form are not continuous, in particular, in Lemma 5.7 and proposition 5.10. Problem 5.17. Show: (J k ()p )j,j +l = otherwise (J k ()p )j,j +l = 0 .
p l
pl , when 0 l min(p, k j ) ,
Problem 5.18. Show: if all eigenvalues of A satisfy || < 1 , then limk Ak = 0 . Problem 5.19. Show that J k ()
2
|| + 1 and if = 0 , then
k 2
J k ()1
j =1
||j .
38
EIROLA
6. Numerical computation of eigenvalues 6.1. Sensitivity of eigenvalues. Consider the changes of eigenvalues when the matrix is slightly perturbed. Theorem 6.1. The eigenvalues depend continuously on the matrix. Proof. The coecients of the characteristic polynomial det(I A) are sums of subdeterminants of A . Hence they depend continuously on the elements of the matrix. On the other hand, the roots of a polynomial are continuous functions of the coecients, which proves the claim. However, the eigenvalues are not necessarily dierentiable at points where the matrix has multiple eigenvalues. When approaching such points the derivative may go to innity, as the following example shows: 1/4 1 when t > 0 t {1, i, 1, i} , 1 1/4 , (A(t)) = A(t) = |t| (1 + i) 1 {1, i, 1, i} , when t < 0 . 2 t The simplest theorem concerning the location of eigenvalues is the following: Theorem 6.2 (Gershgorin discs). For any A Cnn we have
n
(A) G(A) = where Di C is a disc of radius ri =

j =i
Di ,
i=1
|ai,j | and centered at ai,i .
Proof. If (A) , then I A is singular. Let x = 0 be such that (I A) x = 0 and |xi | = maxj |xj | . Then ( ai,i ) xi = ai,j xj ,
j =i
from which
| ai,i |
j =i
|ai,j | ,
i.e., Di .
Problem 6.1. Show: (A) =

det(X )=0 0Version:
G(X 1 AX ) .
April 26, 2013
39
Consider also D 1 AT D .
Problem 6.2. Using Gershgorin discs of matrices D 1 AD , D diagonal, try to nd a small set containing (A) in the case 5 2 1 A = 1 1 0 . 0 2 2
Problem 6.3. Show that any Gershgorin disc that does not intersect others contains an eigenvalue of A . Hint: Consider the Gershgorin discs of At = D + t (A D ) , t [0, 1] , where D is the diagonal of A . Use continuity of eigenvalues.
n n
Problem 6.4. Show: |det(A)|
i=1
j =1
|ai,j |
A norm , dened for matrices is called a matrix norm, if I = 1 and it satises the inequality AB A B for all square matrices A , B of the same size. For example, the norms, which are induced by vector norms in the standard way, satisfy this. Problem 6.5. Show that for any matrix norm holds: A
(A) = max || = lim Ak

(A) k
1/k
Hence for a diagonal matrix we have The following result is often needed Lemma 6.3. Assume A
maxj |dj,j | .
< 1 for some matrix norm. Then I A is invertible.

k k =0 A
Proof. We will show that (I A)1 = Then for m > n : Sm Sn
. Set
S n = I + A + A2 + + An . = An+1 + An+2 + + Am
n+1
(1 + A
k =0
++ A
n+1
+1 A n . 1 A
mn1 )
40
EIROLA
Hence, S m S n < for m and n large enough so that {S n }n0 is a Cauchy sequence, i.e., it converges1. Further, so that taking the limits, we get (I A)S = I . (I A)S n = S n S n+1 + I = I An+1 ,
Remark 6.1. A shorter, but nonconstructive proof goes as follows: If I A is not invertible, then there exists a nonzero matrix X such that (I A) X = 0 . By scaling, we can assume X = 1 so that A = A X AX = X = 1 . Hence A 1 . Denote the distance of a point x from a set S by (x, S ) = inf yS x y , the distance of a set R from S by (R, S ) = supxR (x, S ) and the distance between sets R and S by d(R, S ) = max{ (R, S ), (S, R)} . If a diagonalizable matrix is perturbed, then the sensitivity of eigenvalues depends on how well linearly independent the eigenvectors are: Theorem 6.4 (BauerFike). If X 1 AX = = diag(1 , . . . , n ) , then ((A + E ), (A)) p (X ) E where p (X ) = X pnorm.
p p
X 1
is the condition number of the matrix X in the
Proof. Let be an eigenvalue of matrix A + E . The case (A) needs nothing to prove. Hence assume (A + E ) \ (A) , so that I A is regular and I A E is singular, as well as the matrix According to the previous lemma the distance of this from I is at least one, so that Since I is diagonal, we get ( I )1 from which the claim follows.
p
(I )1 X 1 (I A E ) X = I (I )1 X 1 EX .
p
1 (I )1 X 1 EX
(I )1
X 1
= max | i |1 =
i
1 , mini | i |
In particular, for a normal matrix A we get (an exercise): ((A + E ), (A)) E

1The 2
If the matrix is not diagonalizable, then the Schur decomposition gives:

set of matrices is nite dimensional, hence it is complete with any norm.
41
Theorem 6.5. Let Q AQ = + N be the Schur decomposition of A , where N is a strictly upper triangular matrix, hence |N |k = 0 for some k n . Then
k 1 2 j =0
((A + E ), (A)) max(, 1/k ) ,

k 2
where = E
Proof. If is an eigenvalue of matrix A + E and d = (, (A)) > 0 , then as above I A is invertible and I (I A)1 E is singular, so that 1 (I A)1 E
2
( I A )1
= ( I N )1
Since D = (I )1 is diagonal, DN is strictly upper triangular and (N D)k = 0 , so that (D 1 N )(I + DN + (DN )2 + + (DN )k1 )D = =I (N D )k = I . D
2
=(D1 N + N N DN + N DN + N (DN )k2 N (DN )k1 )D = =

1 d
Since
, this implies
1 2
( I N )
= (D
N)
2
1 d
k 1 j =0
N d
j 2
Now, if d 1 , then 1 (I N )1 On the other hand, if d < 1 , then 1 d 1/k .
E 2 , so that d . d ( I N )1 2 E 2
dk
, and
Finally, we will look how both eigenvalues and eigenvectors change when we perturb a matrix that has simple eigenvalues. For this we need the following. Theorem 6.6. Assume 1 , . . . , k are distinct eigenvalues of A Cnn . Then the corresponding eigenvectors x1 , . . . , xk are linearly independent. Proof. Let 1 x1 + 2 x2 + + k xk = 0 . Multiplication of this by 1 I A gives i.e., 1 (1 x1 A x1 ) + 2 (1 x2 A x2 ) + + k (1 xk A xk ) = 0 , 2 (1 2 ) x2 + + k (1 k ) xk = 0 ,
Similarly, after multiplication with 2 I A , . . . , k1 I A , we have k ( 1 k ) ( 2 k ) . . . ( k 1 k ) x k = 0 ,
42
EIROLA
i.e., k = 0 . Going backwards the intermediate equations we recursively get k1 = 0 , . . . , 2 = 0 , 1 = 0 . Now we show that simple eigenvalues are smooth functions. Theorem 6.7. Let A have simple eigenvalues. Then the eigenvalues of A + z E are analytic functions of z near 0 C , and the eigenvectors can also be taken to be analytic. Proof. Denote the eigenvalues by 1 , . . . , n and corresponding unit eigenvectors by 1 . .. . Then X 0 is inverible. For x1 , . . . , xn . Set X 0 = [x1 . . . xn ] , 0 = X (z ), (z ) Cnn , where is diagonal, consider equation F (z, X (z ), (z )) = 0 , where2 (A + z E ) X X F (z, X , ) = . diag(X 0X I )
n
Clearly F is analytic in z, X , . We will use the implicit function theorem. For this we need to show that the derivative of F with respect to (X , ) at (0, X 0 , 0 ) is invertible. Since it is a linear mapping from C(n+1)n into itself, it suces to show that it is an injection. So, assume Y , D Cnn , D diagonal, are such that D(X ,) F (0, X 0 , 0 )(Y , D) =
1 Then, since A = X 0 0 X 0 , we have 1 1 0 X 0 Y X 0 Y 0 D = 0 .
AY Y 0 X 0 D =0. diag(X 0Y )
1 Denote W = X 0 Y . Since the diagonal of 0 W W 0 is zero, we necessarily have D = 0 . Then 0 W W 0 = 0 implies i wi,j wi,j j = 0 , i.e., wi,j = 0 for i = j . Finally, diag(X 0 Y ) = diag(X 0 X 0 W ) =
x1
w1,1 . . .
xn
wn,n = 0 ,
i.e., W = 0 and, consequently, Y = 0 . So, D(X ,) F (0, X 0 , 0 ) is invertible and the equation F (z, X (z ), (z )) = 0 denes X and as analytic functions of z near zero. Let us compute the derivatives of X and . Dierentiation of (A + z E ) X (z ) = X (z ) (z ) gives AX (0) + EX 0 X (0)0 X 0 (0) = 0 .
2Here
diag(M ) = [m1,1 , m2,2 , . . . , mn,n ] .

1 Denoting Z = X 0 X (0) we get as in the proof above: 1 0 Z Z 0 + X 0 EX 0 (0) = 0 . 1 ( (0))i,i = (X 0 EX 0 )i,i
43
Hence, and
1 Z i,j = (j i )1 (X 0 EX 0 )i,j ,
Z i,i =
1 xi 2
xj , xi Z j,i .
j =i
From these, X (0) = X 0 Z . So: sensitivity of the eigenvalues is X 1 X X

2
sensitivity of the eigenvectors is X 1 Almost eigenvectors.
1 . mini,j |i j |
Let A Cnn be diagonalizable: A = X X 1 . Let 1 be an eigenvalue, close to it, and x a unit vector such that r = x A x is small. We want to estimate how close x is to the eigenspace corresponding to 1 . We may assume that = correspondingly, X = X1 X2 , X 1 = 1 I 0 , where 1 is not an eigenvalue of 2 . Divide, 0 2 Y 1 Y 2 .
The orthogonal projection onto R(X 1 ) the eigenspace corresponding to 1 is given by 1 P = X 1X 1 = X 1 (X 1 X 1 ) X 1 and the length of the orthogonal component is sin .
^ x ^||= sin || ^ x-Px
R(X1 )
^ Px
Lemma 6.8. I P = Y 2 Y 2 .
44
EIROLA
Proof. I P is the orthogonal projection onto R(X 1 ) and so that R(Y 2 ) = R(X 1 ) . Hence, I P = Y 2 Y 2.
Y 1 Y 2
X1 X2 = I ,
Recall that (, (2 )) is the distance from to the spectrum of 2 . Theorem 6.9. sin 2 (Y 2 ) (, (2 )) r
2
where 2 (Y 2 ) = Y 2
Y 2
1.
Proof. ( 1 ) I 0 Y 1 r = X 1 ( I A ) x = Y 0 I 2 2
1 Hence Y 2 x = (I 2 ) Y 2 r and, by the previous lemma,
Y 1 x. Y 2
(I P ) x
1 1 = Y 2 (Y 2 Y 2 ) (I 2 ) Y 2 r
Y 2 (Y which gives the result, since M 2 = M 2 .
1 2 2Y 2)
(I 2 )
Y 2
(I 2 )1 = 1/ (, (2 )) and for any matrix
1 Remark 6.2. If Y 2 has orthonormal columns, then Y 2 = (Y 2 Y 2 ) Y 2 = Y 2 and Y 2 = Y 2 = 1 , so that 2 (Y 2 ) = 1 . This is the case when A is normal.
Remark 6.3. If A is not diagonalizable but only the Jordan blocks corresponding to 1 are diagonal, then one gets a similar result, but (, (2 )) is to be replaced by max{ (I J ())1
2
| (A) \ {1 } } ,
where J () is the the largest Jordan block corresponding to . To get an estimate for this, see problem 5.19. 6.2. Computation of eigenvalues, QRiteration. Numerically it is not advisable to compute eigenvalues as the zeros of the characteristic polynomial since the roots of a polynomial are extremely sensitive functions of the coecients3. The methods of practice are iterative.
fact, usually computation of the characteristic polynomial is done numerically via rst computing the eigenvalues.
3In
45
The simplest iteration for eigenvalues is the so called power method: pick an arbitrary q 0 and iterate for k =1, 2, . . . z k = Aq k1
qk = zk / zk k = Aq k , q k
end If is a simple eigenvalue of A such that others are less in absolute value, then A has the Jordan form: 0 J A = X 1 AX = , 0 J where (J ) = (A) \ {} = || < || . Clearly: q k = k Ak q 0 , where k = z11 . . . z1k . If v k = X 1 q k , then v k = k k 1 0 1 0 ( J )k v0 .
1 Since limk ( J )k = 0 , we see that, if (v 0 )1 = 0 , then
Aq k , q k = XJ A v k , Xv k = (v k ) X X v k + where limk 0 0 v k = 0 , so that 0 J I

k
0 0 vk , 0 J I
lim Aq k , q k = lim Xv k
k
2 2
=.
In practice the power method converges4 to the eigenvector corresponding to the eigenvalue of largest absolute value. This happens with the speed k , where = max(A)\{} || / || . After the eigenvalue of largest absolute value has been found together with the corresponding eigenvectors of A and AT , we can start looking for the next one by using the following. Problem 6.6 (Deation). Let = 0 be an eigenvalue of A , x = 0 a corresponding eigenvector, and u an eigenvector of AT corresponding to scaled such that uT x = . What is the connection between the eigenvalues and eigenvectors of matrices A and A = A x uT ? Hint: see problem 5.8.
vector does not really converge unless is real and nonnegative: it will tend to cycle (/ ||)k q , where q is an eigenvector.
4The
46
EIROLA
Often we want to compute an eigenpair (p , xp ) of A such that p is close to a given . Then we can use the power method with the matrix (A I )1 . The eigenvalues of this matrix equal to 1/(j ) , j = 1, . . . , n , and the eigenvectors are the same. Thus we want that |p | < |i | , i = 1, . . . , n, i = p .
In practice the approximation is also corrected at each step. This results to the following inverse iteration, which is quite powerful. Given 0 , u0 . Solve the system (A k I ) wk+1 = uk . Set: uk+1 = w k+1 / wk+1 , k+1 = Auk+1 , uk+1 .
Orthogonal iteration. The power method can be generalized to the following higher dimensional iteration. Let Q0 Cnp be such that Q 0 Q0 = I . Iterate: for k =1, 2, . . . Z k = A Q k 1 Q k Rk = Z k end Let the eigenvalues of A be ordered such that |1 | |2 | |n | and let 1, . . . , n) + N U AU = T = diag(1 , . . . , n ) + N and V A V = T = diag( be the Schur decompositions of A and A . Then (Golub & van Loan, Theorem Ip 0 7.3.1), if |p | > |p+1 | and V Q0 Q , then 0 V P p 2 < 1 , where P p = 0 0 for every > 0 there exists a C > 0 such that U
(QRdecomposition of Z k )
Qk Q kU
Pp
< C
|p+1 | + | p |
i.e., the columns of Qk span increasingly well the same subspace as the rst p columns of the matrix U . Moreover, a similar estimate holds for the distance ((Rk ), {1, . . . , p }) . QRiteration. Consider more closely the case |1 | > |2 | > > |n | , Qk = k n n 0 0 0 [q k . Then, assuming that V [q 0 1 , . . . , qn] C 1 , . . . , q p ][q 1 , . . . , q p ] V P p 2 < 1 , for all p = 1, . . . , n , we get
k span(q k 1 , . . . , q p ) span(u1 , . . . , up ) k
for all
p = 1, . . . , n ,
47
so that T k = Q k AQk converges to an upper triangular matrix. Noticing that

T k1 = Q k 1 AQk 1 = (Qk 1 Qk )Rk
T k = Q k AQk 1 Qk 1 Qk = Qk Qk Rk Qk 1 Qk = Rk (Qk 1 Qk )
and denoting Qk = Q k 1 Qk we get: if T k1 = Qk Rk , then T k = Rk Qk .
in other words we get the iteration directly for matrices T k , where we form a QR decomposition of the previous one and the next we get, when these are multiplied in reverse order. This is called the QRiteration.
Problem 6.7. Consider the following LU iteration: Set L0 U 0 = A . Iterate: Lk+1 U k+1 = U k Lk , (new LU decomposition), k = 0, 1, . . . Think what it means, if the matrices S k = U k Lk converge to an upper triangular matrix. When does this happen? Hint: consider Lk = L0 L1 . . . Lk and show that Lk U k = ALk1 . QRiteration in the Hessenberg form. Comparing the orthogonal iteration and the QRiteration above, it seems that both require the same amount of oating point operations per step: one QR decomposition and one matrix multiplication, although in the QRiteration the multiplication includes a triangular matrix, so that it is slightly cheaper. QR iteration becomes much more economical, when the matrixes T k are of the Hessenberg form: 1 Choose Q0 such that H 0 = Q 0 AQ0 is a Hessenberg matrix (Problem 4.9).
48
EIROLA
2 For k = 1, 2, . . . , we get the QRdecomposition of the Hessenberg matrix H k1 easily using the Givens rotations: Gn1,n . . . G2,3 G1,2 H k1 # # # ... 0 = Gn1,n . . . G2,3 0 . .
0 # # 0 . . 0 0 0 0 # # 0 # 0 0 0 Gn1,n . . . G3,4 0 . . . . 0 0 0 0 # # 0 # 0 0 0 = Gn1,n 0 . . . . 0 0 0 0 # # 0 # 0 0 0 = 0 . . . . 0 0 0 0 # # # . . 0 0 # # # # . . 0 0 # # # 0 . . 0 0 # # # 0 . . 0 0 # # # # . . # 0 # # # # . . # 0 # # # # ... 0 0 # # # # ... 0 0 # # # # . . # # # # # # . . # # # # # # . . # # # # # # . . # 0 # # # # . . # # # # # # . . # # # # # # . . # # # # # # . . # #
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
= Rk ,
2 i.e., H k1 = G 1,2 . . . Gn1,n Rk . This has the op count 3n . 3 We get H k = Rk G1,2 . . . Gn1,n by applying the same Givens rotations now from the right. The result is again a Hessenberg matrix and work is also 3n2 . Repeat from 2 .
The op count of Hessenberg QRalgorithm is only 6n2 per iteration. Shifted QRiteration. If we replace the matrix A with A I , in the QRiteration (in Hessenberg form), then the p :th subdiagonal element of the matrixes H k converges to zero approximately with speed5 |p+1 |k | p | k .
The shifted QRiteration tries to nd a suitable and updates matrices H k , that are kept similar to A (and not to A I ). The basic strategy is to use the
5Here
we use the ordering |1 | |2 | ... |n |
49
Hessenberg QRalgorithm and choose to be the current element hn,n : H 0 = Q 0 AQ0 for k = 1, 2, . . . into the Hessenberg form
Q k R k = H k 1 I H k = Rk Q k + I end
= (H k1 )n,n
(the QRdecomposition using Givens)
The
where = a/m , = /m , m = |a|2 + ||2 . Hence the lowerright 22 corner of H k becomes b 2 a b a ||2 + b + 2 a = 2 . 0 a b 2 + 2 a | | b
n,n1
The following shows the convergence speed of this. If Rk X then Gn2,n1 . . . G1,2 (H k1I ) = a b , 0 0
Gn1,n
I = 0
element of H k is then
(H k )n,n1 = (a2 b2 + a2 )/m2 =
If || |a| , then the corresponding element of H k is already a very small number and if (H k )n,n1 0 , then (H k )n,n is very close to an eigenvalue of A . The next ones we nd by continuing similarly with the (n1)(n1) Hessenberg matrix H k (1 : n 1, 1 : n 1) . Problem 6.8. Test the Hessenberg QRiteration with MATLAB: H=hess(A), m=something; for k=1:m, [Q,R]=qr(H); H=R*Q; end (if H is not in the real Schur form, then iterate more). Test also the shifted QR : H=hess(A); nh=n; for l=1:n-1 while abs(H(nh,nh-1))>0.000001, mu=H(nh,nh); [Q,R]=qr(H-mu*eye(nh)); H=R*Q+mu*eye(nh); iter=[l,H(nh,nh)], end H, nh=nh-1; H=H(1:nh,1:nh); end
b 2 . |a|2 + ||2
50
EIROLA
Try rst the following matrices: A=randn(4)+i*randn(4); A=randn(7); B=randn(7); A=B*B; A=[0,1,0,0,0;0,0,1,0,0;0,0,0,1,0;0,0,0,0,1;1,0,0,0,0]; Think what causes the diculty in the second and the last cases and how to wake up the convergence. If also the eigenvectors are desired, one can compute those rst for the resulting upper triangular matrix, and from them for A , but then we should keep track of the similarity transformation that, that brings A to H k . This increases considerably the work load of the iteration. A better way is to rst compute (using shifted QR) only the approximations 1 , . . . , n of the eigenvalues and then apply the inverse iteration to improve the eigenvalues and compute the eigenvectors. Correction with inverse iteration. After the approximations 1 , . . . , n of the eigenvalues of A have been found, their accuracy can be improved and the eigenvectors found by the following iteration: for j =1, . . . , n 0 = j , q 0 {x1 , . . . , xj 1 } for k = 1, 2, . . . (A k 1 I ) z k = q k 1 qk = zk / zk
2
k = (q k ) Aq k end x j = q k , j = k end . An interesting phenomenon in ( j , (A) \ {j })k the inverse iteration is that when j , then A j I is close to a singular matrix and numerical solution of the system (A I )z k = q k1 produces large errors. It turns out, however, that these errors tend to be in the direction of the eigenvector corresponding to the eigenvalue j j , so that even the errors improve z k as an approximative eigenvector. Problem 6.9. Continue problem 6.8 by computing some of the eigenvectors using inverse iteration. Check your computations with the Matlab command eig. For each j this converges as | j j | k
51
7. Eigenvalues of a Hermitian matrix, singular value decomposition In this chapter we consider Hermitian matrices. We characterize their eigenvalues as extremal values of certain functions, look at eigenvalues of perturbed matrices. Then we show important interlacing properties of eigenvalues of certain types of perturbations. Finally we consider the singular value decomposition of a general matrix. 7.1. Eigenvalues of a Hermitian matrix. If A is Hermitian: A = A , then it is normal and hence unitarily similar to a diagonal matrix Q AQ = diag(1 , . . . , n ) . 1, . . . , n ) = Q A Q = Q AQ = diag(1 , . . . , n ) , diag( j = j , j = 1, . . . , n , i.e., the eigenvalues are real. We assume, that they so that are numbered in the nondecreasing order: 1 n . The next theorem gives the largest and smallest eigenvalues of a Hermitian matrix as solutions to extremal problems. Theorem 7.1 (Rayleigh-Ritz). Let A Cnn . Then the extremal values of the RayleighRitz quotient: x Ax x Ax and n = max Re 1 = min Re x=0 x=0 xx xx are the smallest and largest eigenvalues of the matrix
1 (A 2
Then:
+ A ) .
Proof. Matrix 1 (A + A ) is Hermitian, so that there exists a unitary U such that 2 1 (A + A )U = D = diag(d1 , . . . , dn ) , where d1 dn . We get: U 2 1 = min Re
Ux
x Ax = min Re x Ax = min x 1 (A + A )x 2 x=0 x 2 =1 x 2 =1 x x 1 = min x U 2 (A + A )U x = min x Dx = d1 .

2 =1
2 =1
Similarly, n = dn . In particular, when A is Hermitian, then 1 and n are the smallest and largest eigenvalues of A . Corollary 7.2. If A is Hermitian, 0 = x Cn and = eigenvalue and an eigenvalue .
0Version: x Ax x x
, then A has an
April 26, 2013
52
EIROLA
Remark 7.1. Let (, x) be an eigenvalueeigenvector pair of Hermitian matrix A and w a small vector. Write w = u + v , where v is perpendicular to x and u , and set x = x + u . Then (x + v ) A (x + v ) x x + v x + x v + v Av (x + w) A(x + w ) = = x x + v x + x v + v v (x + w ) (x + w ) (x + v ) (x + v ) v (A I )v =+ = + O( v 2) = + O( w 2) . 2 x+v This means that if a vector is close to an eigenvector, then the RayleighRitz quotient gives the eigenvalue with accuracy O (2) . On the other hand, if for some u = 0 we have for every v u we get (u + v ) A (u + v ) = u i.e., Au, v + v , Au = 0 . Take v = Au to u . Then Au, Au
Au,u u,u (u+v) A(u+v ) (u+v ) (u+v)
= + O ( v 2 ) , then
+ O( v 2) , u , which clearly is orthogonal
Au,u u,u
u + Au
Au,u u,u
u, Au = 0 ,
i.e., 2 Au 2 u 2 = 2 Au, u 2 . Hence the Cauchy inequality for Au and u is an equality, which is possible only if Au = u for some . We have shown that the eigenvectors and eigenvalues of a Hermitian matrix A are, respectively, the critical points and the critical values of the real valued function Ax x xx (i.e., points where the derivative vanishes and the values at these points). x Matrix A Cnn is positive denite, if x Ax > 0 for all 0 = x Cn and positive semidenite if x Ax 0 for all x Cn . By Theorem 7.1 Hermitian A Cnn is positive denite if and only if all its eigenvalues are positive. Theorem 7.1 gives the smallest and largest eigenvalues of a Hermitian matrix as the extremal values of the RayleighRitz quotient and the remark shows that all eigenvalues are the critical values of it. Next we give an extremal property of all the eigenvalues. Let Un,k be the set of
n k
matrices that have orthonormal columns.
53
Theorem 7.3 (CourantFisher min max theorem). Let 1 n be the eigenvalues of a Hermitian matrix A Cnn . Then for all k = 1, . . . , n k = min = Proof. Let Q AQ = = diag(1 , . . . , n ) . Now U Un,k Q U Un,k , so that v U AU v = min max v U QQ U v = min max v U U v . min max k U Un,k v 2 =1 U Un,k v 2 =1 U Un,k 0=v C v v
U Un,k 0=v Ck
max
U Un,nk+1 0=v Cnk+1
max
min
v U AU v v v v U AU v . v v
If U Un,k , then dim R(U ) = k and, since dim y y1 = = yk1 = 0 = n k + 1 , there exists v Ck , v 2 = 1 such that vector y = U v satises: y1 = = yk1 = 0 . Further, y 2 2 = v U U v = 1 . Hence
n
max v U U v v U U v = y y =
v
2 =1
j =k
j |yj |2 k .
v U U v k . On the other Ik hand, if in the place of U we take the matrix U 0 = , we get U 0 U 0 = 0 diag(1 , . . . , k ) , so that We obtained: for all U Un,k holds max
v
2 =1
max v U 0 U 0 v = k .
v
2 =1
This proves the rst equation. The second is obtained by noticing that that k (A) = nk+1 (A) . Then by the rst equation k = v U AU v U Un,nk+1 0=v Cnk+1 vv v U AU v . min = max U Un,nk+1 0=v Cnk+1 vv min max
Problem 7.1. Show, that the CourantFisher theorem can be rewritten in the form: k =
dim(V )=k 0=xV
min
max
x Ax x Ax = max min . dim(V )=nk +1 0=xV x x x x
54
EIROLA
Here the rst minimization (resp. maximization) is with respect to all subspaces of dimension k (resp. n k + 1). The following theorem gives good bounds for the eigenvalues of a Hermitian perturbation of a Hermitian matrix. Theorem 7.4 (Weyl). If A , B Cnn are Hermitian, then for all k = 1, . . . , n holds: k (A) + 1 (B ) k (A + B ) k (A) + n (B ) . Proof. If x 2 = 1 , then 1 (B ) x Bx n (B ) . Since U Un,k = v 2 , we get k (A+B ) = min max v U (A + B )U v U Un,k v 2 =1 min max (v U AU v + 1 (B )) = k (A) + 1 (B ) min
U Un,k v
2 =1
Uv
U Un,k
max (v U AU v + n (B )) = k (A) + n (B ) .
v
2 =1
In the case of rankone perturbation of A we get a sharper result. Theorem 7.5. Let A Cnn be Hermitian, a Cn and A = A + aa . Then 1 (A) 1 (A) 2 (A) 2 (A) . . . n (A) n (A) . Proof. Since v U aa U v = |a U v |2 , we get: k (A) = min
U Un,k
max v U (A + aa )U v min
v
2 =1
U Un,k
max v U AU v = k (A) .
v
2 =1
In the other direction we use the form 7.1 of the CourantFisher theorem: x Ax x Ax k+1 (A) = max min max min dim(V )=nk 0=xV dim(V )=nk 0=xV x x x x V a
dim(V )=nk +1
max
0=xV xa
min
x (A + aa )x x (A + aa )x max min dim(V )=nk +1 0=xV x x x x
= k (A ) . Another application of the CourantFisher theorem gives inequalities between eigenvalues of a Hermitian matrix and its submatrix:
55
Theorem 7.6. If A =
A b C(n+1)(n+1) is Hermitian, then b c
1 (A) 1 (A) 2 (A) . . . n (A) n+1 (A) . Proof. k (A) =

U Un+1,k
min
v U AU v max vv 0=vCk v U 0 A max U v 0
v v v U AU v = min max = k (A ) . U Un,k 0=vCk vv Using this for A we get

U Un,k 0=vCk
min
k+1 (A) = nk+1 (A) nk+1 (A) = k (A) .
Problem 7.2 (Poincar e separation theorem). Let A Cnn be Hermitian, U Un,m , and B = U AU . Show that for every k = 1, . . . , m holds Hint: consider rst the case U = I 0 and use the previous theorem. Then take U = I 0 Q . Finally let us mention the following without a proof: Theorem 7.7 (Homan-Wielandt). If A and E are Hermitian, then
n
k (A ) k (B ) k +n m (A ) .
j =1
(j (A + E ) j (A))2 E
2 F
7.2. Inertia. The inertia of a Hermitian matrix A is dened as: i(A) = (i (A), i0 (A), i+ (A)) , where i (A)/i+ (A) is the number of negative/positive eigenvalues (with multiplicities) and i0 (A) = n i (A) i+ (A) is the multiplicity of eigenvalue 0. Matrix B is congruent with matrix A , if there exists a regular matrix S such that B = S AS . Theorem 7.8 (Sylvesters law of inertia). Congruent Hermitian matrices have the same inertia.
56
EIROLA
Proof. Let S be regular, B = S AS , 1 (A) p (A) < 0 the negative eigenvalues of A , q 1 , . . . , q p the correspoding eigenvectors, and Q = q 1 . . . q p . Set V0 = R(S 1 Q) . Then x S ASx p (B ) = min max dim(V )=p 0=xV x 2 2 x S ASx y Q AQy max = max 2 . 0=xV0 0=y Cp x 2 S 1 Qy 2 2 Since Q AQ = diag(1 (A), . . . , p (A) and S 1 Qy we get p (B ) max p
0=y C 2 p j =1 j (A) |yj | 2 S 1 2 y 2 2 2
S 1
Qy
= S 1
p S 1
2 2
<0.
Hence i (B ) i (A) = p . Similarly, i (B ) i (A) , i+ (B ) i+ (A) , and i+ (B ) i+ (A) , so that also i0 (B ) = i0 (A) . 7.3. Singular value decomposition. The following theorem is an important tool of analysis. Further it gives means to solve illconditioned systems of equations. Theorem 7.9. For every A Cmn there exist unitary U Cmm and V Cnn such that A = U V , where = diag(1 , . . . , p ) Rmn , p = min(m, n) and 1 2 p .
1 Ax , Proof. Let 1 = A 2 , x be such that x 2 = 1 , Ax 2 = 1 . Set y = 1 so that y 2 = 1 . Let U 1 = y U 1 and V 1 = x V 1 be unitary (we get them by completing y and x to orthonormal basis of Cm and Cn , respectively). Then
U 1 AV 1 = Now B 1 2 + w = 1 w A1 w B which implies

2 2 2
y U1
Ax AV 1 =
1 w =B . 0 A1
, so that
2 2 2 2
2 1 + w 2 1 + w
2 1 + w
2 2
1 = A
= B
= 0 . Continue similarly with matrix A1 .
57
Numbers j are called the singular values of A and the columns of matrices U and V are, respectively the left and right singular vectors of A . From equation A A = V T V and AA = U T U we see, that A A and AA are unitarily similar to diagonal matrices T and T , respectively, and that the columns of U and V are eigenvectors of matrices AA and A A , respectively. Remark 7.2. If A above is a real matrix, then the matrices U and V can be chosen to be real orthogonal matrices. Remark 7.3. The singular value decomposition can also be written in the form:
p
A = U V =
j =1
j uj v j ,
where the vectors uj and v j are the rst p columns of U and V . Problem 7.3 (Polar decomposition). Show, that for every A Cmn there exists a Hermitian positive semidenite P and U Cmn , with U U = I such that A = P U . Furthermore then: P 2 = AA . When A Cnn is invertible, then A1 = V 1 U and the singular values of A1 are 1/j , j = n, . . . , 1 . In particular, A1
2 1 = n
and
2 (A) = A
A 1
= 1 /n .
Problem 7.4. Show:
n j =1 j
= |det A| , A Cnn .
Problem 7.6. Show: every A Cmn satises: A

2 F
Problem 7.5. Let 1 2 r > r+1 = = p = 0 be the singular values of matrix A Cmn . Show that r = rank(A) , R(A) = span(u1 , . . . , ur ) , and N (A) = span(v r+1 , . . . , v n ) . Show also that the pseudoinverse of A is obtained as: 1 1 .. . 1 U . ( A A + I ) A = V A = lim 1 0+ r
r 2 j . j =1
= tr(A A) =
Hint: tr is invariant under similarity transforms. Problem 7.7. Show: if A is invertible, then the singular matrix B closest to A satises: A B 2 = n .
58
EIROLA
The following more general result is also left as an exercise.

Theorem 7.10. Dene Aj = j k =1 k uk v k , j = 1, . . . , r , ( r = rank(A) ) so that Ar = A . Then rank(Aj ) = j and
A Aj
= j +1 =
rank(B )=j
inf
AB
Sensitivity of solution of a linear system, condition number. Let A be invertible and Ax = b . Consider the problems: (A + E )x ( ) = b + f , () = f , from which x (0) = A1 (f Ex(0)) and so that Ex() + (A + E )x x() x A1 ( f + E x )/ x + O (2 ) x E f + ) + O ( 2 ) . A A 1 ( b A
In other words:
relative change in
When the norm is Euclidean, then 2 (A) = A
x (A)(relative change in b + relative change in A) + O (2) .

2
A 1
= 1 /n .
Problem 7.8. Show, that the k :th singular value of matrix A Cmn is given by: AU v 2 . k = max min k U Un,k 0=v C v 2 Let A = U V be the singular value decomposition of matrix A Cmn , m n where U = U 1 U 2 , U 1 Cmn , and 1 .. . U . 1 AV = n Then 1 v V 0 Q= C(m+n)(m+n) 2U 2 2 U 1 U 1
is unitary and Q
0 A A 0
v U 1 1 2A U 2 A U 1 A U 1 V U 1 Q= AV AV 0 2 2U 0 2 = diag(1 , . . . , n , 1 , . . . , n , 0, . . . , 0) .
59
Problem 7.9. Use the Weyl theorem 7.4 to matrices 0 A A 0 and 0 A + E A+E 0 .
and show, that for all k = 1, . . . , n holds |k (A + E ) k (A)| E

n 2
Problem 7.10. Similarly, using the HomanWielandt theorem 7.7 show: |k (A + E ) k (A)|2 E
2 F
k =1
Problem 7.11. Let A Cmn , m n and A = A a Cm(n+1) . Show: 1 (A) 1 (A) 2 (A) 2 (A) . . . n (A) n+1 (A) . Hint: Theorem 7.6 for matrix A A . Problem 7.12. Let A, B Rmn . Find the connection between the singular values A B and singular vectors of matrices A + iB and . B A
60
EIROLA
8. Computation of eigenvalues of a Hermitian matrix 8.1. Hermitian QR iteration. When we apply the Hessenberg QR algorithm to a Hermitian matrix, we notice: 1) H 0 = U 0 AU 0 is Hermitian and Hessenberg form, in partcular it is a tridiagonal matrix h1 g1 g h g 1 .2 .2 . . .. .. .. H0 = g h g
n 2
g n1
n 1
hn
n 1
2) Hermitianity and tridiagonality are preserved in the QR iteration. 3) Only real shifts are needed.
We get the Hermitian QR iteration: T 0 = Q tridiagonal form 0 AQ0 for k = 1, 2, . . . choose a real Q k R k = T k 1 I T k = R k Q k + I end Problem 8.1. Count the oating point operations of this algorithm taking into account that A is Hermitian and T k s are tridiagonal matrices. Problem 8.2. Experiment this algorithm with dierent Hermitian matrices A using shift strategies a) = hn b) = the smaller eigenvalue of matrix hn1 gn1 . g n1 hn (The QR decomposition of T k1 I )
Problem 8.3. The Rayleigh quotient iteration is based on the following heuristics: Ax if x is an approximate eigenvector of A , then = xx is close to an eigenvalue x of A , so that the solution of the system (A I )z = x should be a better approximation of the eigenvector.
0Version:
April 26, 2013
61
Experiment the following algorithm for several Hermitian A x0 2 = 1 for k = 0, 1, . . . k = x k Axk xk+1 = z k / z k end First the work load of this seems big, since at every step we need to solve a system of equations, but in many cases the matrix A is sparse and approximate solution of the system is cheap. 8.2. Jacobi iteration. The idea of Jacobi iteration is based on systematic decreasing of the odiagonal elements of a Hermitian matrix. This is done with Givens rotations, which in this context are also called the Jacobi rotations. Schematically the algorithm goes as: Let A Hermitian. Set o(A) =
j =k
solve (A I )z k = xk
2
|aj,k |2 . such that .
Pick an ap,q = 0 and take unitary
ap,p ap,q aq,p aq,q
b 0 = p,p 0 bq,q
Then the correspoding rotation J p,q gives a unitary similarity transformation B = J p,q AJ p,q . Since the Frobenius norm does not change under unitary transformations and since |ap,p |2 + 2 |ap,q |2 + |aq,q |2 = |bp,p |2 + |bq,q |2 , we get:
n n
o(B ) = B
2 F
k =1
|bk,k | = A
2 F
k =1
|ak,k |2 2 |ap,q |2 = o(A) 2 |ap,q |2 .
In this method the zeros dont stay zeros, but o decreases at each step. Typically the ap,q with largest absolute value is chosen. Problem 8.4. Experiment the Jacobi iteration with some Hermitian matrices. Jacobi iteration is not very eective, but there exists versions of it that are easily parallelizable.
62
EIROLA
where T 1 Cmm and v = em + em+1 . Then in the center we have: 1 hm gm (T 1 )m,m 0 = + g m hm+1 0 (T 2 )1,1 ||2 ,
8.3. Divide and conquer method. If the aim is to exploit parallel computation, then the following approach is very natural. Consider computation of eigenvalues of a tridiagonal Hermitian matrix. Let us write it in the form: a block diagonal matrix plus a rankone correction: h1 g1 .. . T1 0 h g + vv , T = 1 .2 . = 0 T2 .. . . g n 1 g n1 hn
so that we have to take = g m . In the divide and conquer method the eigendecompositions T j = Qj Dj Q j of maQ1 0 we trices T 1 and T 2 are computed in parallel. Then setting U = 0 Q2 get D1 0 U T U = + zz , 0 D2 where z = U v . Hence we get the diagonalization of T from the eigendecompositions of the blocks, provided that we can easily compute the eigenvalues and eigenvectors of a matrix of the form: diagonal plus a rankone matrix. Eigenvalues and vectors of matrix D + zz . Let us start with the case where the eigenvalues of D are simple and all the components of z are dierent from zero. Assume further, that the diagonal elements of D are in the increasing order (which is easily obtained by permutations). Theorem 8.1. Let D = diag(d1 , . . . , dn ) Rnn , d1 < d2 < < dn and z Cn , zj = 0 j . Then (D + zz ) (D ) = and the eigenvalues 1 < < n of the matrix D + zz and correspoding eigenvectors v 1 , . . . , v n satisfy: a) v j = j (D j I )1 z b) (D + zz ) = f () = 0 , where f () = 1 + z (D I )1 z . c) d1 < 1 < d2 < 2 < . . . dn < n . Proof. Let (D + zz ) and v = 0 be such that (8.1) (D + zz )v = v .
63
We have (D ) , since if = dj , then

0 = eT j ((D dj I )v + zz v ) = zj z v ,
from which z v = 0 and Dv = dj v , so that v = ej and z v = zj = 0 , a contradiction. Hence (D + zz ) (D ) = . From equation (8.1) we also see, that z v j = 0 for all j , and that (D j I )v j = z v j z , which imply a). c) is given directly by theorem 7.5, since we dont have equalities. Multiplication of (8.1) by z (D j I )1 gives: z v j (1 + z (D j I )1 z ) = 0 , so that the eigenvalues are the zeros of f . Other zeros do not exist since
n
f () = 1 +
j =1
|zj |2 dj
and
f () =
j =1
|zj |2 >0, (d j )2
so that f is strictly increasing on each interval (, d1), (d1 , d2 ), . . . , (dn1, dn ), (dn , ) and lim|| f () = 1 , i.e., f has exactly n zeros. Problem 8.5. Show, that the general case D = diag(d1 , . . . , dn ) Rnn , z Cn can be brought by unitary transformation to the form U (D + zz ) U = D1 + z z 0 0 D2 ,
where D 1 + z z satises the conditions of the previous theorem. Hint: if dj = dk , then take a rotation Gj,k such that (Gj,k z )k = 0 . This does not change D : Gj,k DG j,k = D . Do this to all multiple eigenvalues and nally apply a permutation. Problem 8.6. Estimate the op count of the divide and conquer method, when n = 2k and all problems are divided into two of half the size until we have one dimensional problems. Assuming we have n processors that can be run in parallel what is the computing time? 8.4. Computation of the singular value decomposition. In principle one can compute the singular values for example by applying the Hermitian QR iteration to the matrix A A . The following approach, however, turns out to be more ecient.
64
EIROLA
i.e., B is a real bidiagonal matrix.
Problem 8.7. Given A Cmn , m n , show, how to nd unitary U 0 and V 0 such that b1 c1 .. .. . . B b c Rmn , n 1 n 1 U 0 AV 0 = = 0 b n After this the problem is to nd the singular value decomposition of the bidiagonal matrix B . This can be done using the following GolubKahan iteration, which essentially does the same as the Hermitian QR iteration for the matrix B T B , but transforms only the matrix B . 1) Choose a shift , for example, the smaller eigenvalue of the lower right 2 2 b2 + c2 bn1 cn1 n 2 of the matrix B T B . corner n1 2 + c bn1 cn1 b2 n 1 n b2 # 0 1 2) Take G1,2 = such that = , i.e., as b1 c1 0 0 I the rst rotation for B T B . 3) Update the matrix BGT 1,2 with rotations in turn from the left and from the right such that the result is again a bidiagonal matrix: # # # # # # # # 0 # # U 1 [] []V T 2 # # # # BGT = . . . . .. .. .. .. 1,2
# # # # # #
# #
0 0 # # # # # ... ... # # # # #
4) Set B = B and return to 1).
...
0 0 ## 0 0 # # 0 ... ... 0 0 # # # #
U 2 []
# #
0 0 # # # 0 # #
.. .. . .
# # #
U n1 []
# #
[]V T 3 =B
0 0 # # 0 0 # # 0 ... ... 0 0 # # 0 #
65
One iteration step is cheap: only O (n) ops. The result is:
T T T T T T B B =(U n1 . . . U 1 B GT 1,2 V 2 . . . V n1 ) U n1 . . . U 1 B G1,2 V 2 . . . V n1 T T T = V n1 . . . V 2 G1,2 B T BGT 1,2 V 2 . . . V n1 .
It turns out, that this is the same result that we would have got by a shifted QR iteration step for the matrix B T B (see: Golub & Van Loan: Matrix Computations). Problem 8.8. Compute the number of ops in the previous iteration. If some cj above becomes zero during the iteration, then the problem becomes block diagonal, i.e., divides into two smaller problems. Similarly if some bj becomes zero, we can divide the problem into two in the following way: B B
1
# # # # # ... ... # # #
Gj,j +2
0 0 # # # ... # # ... ... # # #
Gj,j +1
0 # # # # # ... ... ## #
Gj,n
B1
0 0 0 ... 0 # # # # ... ... # # #
66
EIROLA
9. Krylov subspace iterations for eigenvalue problems From now on we mostly consider methods that do not assume that when solving an eigenvalue problem we would have the matrix in the memory and we could manipulate it. We think having only a linear operator: A : Cn Cn . Such situations we encounter for example, when A is a subroutine which for given vector x Cn returns the vector Ax Cn . Often n is also so large, that storing an nn matrix in the memory is not reasonable. A subspace of the form Kj (A, b) = span(b, Ab, A2 b, . . . , Aj 1b) is called a Krylov subspace. The corresponding matrix K j (A, b) = b Ab . . . Aj 1b is called a Krylov matrix. One connection of such a matrix with the eigenvalues of A is seen in the following: Assume, that X = K n (A, b) is regular. Then X = x1 . . . xn gives a similarity transformation to an interesting form: X 1 AX =X 1 x2 x3 . . . xn Axn = X 1 X 0 1 1 1 2 = B , = . .. . . . 1 n1 0 I
where = X 1 An b . This is called the companion matrix of A . It satises: z 1 0 z 1 z 1 1 z 2 1 z 2 + 0 = =z . .. .. . . . . .. .. . . . . . 1 z + n1 1 z + n1
det(z I B ) =
=z det(z I B 1 ) + 0 = z n + n1 z n1 + + 1 z + 0 .
in other words = K n (A, b)1 An b gives the coecients of the characteristic polynomial of B and hence also of A . Looking for the eigenvectors of B from the
0Version:
April 26, 2013
67
system of equations
we see, that if vn = 0 , then v = 0 and on the other hand, if vn = 1 , then other components are uniquely dened. Hence geometric multiplicity of each eigenvalue is one. The following works also for nonregular K n (A, b) . Let p(z ) = z m + a1 z m1 + + am1 z + am be the minimal polynomial of A|Kn (A,b) , i.e., the monic polynomial of lowest degree such that p(A)b = 0 . Set c0 = b and cj = Acj 1 + aj 1 b , j = 1, . . . , m . Then c0 = b c1 = A b + a1 b c2 = A2 b + a1 Ab + a2 b . . . cm = Am b + a1 Am1 b + + am1 A b + am b = 0 and Acj = cj +1 aj +1c0 . Set C = [c0 . . . cm1 ] . Then A C = [A c0 A c1 . . . A cm1 ] = [c1 a1 c0 c2 a2 c0 . . . cm am q 0 ] a1 a2 . . . am 1 1 = [c0 c1 c2 . . . cm1 ] . .. 1 =CH .
1 1 .. .
. 1 + n1
..
0 1 2 . . .
v=0
Since C has linearly independent columns (why?) we get by lemma 5.1 that (H ) (A) , i.e., we obtain m eigenvalues of A by computing the roots of p. Another motivation for the use of Krylov subspaces we get from the following. Let A be real and symmetric. Then the largest and smallest eigenvalue of A are obtained xT Ax from the extremal values of the function r : Rn \ {0} R : r (x) = . xT x
68
EIROLA
Let q 1 , q 2 , . . . be orthonormal, Qj = q 1 . . . q j largest and smallest eigenvalue of B j satisfy:

v
2 =1
and B j = QT j AQj . Then the
T mj =1 (B j ) = min v T QT j AQj v min x Ax = 1 (A) x

2 =1
Mj =j (B j ) = max v
v
2 =1
QT j AQj v
max xT Ax = n (A) .
x
2 =1
How should we choose q j +1 such that mj +1 and Mj +1 would be clearly better approximations for the smallest and largest eigenvalue of A ? Since r (x) = xT Ax 2 2Ax 2 x = (Ax r (x)x) , xT x (xT x)2 xT x
and if uj , , v j span(q 1 , . . . q j ) are such that r (uj ) = mj and r (v j ) = Mj , then we would like to have: r (uj ) , r (vj ) span(q 1 , . . . q j +1) . This is possible, if This we have, if q 1 , . . . , q j is an orthonormal base of some Krylov subspace: span(q 1 , . . . q j ) = span(b, Ab, . . . , Aj 1b) , span(q 1 , . . . q j +1 ) = span(b, Ab, . . . , Aj b) . This real symmetric case1 was just for the motivation for the use of the Krylov subspaces. Now we continue with a general Hermitian matrix A Cnn . 9.1. Lanczos iteration for a Hermitian operator. We can obtain an orthonormal basis from the GramSchmidt process, i.e., from the QR decomposition. So, let K j (A, b) = b Ab . . . Aj 1b = Qj Rj . Then 0 AQj Rj = AK j (A, b) = Ab A2 b . . . Aj b = Qj Rj Aj b , I which gives 0 j 1 1 Q j A b Rj = H j R j , I where H j is a Hessenberg matrix. A product of such and an upper triangular matrix is again a Hessenberg matrix. Now, since A is Hermitian, also T j = Q j AQj is Hermitian and of the Hessenberg form, i.e., a tridiagonal matrix. Q j AQj = Rj Problem 9.1. ( A is of degree m at b .) Let m = dim(Kn (A, b)) . Show:
complex case can be done similarly, but since z z is not dierentiable, this would have caused extra confusing details.
1The
A span(q 1 , . . . q j ) span(q 1 , . . . q j +1 ) .
69
a) AKm (A, b) Km (A, b) and Km (A, b) = Kn (A, b) b) There exists a monic2 polynomial p of degree m such that p(A)b = 0 . c) AQm = Qm T m . The Lanczos idea is to compute the vectors q j and the matrix 1 1 1 2 2 . . .. 2 . . Tm = . . . m1 m1 m1 m
1 1 2
directly. Comparing the columns of the equation AQm = Qm T m we get: 1 q Aq = 1 q + 2 q Aq 2 = 1 q 1 + 2 q 2 + 3 . . . j q Aq = j 1q + j q +

j j 1 q j A q j and j
j +1
q j +1 has to be a unit vector in the Orthonormality implies: j = direction of rj = Aq j j 1q j 1 j q j . j can be chosen to be real: j = r j 2 . Continue this way until r j = 0 . Problem 9.2. Show: rj = 0 , when j < m and r m = 0 . We get the Lanczos iteration: 0 q 0 =0 , q 1 = b/ b 2 for j = 1, 2, . . . j = q j A qj rj = Aq j j 1 q j 1 j q j j = r j 2 if j = 0 stop
else q j +1 = r j /j end In the end of the iteration (j = m) we have AQm = Qm T m , so that (T m ) (A) . To obtain the Schur decomposition of the real tridiagonal matrix T m we can use the symmetric QR iteration or the divide and conquer method.
2Monic:
the coecient of the highest power is one.
70
EIROLA
The main advantage of the Lanczos iteration is that the eigenvalues of T j are good approximations of the eigenvalues of A (especially of the largest and the smallest) often already for j m . The next result shows that if j is small, then AQj Qj T j , Proposition 9.1. Above we have: AQj = Qj T j + rj eT j . Proof. K j +1 (A, b) = b AK j (A, b) = b AQj Rj = Qj +1 Rj +1 , where . Qj +1 = Qj q j +1 , We get: Rj +1 = r1,1
0 . . 0 0
Rj w j = 0 j +1
Hj
0 ... 0 j +1
1 1 T AQj = Qj H j R j + j +1 q j +1 ej Rj = Qj T j +
j +1 q eT . j j +1 j
Comparing the last columns of this to the equation Aq j = j 1 q j 1 + j q j + j q j +1 we see:

j +1 q j +1 j
= j q j +1 = r j , which shows the claim.
From this we further get the error of the eigenvalue equation: Proposition 9.2. If T j = S j diag(1 , . . . , j )S j is the Schur decomposition of T j , and Y = y 1 . . . y j = Qj S j , then Ay k k y k Proof. AY j = Y j diag(1 , . . . , j ) + r j eT j Sj , in particular, Ay k = k y k + r j eT j S j ek . Problem 9.3. Show, that above we get: (k , (A)) j |sj,k | j . The convergence theorems of the Lanczos iteration are collected under the title KanielPaige theory. The following is a sample of them. It tells that the convergence of the extremal eigenvalues is fast.
2
= rj
|sj,k | .
71
Theorem 9.3. Let v 1 be the unit eigenvector of A correspoding to 1 and 1 = arccos(|q 1 v 1 |) . Then for the iteration above we have: 0 1 1 (n 1 ) tan(1 )2 , tj 1 (1 + 1 )2
2 1 and tj 1 is the Tshebyshev polynomial where 1 = 2 n 2
of degree
j 1 .
Proof. y Q y T j y w Aw w Aw j AQj y 1 = min = min = min min = 1 . y =0 y y y =0 (Qj y ) Qj y 0=w w w 0=w Kj (A,b) w w Every w Kj (A, b) is of the form p(A)b , where p(A) is a polynomial of A of degree at most j 1 (denote: p Pj 1 ), so that 1 = If q 1 =
n k =1 ck v k 0=pPj 1
min
b p(A) Ap(A)b . b p(A) p(A)b

n
, where v 1 , . . . , v n are the orthonormal eigenvectors of A , then p(A)q 1 =

k =1
ck p ( k ) v k ,
so that for arbitrary p Pj 1 \ {0} we get 1

q 1 p(A) Ap(A)q 1 = q 1 p( A ) p( A ) q 1
n x Now, choosing p(x) = tj 1 (2 1) , we have |p(k )| 1 , k = 2, . . . , n , and n 2
n 2 k =1 |ck p(k )| k n 2 k =1 |ck p(k )| 2 |c1 p(1 )|2 1 + n n k =2 |ck p(k )| = 1 + 2 |c1 p(1 )|2 + n | c p ( ) | k k k =2
( n 1 )
n 2 k =2 |ck p(k )| n 2 k =1 |ck p(k )|
1 1 + ( n 1 )
n 1 1) |c1 |2 tj 1 (2 n 2
1 | c1 | 2
= 1 +
(n 1 ) tan(1 )2 . tj 1 (1 + 1 )2
Here we used: |c1 | = cos(1 ) . Similarly we can show that 0 n j (n 1 ) tan(n )2 , tj 1 (1 + n )2
n n1 where n = arccos(|q 1 v n |) and n = 2 n1 1 . 3Tsebyshev
these later.
polynomials are dened by tk (x) = cos(k arccos(x)) , (when |x| 1 ). More about
72
EIROLA
Problem 9.4. how fast the numbers tj (1 + ) grow with j . Answer: Compute 1 j 2 2 (1 + + 2 + ) . Hint: tj +1 (x) = 2x tj (x) tj 1 (x) . So, the Lanczos iteration seems to be an ideal method. The bad news then is that this is true only for exact arithmetic. Including the roundo errors changes the situation essentially. It is not dicult to show that for the computed vectors q j , r , and for the tridiagonal T j the error in the equation AQj = Qj T j + r j eT j + Ej usually is moderate: E 2 A 2 . The problem is that due to the roundo errors, the computed Qj loose the orthogonality of the columns quickly. This is because of the equation r j rj = Aq j j q j j 1 q j 1 , where in the computation the roundo errors are of the size Aq j j = r j 2 itself is small, then, for example,
q j q j +1 = q j r j /j 2
. When
Aq j j
This loss of orthogonality seems to happen hand in hand with the convergence. As a remedy for this a reorthogonalization strategy is proposed, i.e., make sure that the computed q j +1 is orthogonal to the previous ones by subtracting from it the components in their directions. The algorithm obtained this way is equivalent to the Arnoldi iteration (introduced later), that works also for nonHermitian matrices. But now we would loose the advantages brought by Hermitianity. A more detailed analysis shows, that the errors in the vectors q j are mainly in the direction of the already converged A eigenvectors. Then it suces to remove these components, i.e.,ortogonalize only against all the converged eigenvectors. This carries the name selective reorthogonalization in the literature. Problem 9.5. Test the Lanczos iteration for a real symmetric matrix starting with an almost eigenvector. Does the reorthogonalization help? 9.2. Bi-orthogonal Lanczos. The Lanczos iteration for a Hermitian matrix started with the attempt to compute a unitary similar transformation to a Hermitian tridiagonal form: Q AQ = T . Let A now be a general square matrix and let us try to
73
Denote V = v 1 . . . v n , V 1 = w1 . . . w n , so that w j v k = j,k . Rewrite the equations AV = V T and A W = W T columnwise: Av 1 = 1 v 1 + 1 v 2 1 w2 A w 1 = 1w1 + Av 2 = 1 v 1 + 2 v 2 + 2 v 3 2 w 3 A w 2 = 1 w1 + 2w2 + . . . Av j = j 1v j 1 + j v j + j v j +1 j wj +1 A w j = j 1w j 1 + j wj +
Multiplying the Av j equation by w j gives j = w j Av j . Denote
compute a similarity transformation to a tridiagonal form: 1 1 1 2 2 .. .. 1 . . 2 V AV = T = . .. . n1 n1 n 1 n
r j = Av j j v j j 1 v j 1 so that v j +1 = rj /j , For j and j we get the condition: 1= w j +1 v j +1 s j rj = . j j
sj = A wj j wj j 1w j 1 , j . w j +1 = sj /
which causes j to have the same absolute value j = The iteration can be continued until
1/2 This does not determine j and j uniquely. The usual choice is j = |s , j rj | s j rj j
s j r j = 0 . Then, if r j = 0 , we are in a good
situation, since then holds: AV j = V j T j , where V j = v 1 . . . v j and T j is the upper left j j corner of T and (T j ) (A) . Further, from the eigenvectors y k of T j we get eigenvectors V j y k of A . Similarly for A , if sj = 0 .
74
EIROLA
The situation becomes more complicated if s j r j = 0 without either of them being nullvector. Then the strategy is to try a bigger step. Release the requirement that T should be a tridiagonal and let it be thicker at some places. This strategy is called the Look ahead Lanczos iteration and it is a subject of active current research. The loss of orthogonality ( w j v k = j,k ) caused by the round-o errors is also here a problem and this is again tried to be remedied by (selective) reorthogonalization.
9.3. Arnoldi iteration. The most reliable Krylov subspace iteration is based on the computation of an orthonormal basis for Kj (A, b) , similarly to the Lanczos iteration for Hermitian matrices. For general A we dont have the luxyry to obtain a tridiagonal form and just a few inner products per step. Full orthogonalization of the basis vectors is then needed. Think again that we would perform the QR decompositions to the Krylov matrices K j (A, b) = b Ab . . . Aj 1b = Qj Rj . Then K j +1 (A, b) = b AK j (A, b) = b AQj Rj = Qj +1 Rj +1 , where Qj +1 = Qj q j +1 , We get: Rj +1 = Rj y j = 0 j +1 r1,1
0 . . 0 0
Hj
0 ... 0 j +1
AQj Rj = Qj H j + j +1 q j +1 eT j , i.e.,
1 T T AQj = Qj H j R j + j +1 q j +1 ej /j = Qj H j + hj +1,j q j +1 ej , 1 where H j = H j R is a Hessenberg matrix and hj +1,j = j +1 /j . As in the j problem 9.1 we get again: if m = dim(Kn (A, b)) , then
a) AKm (A, b) Km (A, b) and Km (A, b) = Kn (A, b) b) If A is invertible, then there exists a polynomial p of degree m such that p(0) = 1 and p(A) b = 0 . c) AQm = Qm H m .
75
are computed directly, without performing the QR decomposition of K j (A, b) . From equation Aq j = Qj w j + hj +1,j q j +1 we see, that w j = Q j Aq j . We get the Arnoldi iteration (with modied GramSchmidt): q 1 = b/ b 2 , Q1 = q 1 for j = 1, 2, . . . r j = Aq j for k = 1, ..., j , hj +1,j = r j end In the end of the iteration
(j =m) 2
In the Arnoldi iteration the vectors h1,1 h2,1 0 Hm = 0
q j and the Hessenberg matrix h1,2 h1,3 . . . h1,m h2,2 h2,3 . . . h2,m h3,2 h3,3 . . . h3,m . .. . . . 0 0 . . . hm,m
hk,j = q k rj ,
r j = rj hk,j q k
end
if hj +1,j = 0 stop else q j +1 = r j /hj +1,j ,
Qj +1 = Qj q j +1
we have AQm = Qm H m , so that (H m ) (A) .
Modied GramSchmidt means that the operation rj = (I Qj Q j )Aq j is computed such that the components of r 0 = Aq in the directions of each q k are j j removed one at a time. This way the non-orthogonality that already exists in the vectors q 1 , . . . , q j causes less errors. Arnoldi iteration is commonly used in situations where we are not interested in all the eigenvalues of A but only the outer ones hoping that the eigenvalues of H j would approximate these well already for small values of j like in the Lanczos iteration in the Hermitian case. If H j y k = k y k ,
(j ) (j ) (j ) (j )
yk
(j )
=1,
(j ) (j )
then the numbers k are called the Ritz values and the vectors uk = Qj y k the Ritz vectors. These are approximate eigenvalues and eigenvectors of A . Equation AQj = Qj H j + hj +1,j q j +1 eT j gives now Auk k uk
(j ) (j ) (j ) 2
= |hj +1,j | (y k )j
(j )
and the BauerFike theorem 6.4 gives an error bound for the Ritz values:
76
EIROLA
Theorem 9.4. If A is diagonalizable : V 1 AV = D , then (k , (A)) 2 (V ) |hj +1,j | (y k )j . In the sequel denote by P j = Qj Q j the orthogonal projection onto Kj (A, b) . For every k (A) dene: k = min
(j ) (j ) (j ) (j )
pPj 1 p(k )=1
(A)\{k }
max
| p( ) | .
So, k measures, how small a polynomial of degree j 1 can become on the rest of the spectrum of A when at k it has the value one. Lemma 9.5. Let A be diagonalizable : Av k = k v k , det(v 1 , . . . , v n ) = 0 . Let q 1 = n k =1 ck v k . Then vk P j vk
2
vk
= 1 , k = 1, . . . , n ,
k | ck |
(j )
i=1 i=k
| ci | .
Proof. ( I P j ) ck v k
2
= min
pPj 1
ck v k p(A)q 1
n
= min
pPj 1
ck v k
n
ci p ( i ) v i
i=1
2 n
min
pPj 1 p(k )=1
ci p ( i ) v i
i=1 i=k
(j ) k
i=1 i=k
|ci |
Each eigenvector v k is approximately in (j ) Kj (A, b) if the correspoding k is small. For example, if (A) \ {1 } is in the disc of radius and centered at c and if 1 outside the disc then using the polynomial we get p ( z ) = ( z c) / ( 1 c) 1
(j +1) j j
c
(A) n f
1g
j . | 1 c| j
Denote: Aj = P j A|Kj (A,b) .
77
Lemma 9.6. p Pj 1 = p(A) b = p(Aj ) b and p Pj = P j p(A) b = p(Aj ) b . Proof. If 0 i < j , then Ai b Kj (A, b) , so that P j Ai b = Ai b . We get recursively:
2 Ai j b = P j A . . . P j AP j AP j Ab = P j A . . . P j AP j A b
= P j A . . . P j A3 b = P j Ai b = Ai b .
j Finally: Aj jb = P jA b .
The following theorem tells something about, where the Ritz values tend to be. Theorem 9.7. The characteristic polynomial pj of H j solves the minimization problem pj (A)b 2 = min p(A)b 2 .
pPj p monic j j (I Qj Q j )A b
Moreover, this minimal value is
= |j +1 | = b
2 k =1
|hk+1,k | .
Proof. Since Aj = P j A|Kj (A,b) = P j AP j |K

j (A,b)
= Qj Q j AQj Qj |
Kj (A,b)
= Qj H j Q j|
Kj (A,b)
, the matrices Aj and H j are similar. By the and because Qj = Q j |K (A,b) j CayleyHamilton theorem we have: pj (Aj ) = 0 . Hence for every y Kj (A, b) we get using the lemma above: 0 = y , pj ( A j ) b = y , P j pj ( A ) b = P j y , pj ( A ) b = y , pj ( A ) b . In other words pj (A)b Kj (A, b) . If p is monic and of degree j , then p pj Pj 1 , and (p pj )(A)b Kj (A, b) , so that by the Pythagorean theorem: p(A)b
2 2
= pj (A)b + (p pj )(A)b
2 2
= pj (A)b 2 2 + (p pj )(A)b
2 2
pj (A)b
2 2
Hence pj solves the minimization problem. Since pj (A)b = Aj b rj 1 (A)b Kj (A, b) , where rj 1(A)b Kj (A, b) , we notice, that rj 1 (A)b is the orthogonal projection of Aj b onto Kj (A, b) . In other j j words rj 1 (A)b = Qj Q j A b . Further, A b = Qj +1 Rj +1 ej +1 , so that pj ( A ) b
2
= (I Qj Q j ) Qj q j +1
Rj w j e 0 j +1 j +1
= |j +1 | .
78
EIROLA
On the other hand Aj b = AQj Rj ej = (Qj H j + hj +1,j q j +1 eT j )Rj ej , from which

j (I Q j Q j )A b 2
= |hj +1,j | |j | .
This gives: pj ( A ) b
2
= |j +1 | = |hj +1,j | |j | = |hj +1,j | |hj,j 1| |j 1 |

j
= |hj +1,j | |hj,j 1| . . . |h2,1 | |1 | = b
2 k =1
|hk+1,k | .
Problem 9.6. How pj is seen in the matrix K j (A, b) AK j (A, b) ? Using the theorem above we nally get the result showing the speed of approximation of the eigenvalues of A . Here the main role is played by the geometric mean of the subdiagonal entries of the Hessenberg matrix H j . Theorem 9.8. If A is diagonalizable : V 1 AV = D , V = v 1 , . . . , v n , and b= n i=1 ci v i , then (k , (H j )) V 1 | ck |
1/j 2
|j +1 |
V 1 2 b | ck |
j l=1 (z (j )
j 2 k =1
1/j
|hk+1,k |
Proof. Set = (k , (H j )) . Now pj (z ) = From the previous theorem we get:

n
l ) , so that |pj (k )| j .
n
|j +1 | = pj (A)b 1 V 1
2
= VV
n i=1 2
1 i=1
ci p j ( i ) v i
2 1/2
= V
i=1
ci pj (i )ei
| ci | | p j ( i ) |
| ck | j |ck | |pj (k )| . V 1 2 V 1 2
In the following gures the eigenvalues of A are denoted by plus and the eigenvalues of H j by dots with darkness increasing with the iteration index. On the left there are ten iterations, on the right j = 20 . These show clearly, how the outer eigenvalues are found soon.

1 1
79
0.75
0.75
0.5
0.5
0.25
0.25
-0.75
-0.5
-0.25
0.25
0.5
0.75
-0.75
-0.5
-0.25
0.25
0.5
0.75
-0.25
-0.25
-0.5
-0.5
-0.75
-0.75
0 1 Cnn and b = e1 . Compute the Ritz values. I n 1 0 This is a bad case for the Arnoldi iteration. Problem 9.7. Let A =
80
EIROLA
10. Classical iterations for linear systems In the rest of these notes we consider iterative solution of an equations Ax = b .
n n
system of linear
Assume A is regular, so that the equation has a unique solution x . Many iterations are based on an invertible approximation M of A , such that the system M y = c is easy to solve. Such M is called a preconditioner. Then the equation is written in the form M x = N x + b , where N = M A . Starting with an initial guess x0 we iterate (10.1) M xk+1 = N xk + b , k = 0, 1, 2, . . .
Taking the limits of both sides, we see that if this converges, then the limit is a solution. Iteration 10.1 is economical if it converges fast and if op counts of the operations: multiplication by N and solving the system M y = c are small. This is often the case especially when A is sparse, i.e., when most of its elements are zero. Denote the equation error, i.e., the residual by r k = b Axk . Then =(I AM 1 )b (A AM 1 A)xk = (I AM 1 )(b Axk ) = N M 1 rk , ek = x xk = A1 b A1 A xk = A1 r k satises, respectively, ek+1 = A1 r k+1 = A1 (I AM 1 )r k = (I M 1 A)ek = M 1 N ek
r k+1 =b AM 1 (N xk + b) = b AM 1 b AM 1 (M A)xk so that r k = (N M 1 )k r 0 . The error in the solution
and ek = (M 1 N )k e0 .
The spectral radius is dened: (A) = max(A) || . By problem 5.18 we get: Lemma 10.1. Iteration 10.1 converges for all x0 , if and only if (M 1 N ) < 1 . The spectral radius can be characterized also by: Lemma 10.2. (A) = lim Ak
k
1 k
Here the norm can be any matrix norm. In the proof we need the following result.
0Version:
April 26, 2013
81
Problem 10.1. Let and be two matrix norms in Cmn . Show that there exist positive constants c, C such that for all M Cmn . This means that the norms are equivalent. Proof. By the previous problem we have positive c, C such that for every matrix M holds c M 2 M C M 2 , c k Ak Since c , C
1 k 1 k 1 1 k
c M M
C M
from which
be the Jordan form of A . Then Ak = XJ k X 1 so that ( X

1 2
1 , it suces to prove the claim for the 2norm. Let J = X 1 AX X 1

1 1 )k 2
Ak
1 k
C k Ak
1 k
Jk
1 k
Ak
1
1 k
( X
X 1 2 ) k J k
1 k
k is enough. Since J k 2 = maxj J (j , rj )k 2 , so that showing (J ) = limk J k 2 where the matrices J (j , rj ) are the Jordan blocks of J and for these problem 5.17 gives1:
||k J (, r )k 2
r max
0l<r
1
k l
||kl
r k
r max(1, r ) k r ||k .
The claim follows then from lim (k r ) k = lim e k log(k ) = 1 .

k
Lemma 10.3. For every A Cnn and > 0 there exists a vector norm such that for the corresponding matrix norm we get A
(A) + .
1/k
1 = A and x = supk0 B k x . Since limk Ak Proof. Set B = (A )+ m m (A) there exists m such that A < ((A) + ) . Hence x is nite for every x . Then we get:
1) x
= 0 =
x = 0 = x = 0 supk0 B k x + supk0 B k y
2) x + y y 3) x
1using
= supk0 B k (x + y )
= x
= supk0 B k x = || x
A
2
also: A Cnn =
n maxi,j |ai,j |
82
EIROLA
I.e., we really got a norm. Finally, Ax
= sup B k Ax = ((A) + ) sup B k x

k 0 k 0
((A) + ) sup B x = ((A) + ) x Remark 10.1. We have the following inequalities where C = maxk0 B k . x x
k 1
C x ,
Problem 10.2. Using a suitable norm for M 1 N give another proof for lemma 10.1. Problem 10.3. Assume you want to solve x = Cx + c . If (C ) 1 the iteration xk+1 = Cxk + c does not converge. Consider iterating for = 0 . Show that if it converges, then the limit is a solution. Discuss the choice of . This technique is called relaxation. (Underrelaxation for < 1 and overrelaxation in case > 1 .) 10.1. Jacobi and GaussSeidel iterations. Write A = D + L + U , where D = diag(a1,1 , . . . , an,n ) , L consists of the subdiagonal elements of A and U is the part above the diagonal. In the Jacobi iteration we choose M J = D , so that the iteration becomes: Now the convergence depends on the spectral radius of the matrix E J = D 1 (L + U ) . Typically it converges the better the more diagonally dominant A is. Lemma 10.4. Let A be strictly diagonally dominant, i.e., for all j holds |aj,j | > k=j |aj,k | . Then the Jacobi iteration converges. Proof. Let (E J ) , and let x be the corresponding eigenvector and xj the component of x with the largest absolute value. From equation (L + U )x = Dx we then get |aj,k | |xk | , aj,k xk |aj,j xj | =
k =j k =j
xk+1 = ((1 ) I + C ) xk + c
xk+1 = D 1 (b (L + U )xk ) .
from which
||
k =j
|aj,k | |xk | <1. |aj,j | |xj |
83
In the GaussSeidel iteration we take M GS = D + L . Then in each step we need to solve a lower triangular system (with forward substitutions): (D + L)xk+1 = b U xk and the iteration becomes: xk+1 = (D + L)1 (b U xk ) . The convergence of this is determined by the spectral radius of the matrix E GS = (D + L)1 U . Example 10.1. For A = Gauss-Seidel: EJ = 0
1 2
2 1 1 2 ,
we get the iteration matrices for Jacobi and

1
1 2
E GS =
2 0 1 2
0 1 0 = 0 0 0
1 2 1 4
1 Hence (E J ) = 1 , (E GS ) = 4 . 2
Problem 10.4. Show: if A is strictly diagonally dominant, then the GaussSeidel iteration converges. For this we further have Theorem 10.5. If A is Hermitian and positive denite, then the GaussSeidel iteration converges. Proof. If A is Hermitian, then U = L and we have to show that ((D + L)1 L ) < 1 . Now the entries of D are positive so that it has a positive square root D 2 and (D + L)1 L is similar to matrix D 2 (D + L)1 L D 2 = (I + D 2 LD 2 )1 D 2 L D 2 = (I + L1 )1 L 1 , where L1 = D 2 LD 2 . If (I + L1 )1 L x 2 = 1 , then L 1 x = x , where 1x = x + L1 x , so that = / (1 + ) , where = x L1 x . Since D 2 AD 2 = I + L1 + L 1 is positive denite, we get 1 + + > 0 , from which ||2 2 || = <1. = 1+ 1+ 1++ + ||2
1 1 1 1 1 1 1 1 1 1 1
84
EIROLA
10.2. SOR iteration (overrelaxation). The GaussSeidel iteration can be written as: where the components of xk+1 are already computed when they are needed. The Successive Over Relaxation method starts from the idea that if this is a good step, then why not take it a little longer (10.2) where > 1 (usually). Writing SOR in the standard form with M and N we get: i.e., M xk+1 = N xk + b , where M =
1 1 1 ( D + L)xk+1 = [( 1)D U ] xk + b ,
xk+1 = D1 (b U xk Lxk+1 ) = xk + (D1 (b U xk Lxk+1 ) xk ) ,
xk+1 = xk + [D 1 ( b U xk Lxk+1 ) xk ] ,
(D + L ) ,
N =
With the value = 1 SOR is just the GaussSeidel iteration. The convergence of 1 SOR is determined by the spectral radius of the matrix E = M N . First result concerning this is Lemma 10.6. (E ) | 1| . Hence it is necessary for the convergence that (0, 2) . Proof. Since the determinant is the product of the eigenvalues and the determinant of a triangular matrix is the product of the diagonal elements we get:
j 1 1 |j | = det(M N ) = det((D + L) ((1 )D U ))
((1 )D U ) .
Hence at least one of the eigenvalues has absolute value |1 | .
= det(D + L)1 det((1 )D U ) = |1 |n .
Theorem 10.7. If A is Hermitian and positive denite, then SOR converges for all (0, 2) .
1 Proof. Fix (0, 2) . We have A = D + L + L and M = D + L . Now the matrix
is positive denite. Set Q := A1 (2 M A) . If , x is an eigenpair of Q then (2 M A) x = A x and

x (2 M A) x = x Ax ,
M + M A =
2 D + L + L D L L = ( 1) D
x (2 M A) x = x Ax
85
i.e., x (M + M A) x = Re x Ax . Since M + M A and A are positive denite, we get Re > 0 . Now 1 1 M (Q I )(Q + I )1 = (2 A1 M 2 I ) 2 A = E .
Hence the eigenvalues of E are of the form i , > 0 we have 1 = +1
1 +1
, (Q) and for = +
(1 )2 + 2 <1. (1 + )2 + 2
If SOR converges for some then, since spectral radius is a continuous function of the matrix, it has a minimum < 1 on the interval (0, 2) . This minimum point opt is called the optimal relaxation parameter. Generally it is a dicult task to nd it, but there are many cases, where it is well known. In particular, in some cases of discretized partial dierential equations. There a thumb rule is that if the Jacobi iteration converges, then GaussSeidel converges twice that fast and the optimal SOR much faster. Denition 10.1 (R. Varga). Matrix A = L + D + U is said to be consistently ordered, if the spectrum of B = D 1 L + is independent of C \ {0} . An important set in this class is the following. We say that the matrix A has property AP , if it can be similarity transformed with a permutation to the form: D1 L1 T A = 0 . . . 0 U1 0 ... 0 . .. . . . .. . 0 , .. . U r 1 Lr1 D r
1
D 1 U
D2 U 2 L2 D 3 .. .. . . ... 0
(10.3)
where matrices D j are invertible and diagonal.
86
EIROLA
Problem 10.5. Let B Rnn be a tridiagonal matrix, whose diagonal is constant 4 and other elements are ones. Find a permutation that takes the nmnm matrix B I 0 ... 0 . .. . . I B I . . A = 0 I B .. 0 . . . . .. .. .. I . . 0 ... 0 I B into the form above. This A comes from the uniform discretization of the Laplace operator u = uxx + uyy in a rectangle of the plane. Theorem 10.8. Assume A has property AP . Then it is consistently ordered. Proof. Let A = AT = D + L + U be of the form of (10.3). Since a permutation similarity transformation maps the diagonal elements onto the diagonal and others outside it we have: D = D T and L + U = (L + U )T . Now E J = 1 D (L + U ) is similar to E J : so that (E J ) = (E J ) . Set B = D1 L + 1 D 1 U . Then B T = D ( L + 1 U ) =
1 I I . ..
E J = D 1 T (L + U )T = E J T , D (L + U )
r 1 I 1
1 I . ..
,
1r I
so that B E J E J for all C . The following result reveals a connection between the convergence speeds of the Jacobi iteration and SOR for consistently ordered matrices Theorem 10.9. Assume A is consistently ordered. Then a) (E J ) = (E J ) . b) If = 0 , then for any (E J ) there exists (E ) such that and for any (E ) , = 0 there exists (E J ) such that the equation above holds. Proof. a) B 1 = E J and B 1 = E J . b) Write:
1 1 E = ( D + L)1 ( (1 ) D U ) = (I + D1 L)1 ((1 ) I D 1 U ) .
( + 1)2 = 22
87
Since det(I + D1 L) = 1 , we get det(I E ) = det(I + D1 L) det(I E ) = det(I + D1 L (1 )I + D1 U )
= det(( + 1)I + D1 L + D1 U ) .
2 2 2 Suppose ( E J ) and satises ( + 1) = . By a) we can assume + 1 = .
Now, if = 0 , then = 1 and det(E 1 ) = det((D + L)1 U ) = 0 , since det(U ) = 0 . Hence (E 1 ) . 1 det(I E ) = det(( + 1)I + ( D 1 L + D 1 U ) 1 1 = ( )n det( + D 1 U ) . D 1 L + I + ( 1 1 But + = ( E ) = ( D 1 U ) . Hence det(I E ) = 0 . D 1 L + J For the reverse, read the proof above backwards. Problem 10.6. Show: if A is consistently ordered then the GaussSeidel iteration converges if and only if the Jacobi iteration converges and (E GS ) = (E J )2 . Problem 10.7. Show: if A is Hermitian and is consistently ordered, then the Jacobi iteration converges. Further, if the eigenvalues of the Jacobi iteration are real, then the following gives the optimal value of the relaxation parameter. Theorem 10.10. Let A be consistently ordered and (E J ) (1, 1) . Then 2 opt = 1 + 1 (E J )2 and 1 1 (E J )2 . (E opt ) = opt 1 = 1 + 1 (E J )2 Proof. Let = (E J ) < 1 . Then (E J ) and the corresponding eigenvalue of SOR satises 2 2(1 + 1 2 2 ) + ( 1)2 = 0 , 2 and a short computation gives (10.4) =
1 2
If = 0 , then
1 + 2 2 /4
88
EIROLA
Let 0 (0, 2) be the zero of the discriminant:

2 1 0 + 2 0 /4 = 0 ,
i.e.,
0 =
2 1+ 1 2
1 + When < 0 , the function f ( ) = 2
f ( )
=1 2
On the other hand, when 2 > > 0 , we get: || =

1 2
1 + 2 2 /4 1 2 = 1 + 2 2 /4 1 + 1 2 2 2 2 1 + /4 1 1 < =0. (1 /2)2 1 + 2 2 1 + 2 2 /4 +i 1 2 2 /4

2
1 2
2 1 + 1 2
1 + 2 2 /4 is decreasing:
=2 2 /4 + 1 2 2 /4 = 1 ,
so that || is increasing. Hence 0 gives the minimum for || . Still we need to check the other eigenvalues of SOR. If (E 0 ) , then we get it from some 2 eigenvalue [0, ] of E J like the formula (10.4), where now 1 0 + 2 0 /4 < 0 , so that
2
| | =
1 0 2
+i
0 1
2 2 0 /4
= 0 1 .
Hence the optimal relaxation parameter shifts all the eigenvalues of the iteration matrix to the circle of radius 0 1 . Example 10.2. Let A nn be a tridiagonal matrix with diagonal constant 2 and ones on both sides of it so that it is already in the form (10.3): A= Since we see, that v = (sin(), sin(2), . . . , sin(n)) is an eigenvector of A provided that sin((n + 1)) = 0 j = 2(cos( nj ) + 1) , +1 , i.e., if = j/(n + 1) , j = 1, . . . , n . Hence A has eigenvalues and eigenvectors: v j = (sin( nj ), sin(2 nj ), . . . , sin(n nj )) . +1 +1 +1
1 (A 2I ) , we get (E J ) = cos(/(n + 1)) . j = 1, . . . , n . Now, since E J = 2 2 1 1 2 1 . 1 .2 ... .. .. 1 1 2
sin((k 1)) + 2 sin(k) + sin((k + 1)) = 2(cos() + 1) sin(k) ,
89
If n = 20 , we get (E J ) = 0.988831 and the optimal relaxation parameter is opt = 1.74058 which gives the spectral radius (E opt ) = 0.74058 for SOR. In the following = (E ) . On the right eigenvalues of the Jacobi iteration are drawn grey, those of GaussSeidel are small black and of SOR larger black points.
0.8 0.6
0.8 0.6 0.4 0.2
0.4 0.2 -1 -0.5 -0.2 0.5 1
0.5 1 1.5
-0.4 -0.6 -0.8
opt
Problem 10.8. In the problem 10.5 compute the optimal for SOR iteration. Hint: Look for the eigenvectors of A as Cartesian products of the eigenvectors of Example 10.2. 10.3. SSOR (Symmetric SOR). In the next section we will consider Tsebyshev acceleration of an iteration. It can become very eective when the spectrum of the iteration matrix is real. Even if A is Hermitian, the iteration matrix of the SOR iteration
1
E = (D + L)1 ((1 )D U )
1
(D + L) xk+1 =
is usually not Hermitian, and it does not have real spectrum. The idea of symmetric SOR (SSOR) is to change the roles of L and U after every step, i.e., every second iteration is performed using matrices M =
1
[(1 )D U ] xk + b
(D + U )
and
N =
Usually SSOR is written in the form:

2
[(1 )D L] .
(D + L) xk+ 1 = [(1 )D U ] xk + b
2
(D + U ) xk+1 = [(1 )D L] xk+ 1 + b .
Then, if A is Hermitian, we have U = L , M = M , N = N and the iteration matrix becomes 1 1 E SSOR = (M ) N M N .
90
EIROLA
Neither this is usually Hermitian, but since2

1 1 1 D 1 N D M D = D ((1 )D L) (D + L)
= ((1 )I D1 L) (I + D1 L)1
we get
1 = (D (I + D1 L))1 D ((1 )I D1 L) = M N ,
= (I + D1 L)1 ((1 )I D1 L)
1 1 1 E SSOR = (M ) DM N D N . 1 Now, if D has positive diagonal elements, then N D N = K K , where K = 1 D 2 N , and 1 KE SSOR K 1 = K (M D 1 M ) K has real eigenvalues. is Hermitian and hence the iteration matrix E SSOR
We will not consider the selection of the relaxation parameter for symmetric SOR. Let us mention only that the of SOR is usually good also for SSOR and that SSOR is not very sensitive with respect to the parameter. Problem 10.9. Experiment Jacobi, GaussSeidel, SOR and SSOR for the problem 10.5 with b a random vector and m n 20 . Take the from problem 10.8.
Theorem 10.11. Assume A = A is positive denite. Then SSOR converges for every (0, 2) , Proof. Write
1 1 1 1 E SSOR = ( (D + U ))1 ((1 )D L) ( (D + L))1 ((1 )D U ) 1 1 I + K )1 ( 1 I K) ( I + K )1 ( 1 I K ) D 2 , = D 1/2 ( 1 1 = ( D + L )1 ( 1 D L) ( D + L)1 ( 1 D L )
1
is similar to where K = D 2 LD 2 . Hence E SSOR which further is similar to

1 1 S := ( I + K )1 ( 1 I K) ( I + K )1 ( 1 I K ) ,
i.e., a positive semidenite matrix. Hence the spectrum of E SSOR is nonnegative. 1/2 1/2 Set B = D AD = K + I + K . Clearly B is positive denite. Since
1 1 1 1 ( I + K ) S ( I + K )1 = ( I + K )1 ( 1 I K ) ( 1 I K ) ( I + K )1 ,
1 1 1 1 ( I +K )1 ( 1 I K ) = ( I + K )1 ( I +K K I K ) = I ( I + K )1 B ,
we use the following result: if A Cnn p and q are polynomials, det(q (A)) = 0 , then p(A)q (A) = q (A)p(A) and p(A)q (A)1 = q (A)1 p(A) .
2Here
91
we get Hence S is similar to

1 1 I + K ) 1 B ] [I ( I + K )1 B ] . S = [I (
1 1 B 1/2 SB 1/2 = [I B 1/2 ( I + K )1 B 1/2 ] [I B 1/2 ( I + K )1 B 1/2 ]
1 1 1 1 I + K )1 [ I +K + I + K I K K ] ( I + K )1 B 1/2 = I B 1/2 ( 1 1 2 1)B 1/2 ( I + K )1 ( I + K )1 B 1/2 . = I (
1 1 1 1 = I B 1/2 [( I + K )1 + ( I + K )1 ( I + K )1 B ( I + K )1 ]B 1/2
Here a positive denite matrix is subtracted from I . Thus the eigenvalues of S are less than one. 10.4. Tshebysheviteration. Assume that for the problem Ax = b we have obtained some splitting A = M N and the corresponding iteration: M xk+1 = N xk + b ,
k
E = M 1 N .
Let us try to accelerate the iteration in the form yk =

j =0
k,j xj
trying to choose the coecients k,j such that y k would be a better approximation to the solution. If x0 already happened to be the solution then xk = x for all k , so that naturally we then want also y k = x . This we get by requiring k j =0 k,j = 1 j j for all k . Then, since xj x = E (x0 x) = E e0 , we obtain that the error of y k satises
k k
(10.5)
yk x =
j =0
k,j (xj x) =
k,j E j e0 = pk (E )e0 ,
j =0
where pk is a polynomial of degree k such that pk (1) = 1 . Naturally we would like to have pk such that pk (E ) 2 is small. Consider here only the case where E is Hermitian and assume further that (E ) < 1 , i.e., that the basic iteration converges, and that we know some bounds for the spectrum of E : Then E is unitarily similar to a diagonal matrix: E = QQ , so that pk (E ) = Qpk ()Q . We obtain: pk ( E )
2
1 < 1 n < 1 .
= max |pk ()| max |pk ()| .

(E ) [, ]
92
EIROLA
Problem 10.10. Corresponding to this what do you get for the case of SSOR when A is Hermitian. Then E is not symmetric, but it is diagonalizable and has real eigenvalues. Now our task is to nd a polynomial pk such that pk (1) = 1 and max[, ] |pk ()| is the smallest possible. The solution is found from the Tshebyshev polynomials: tk ( ) = cos(k arccos( )) . Lemma 10.12. If qk is a polynomial of degree at most k and satises |qk ( )| 1 for all [1, 1] , then qk (s) tk (s) for all s>1.
Proof. Assume the contrary: let qk satisfy the assumptions but for some s > 1 holds qk (s) > tk (s) . Let < 1 be such that qk (s) > tk (s) . Now Since |qk ( )| for all [1, 1] , setting r ( ) = qk ( ) tk ( ) we get a polynomial of degree k which changes its sign on each interval (j +1 , j ) . Hence r has k zeros on the interval [1, 1] and r (1) < 0 . Moreover, r (s) > 0 , so that r has k + 1 zeros, which is impossible. tk (j ) = (1)j , where j = cos(j/k ) , j = 0, . . . , k
1.5 1
tk
-1 -0.5
q
0.5
0.5 -0.5 -1
By mapping the interval [, ] linearly onto the interval [1, 1] and scaling gives us the best possible polynomials: pk ( ) = tk where = Formulas cos((k 1)) = cos cos k + sin sin k cos((k + 1)) = cos cos k sin sin k
2 2 +
/tk
1 t ( ck k
) ,
, =
and ck = tk ( ) = tk () , = .
93
imply cos((k + 1)) + cos((k 1)) = 2 cos cos k , and substitution = arccos gives the following recursion for the Tshebyshev polynomials: tk+1 ( ) = 2 tk ( ) tk1 ( ) , t0 ( ) = 1 , t1 ( ) = .
Using this we get the iteration directly for the y k vectors without computing the basic iteration. From equation (10.5) we get y k = pk (E ) e0 + x =
1 t ( E ck k
I ) e0 + x
and from this tk ( E I )e0 = ck (y k x) . This and the recursion for the Tshebyshev polynomials give y k+1 = ck1 tk+1 ( E I )e0 + x +1
= ck1 2( E I ) tk ( E I ) tk1 ( E I ) e0 + x +1 2( E ) ck (y k x) ck1 (y k1 x) + x = ck1 +1

c k 1 y ck+1 k 1
ck = c2 ( E ) y k k+1
+ 1+
2ck ck+1
c k 1 ck+1
2ck Ex ck+1
Equations Ex = M 1 N x = M 1 (M A)x = x M 1 b and ck+1 + 2ck + ck1 2ck = ck+1 + ck1 2ck = 0 imply y k+1 = Denote z k = M 1 (b Ay k ) = M 1 b + Ey k y k , = Then
c k 1 ck+1 2ck ck+1
(E y k + M 1 b)
2ck ck+1
yk
c k 1 y ck+1 k 1
2 2
and
ck k = 2 c k . +1
2ck ck+1 ck+1
= k 1
and the algorithm can be written:
94
EIROLA
Tshebyshev iteration: = (2 )/( ) , = 2/(2 ) y 0 = x0 , y 1 = x1 for k = 1, 2, . . . M zk = b A yk k = 2 tk ()/tk+1() y k+1 = k (y k y k1 + z k ) + y k1
end
Problem 10.11. In fact we dont need the Tshebyshev polynomials here (except as a theoretical tool): show, that the numbers k satisfy: 0 = 2 , k+1 = 1 1 k /(42 ) and k+1 < k .
In the following picture we have solved the problem of example 10.2 in the case n = 50 with Jacobi, GaussSeidel, SOR, SSOR, and with Tshebyshev accelerations of Jacobi (TshJacobi) and SSOR (TshSSOR). It is clear from the picture which is the fastest. The dashed line shows the speed of the conjugate gradient method, which here in the unpreconditioned form does not reveal its power yet. The picture is not completely fair, since workwise the Jacobi and TshJacobi are clearly the cheapest per step, GaussSeidel and SOR have about the same work per step, while the work of SSOR and TshSSOR is about twice that much.
log10 (krk k2 )
Jacobi Gauss-Seidel SOR SSOR TshJacobi
-1 -2 -3
TshSSOR
-4 -5
CG
10
20
30
40
50
Problem 10.12. The symmetric SOR is not directly in the form M xk+1 = N xk + b , so that Tshebyshev iteration cannot be immediately written in the form above.
95
How to modify it to become suitable? (A couple of rows in the place of M z k = b Ay k .) Problem 10.13. Experiment the Tshebyshev acceleration of the Jacobi iteration and the symmetric SOR for problem 10.9.
96
EIROLA
11. The conjugate gradient method In this section we consider iterative solution of an nn linear system Ax = b in the case where A is Hermitian and positive denite. This presentation derives the conjugate gradient method from the Lanczos algorithm, so that the properties of the latter become available immediately. The classical way of deriving the conjugate gradient method is outlined in a sequence of problems. Consider the minimization of the function f : Cn R (11.1) Calculation f (A1 b + d) = (A1 b + d) A(A1 b + d) (A1 b + d) b b (A1 b + d) = b A1 (b + Ad) + d (b + Ad) 2b A1 b d b b d = d Ad b A1 b f (x) = x Ax x b b x .
shows that x = A1 b is a strict minimum of f and the substitution d = A1 (b Ax) gives (11.2) f (x) = (b Ax) A1 (b Ax) b A1 b .
11.1. Minimization in the Krylov subspace. Let us start searching the minimum of f in the Krylov subspace Kj = V (b, Ab, . . . , Aj 1 b) . Recall that the columns of the matrix Qj = q 1 . . . q j = b Ab . . . Aj 1 b R1 form an orthonormal basis of Kj and that AQj = Qj T j + r j eT j ,
where T j is the Hermitian tridiagonal matrix produced by the Lanczos algorithm. We added the tilde: r j here, since r will be needed for other purposes. So, the task is to minimize
f (Qj v ) = v Q j AQj v v Qj b b Qj v 1 1 with respect to v . This has the solution: v j = (Q j AQj ) Qj b = T j Qj b , 1 xj = Qj T j Qj b and the corresponding residual satises: T 1 1 rj = b Axj = b AQj T j Qj b = (I Qj Qj )b r j ej T j Qj b = j b r j , 0Version:
April 26, 2013
97
1 where j = Qj T j ej . In particular, the residuals are orthogonal. Since in the Lanczos j = r j , we get
q j +1 = r j /j = The vectors j satisfy:
1 rj j jb
and
j =
rj . jb
so that these are Aorthogonal: k < j = Since
1 T 1 A j = AQj T ej r j V (q j , q j +1 ) , j ej = Qj ej + ej T T 1 T 1 ej j q j +1] = 0 . k A j = ek T k Qk [q j + ej T
1 T j +1 Q j +1 (xj +1 xj ) = T j +1 Qj +1 Qj +1 T j +1 Qj +1 b T j +1
Q 1 j Qj T j Qj b q j +1
= Q j +1 b =
T j j ej j eT j +1 j
1 T j Qj b 0
Q Q jb jb = (q j +1 b j j b)ej +1 = j ej +1 q j +1 b j jb xj +1 = xj + j j +1 .
and since xj +1 xj R(Qj +1 ) , we get (11.3)
Let us derive a recursion for the vectors j : if T j wj = ej , then T j j ej j eT j +1 j so that wj +1 =

1 dj
0 j w j = = dj ej +1 , 2 T j +1 j ej w j 1
j w j 2 T , where dj = j +1 j ej w j . We get: 1 1 Qj q j +1 dj 1 j j w j j + q j +1 . = 1 dj dj
j +1 = Qj +1w j +1 =
Since q 1 = b/ b , we also see that

T 1 T j b = ej T j Qj b = b w j e1 j j T T w j e1 = b. j +1 b = b w j +1 e1 = b dj dj j
We can write a recursion in nicer form for the vectors j 1 pj :=d2 j + rj j j +1 b j +1 = dj j j b dj dj j jb (11.4) 2 j 2 =j j b j + r j = 2 pj 1 + r j = j pj 1 + rj . d j 1
98
EIROLA
Then (11.3) implies: (11.5) xj +1 = xj + j pj and r j +1 = r j j Apj .
Since j R(Qj ) and r j 1 Qj 1 = 0 , we obtain r j 1 j from which d2 j 1 =

j 1 r j 1 j j 1 b
1 d j 1
r j 1 q j r j 1
4
r j 1 2 = , dj 1j 1 j 1 b = r j 1
2 2
and the coecients entering the pj we get (11.6)
r j 1 j
2 j r j 2 r rj 2 j 1 j . j := 2 = 2 = d j 1 r j 1 2 r j 1 2 jb
This is because A j V (q j , q j +1 ) implies
j r j 1 = j (b AQj 1 v j 1 ) = j b .
The coecient j of equation (11.5) can be computed directly from the condition that f (xj + pj ) is minimized when is j , so that (11.7) j = p rj 2 j rj = . p p j Apj j Apj
Collecting the equations (11.4) (11.7) we get Conjugate Gradient method : x0 = 0 , r 1 = r 0 = b , p1 = 0 for j = 0, 1, 2, . . . j = r j 2 / r j 1 p j = j p j 1 + r j
2
end
j = r j 2 /p j Apj xj +1 = xj + j pj r j +1 = r j j Apj
Usually the conjugate gradient method is derived in the following way:
99
Problem 11.1. The gradient method for minimizing the function f : Rn R is the iteration: xj +1 = xj j f (xj ) , where f (xj +1 ) = min f (xj f (xj )) .
>0
Here we go in the direction opposite to the gradient and minimize a function of one real variable. Which iteration this gives for minimizing the function f (x) = 1 T x Ax xT b ? 2 In the following picture the steps of the gradient method are in black, those of the conjugate gradients in grey.
Problem 11.2. Let xj +1 = xj + j pj be some iteration for minimizing the function 1 T f (x) = 2 x AxxT b , where j minimizes f (xj +pj ) . Assume, that the directions pj are Aorthogonal: pT k Apj = 0 for all k = j . Show that f (xj +1 ) =
v V (p0 ,...,pj )
min
f (x 0 + v ) .
Problem 11.3. Let p0 = r 0 above and pj = r j + j pj 1 , where j is chosen such that pT j 1 Apj = 0 . Show, that the resulting algorithm is the same as the conjugate gradient method . The use of the Lanczos process for deriving the conjugate gradient method gives now the following theorem free, otherwise it would require a lengthy proof. Theorem 11.1. The conjugate gradient method satises: a) the residuals are orthogonal: r T k r j = 0 for all k = j
100
EIROLA
b) vectors pj are Aorthogonal: p k Apj = 0 for all k = j 1 c) xj minimizes (b Ax) A (b Ax) in the subspace Kj d) the iteretion takes at most n steps: rj = 0 , i.e., xj = x for some j n . 11.2. Convergence of the conjugate gradient method. denote: x A = x Ax . Measuring in this energy norm we get at least the following speed: Theorem 11.2. x xj 2 x 1 +1
j
where = n /1 is the condition number of A . Proof. Since xj = pj 1(A)b Kj , where pj 1 is polynomial of degree at most j 1, we get: x xj
2 A
= (A1 b pj 1 (A)b) A(A1 b pj 1(A)b) = (b Apj 1 (A)b) A1 (b Apj 1 (A)b)

A
Hence xj minimizes also x xj x xj

A
= (b Axj ) A1 (b Axj ) = f (xj ) b A1 b.

pPj 1 pPj 1
= min
x p( A ) b
in the subspace Kj , so that

A
= min
A
pPj 1
pPj p(0)=1
x Ap(A)A1 b
A
= min
x Ap(A)x
= min p(A)x
Let U perform the unitary transformation of A into a diagonal matrix = U AU . Then p(A)x
2 A
= x p (A)Ap(A)x = x U p ()p()U x =
k i
|p(k )|2 k |(U x)k |2 max |p(i )|2

i 2
k 2 A
k |(U x)k |2 .
= max |p(i )|
2 A
max |p()|
[1 ,n ]
Again the best polynomial , i.e., the solution of the problem

pPj p(0)=1
min
[1 ,n ]
max
|p()|
is found using the Tshebyshev polynomials with change of variables that maps the interval [1 , n ] onto [1, 1] (reversing the direction, so that zero is mapped to a number bigger than one) pj ( ) =
n +1 tj ( n 1 2 n 1 n +1 tj ( ) n 1
101
Hence the minimum value is:

pPj p(0)=1
min
[1 ,n ]
max
n +1 1 +1 1 |p()| = tj ( ) = tj ( ) , 1 n 1
where = n / next problem > 1 = 1 . By the 1 j 2 tj ( ) > 2 ( + 1) . Since 2 +1 ( + 1)2 +1+2 +1 +1 + = 1= = 1 1 1 ( + 1)( 1) 1 we get + 1 1 tj ( ) <2 1 1 +1
j
which implies the claim.
1 Problem 11.4. Show > 1 = tj ( ) = [ ( + 2 1)j + ( 2 1)j ] . 2 Hint: the tj recursion. Problem 11.5. Show, that if A has p distinct eigenvalues, then the conjugate gradient method stops latest at step p . Hint: the previous proof. Since the full method is a continuous function of A this implies also, that the method is fast if the eigenvalues are in small clusters. 11.3. Preconditioned conjugate gradients. Let M be Hermitian and positive denite some approximation of A such that the systems M y = c are easy to solve. Then M has a Hermitian and positive denite square root S : M = S 2 = S S . Think of solving by the conjugate gradient method the following equation that is equivalent to Ax = b Ax = b , where A = S 1 AS 1 , x = Sx and b = S 1 b . The iteration for this is: x 0 = 0 , r 1 = r 0 = b , p 1 = 0 j = r j j = r j
2 2
/p j A pj
/ r j 1
p j = j p j 1 + r j xj +1 = xj + j pj
Now, set xj = S 1 xj , pk = S 1 pj . Then r j = b Axj = S b S ASxj =S (b Axj ) = S rj

2 2 j = r r j /r r j 1 jS j 1 S 2 r j /p j = r j Apj jS
r j +1 = r j j A pj
xj +1 = xj + j pj
pk = S 2 r j + j pj 1
102
EIROLA
Denoting z j = S 2 r j , i.e., M z j = r j , we get Preconditioned conjugate gradient method : x0 = 0 , r 1 = r 0 = b , p1 = 0 for j = 0, 1, 2, . . . M z j = rj

j = r j z j / r j 1 z j 1 p j = j p j 1 + z j j = r j z j /pj Apj xj +1 = xj + j pj r j +1 = r j j Apj
end
Problem 11.6. Derive the preconditioned conjugate gradient method for the case, where M 1 A is Hermitian and positive denite. Problem 11.7. Prove a version of theorem 11.2 for these preconditioned methods. In the following picture the Jacobi iteration and SSOR are accelerated by the Tshebyshev iteration and by the conjugate gradient method (i.e., the Jacobi and SSOR matrices M are used as preconditioners). The system is that of example 10.2 matrix.
log10 (krk k2 ) 0
TshJacobi
-1 -2 -3
TshSSOR
-4 -5
CGSSOR
CGJacobi
10
20
30
40
50
For Jacobi both accelerators are about as good, while for SSOR the preconditioned conjugate gradient is here clearly better. Explain the reason of this from the spectra of the corresponding iteration matrices in the following gure.
103
Jacobi
-1 1
SSOR
-1 1
When compared with the Tshebyshev iteration the conjugate gradient method has the advantage that no parameters are needed.
104
EIROLA
12. Krylov subspace iterations for nonsymmetric systems In this section we consider approximation of the solution of the general system of equations Ax = b by vectors in the Krylov subspace: xj Kj (A, b) . In section 11 the conjugate gradient method was derived from the Lanczos iteration of a Hermitian matrix. Similarly here we get from the Arnoldi iteration the so called generalized minimal residual method (GMRes), from the nonsymmetric (biorthogonal) Lanczos the biconjugate gradient method (BiCG) and from that, further, the biconjugate gradient squared method (BiCGS) and from the look ahead Lanczos the quasiminimal residual method (QMR). Hence the iterations for eigenvalues and for systems of equations have much in common: Lanczos Arnoldi Biorthogonal Lanczos Look ahead Lanczos Conjugate gradient method (CG) Generalized minimal residual method (GMRes) Biconjugate gradient method (BiCG, BiCGS) Quasiminimum residual method (QMR)
12.1. Generalized minimal residual method (GMRes). The GMRes method nds a vector xj x0 + Kj (A, r0 ) , that minimizes the norm of the residual r j = b Axj . This is obtained using the orthonormal basis of the Krylov subspace Kj (A, r 0 ) produced by the Arnoldi process. Let x0 be an initial approximation for the problem Ax = b . Set up the Arnoldi iteration (see section 9.3) with starting vector q 1 = r 0 / r 0 2 = r0 / , where r 0 = b Ax0 . Then we get: AQj = Qj H j + hj +1,j q j +1 eT j = Qj +1 H j , where H j = Hj 0...0 hj +1,j C(j +1)j .
Since the columns of the matrix Qj = q 1 . . . q j form an orthonormal basis of Kj (A, r 0 ) we look for the vector xj x0 + Kj (A, r0 ) in the form xj = x0 + Qj j . The solution that minimizes the norm of the residual is obtained from: b A(x0 + Qj j )
2
= min b A(x0 + Qj )
= min r 0 Qj +1 H j
2
= min Qj +1( e1 H j )
= min e1 H j
since Qj +1 has orthonormal columns. This is a standard least squares problem that can be solved using the QR decomposition. So, let H j = S j U j , where the
0Version:
April 26, 2013
105
columns of the matrix S j C(j +1)j are orthonormal and U j Cj j is an upper triangular matrix. Then by the Pythagorean theorem: e1 H j
2 2 = S j (S j e1 U j ) + (I S j S j )e1 2 2 2 2
= S j e1 U j
2 2
+ 2 (I S j S j )e1
1 from which j = U j S j e1 and the residual r j = b A(x0 + Qj j ) satises:
rj
= (I S j S j )e1
In the GMRes method we need not form the intermediate vectors j , xj and r j . It suces to run the Arnoldi process, perform the QR decompositions of the matrices H j , and monitor when (I S j S j )e1 2 becomes suciently small, and then compute the corresponding iterate. We get the QR decompositions of the H j matrices inexpensively using the following recursion. If H j 1 is brought by Givens rotations to the upper triangular form: Gj 1,j . . . G1,2 H j 1 = then for the the following we rst get: Gj 1,j . . . G1,2 H j = Gj 1,j . . . G1,2 Now the next rotation is chosen such that Gj,j +1 and we set: U j = to the new vector U j 1
0...0
U j 1
0...0
H j 1
0...0
wj
hj +1,j
U j 1 = 0...0
0...0
hj +1,j
wj .
wj hj +1,j
uj 0
uj . In each step we need to apply the rotations only .
wj hj +1,j
f j 1 Cj +1 . Then j can 0 nally be computed directly from the triangular system U j j = [I 0 ] f j . Dene the vectors f j as follows f 0 = , f j = Gj,j +1 The monitoring of the convergence is simple when the following result is used: Problem 12.1. Show that (f j )j +1 = (I S j S j )e1
2
106
EIROLA
Hence we get1: GMRes algorithm: r0 = bAx0 , = r0 , q 1 = r0 / , Q1 = [q 1 ]
f 0 = e1 , U 0 = [ ] , ready = false , j = 1 while not ready pj = Aq j for k = 1, ..., j , hk,j = q pj = pj hk,j q k k pj , w j = h1:j,j , hj +1,j = pj
2
end
if hj +1,j = 0 then q j +1 = pj /hj +1,j , Qj +1 = [Qj q j +1 ] w j = Gj 1,j . . . G1,2 w j uj w = Gj,j +1 h j j +1,j 0 f j = Gj,j +1f j 1 end
1 xj =x0 + Qj U j [I 0] f j
Uj =
U j 1
0...0
uj
if (f j )j +1 = 0 then
ready = true else j = j + 1
Remark. If hj +1,j = 0 above, then Gj,j +1 = I and (f j )j +1 = 0 . A preconditioned GMRes is obtained from this by simply replacing the multiplication Aq j by M 1 Aq j . The work load of the Arnoldi iteration increases for each step: at step j in addition to the matrixvector multiplication Aq j (or M 1 Aq j ) we have to compute j inner products q k pj of length n and j vector updates pj = pj hk,j q k . The inner products and updates require approximately 4jn oating point operations, i.e., in m steps this work is 2m2 n . The ops required for the QR decompositions are neglible compared to this. Since the work/iteration grows easily too big, the common way is to stop after a xed number m of steps, form the vector xm and restart the iteration with this initial approximation. This is called the GMRes(m) algorithm. On the other hand, since GMRes minimizes the norm of rj , no other Krylov subspace iteration can be faster, when measured in the dimension of the subspace, than the (full) GMRes. The other methods that we study next try to perform as well as GMRes, but using much less arithmetic operations.
1The
rj vector of the Arnoldi iteration is here pj .
107
12.2. Biconjugate gradient method. The biorthogonal Lanczos is much less expensive than the Arnoldi iteration. If we start the process with vector r0 = b Ax0 , it proceeds as (see section 9.2): v 1 = r0 / r 0 2 , w 1 = v 1 , 0 = 0 = 0 j = w j Av j r j = Av j j v j j 1v j 1 sj = A wj j wj j 1w j 1 j = s j rj , j = s j r j /j j . = rj /j , wj +1 = sj /
1/2
v j +1
Writing V j = v 1 . . . v j , W j = w1 . . . w j and 1 1 1 2 2 .. .. . . 2 Tj = , .. . j 1 j 1 j 1 j we get: AV j = V j +1 T j , W jV j = I .
Tj =
Tj 0...0 j
C(j +1)xj
Since the basis v 1 , . . . , v j of the Krylov subspace Kj (A, r0 ) is not orthogonal, the true minimization of the residual can not be done easily here. The biconjugate gradient method chooses the vector xj x0 + Kj (A, r0 ) such that W j (b Axj ) = 0 . Writing xj = x0 + V j j we get: 0 = W j (b A(x0 + V j j )) = r 0 e1 W j V j +1 T j j = r 0 e1 T j j .
1 From this we solve j = r 0 2 T j e1 and get xj . Similarly to the derivation of the conjugate gradient method from the Lanczos iteration (i.e., a long computation) also here we get a recursion directly for residuals rj and the change directions pj . The result is:
108
EIROLA
Biconjugate gradient method (BiCG): r 0 = bAx0 , r 0 = r 0 , 1 = r 0 r 0 , p 1 = p 1 = 0 for j = 0, 1, 2, . . . j = r j r j , j = j /j 1 j = j /p j Apj xj +1 = xj + j pj p j = r j + j p j 1 , p j = r j + j p j 1
end
r j +1 = rj j Apj , r j +1 = r j j A pj
Above r 0 can be any vector satisfying r 0 r 0 = 0 (like the vector w 1 in the biorthogonal Lanczos). The biconjugate gradient method inherits the problems of the biorthogonal Lanczos. First the norms of the residuals behave usually very irregularly. Below is a typical case:
log10 (krk k2 )
-2
-4
-6
20
40
60
80
100
120
Here A is a random 100x100 matrix. The dashed line depicts the GMRes iteration. From the picture we also see that BiCG does not nd the solution in n = 100 steps although theoretically it should. This is due to the loss of biorthogonality caused by roundo errors. Second, there may occur divisions by small numbers (zeros) that lead to instabilities. This can happen in two places: either p j Apj = 0 or r j rj = 0 with rj = 0 . Problem 12.2. Show that p j Apj = 0 means that T j is not invertible. 12.3. Quasiminimal residual method (QMR). The rst possible division by zero in the biconjugate gradient method p j Apj = 0 ( T j is not invertible) can be
109
xed as follows. Instead of the requirement W j (b Axj ) = 0 notice that: b Axj = V j +1 ( r0

2
e1 T j j )
and choose j to be such that it minimizes the norm of r 0 2 e1 T j j . This quasiminimization can be done as in the GMRes method: using the QR decomposition, now even more simply, since T j is a tridiagonal matrix, so that in its QR decomposition T j = S j U j the upper triangular matrix U j has only three diagonals dierent from zero (why?). Direct application of this leads to the following preliminary version of the quasiminimal residual method QMR version 02: r 0 = b Ax0 , v 1 = w1 = r 0 / r 0 , G0,1 = I 1 = w 1 Av 1 r 1 = Av 1 1 v 1 , s1 = A w 1 1w1 1 =
|s 1 r 1 | , 1 = s1 r 1 /1 1 v 2 = r 1 /1 , w2 = s1 /
U1 = G1,2 1 , 0 1
f 1 = G1,2
r0 0
1 x1 = x0 + v 1 U 1 (f 1 )1 for j = 2, 3, . . . j = w j Av j
j =
r j = Av j j v j j 1 v j 1 , sj = A wj j wj j 1 wj 1 0 w j = Gj 1,j Gj 2,j 1 j 1 j
s j r j , j = sj r j /j
uj wj = Gj,j +1 , 0 j xj = x0 + v 1 . . . v j v j +1 = r j /j , w j +1 end
Uj =
1 U j
U j 1
0...0
uj ,
f j = Gj,j +1
f j 1 0
[I 0] f j j = sj /
In this version we need to store all the vectors v 1 , . . . , v j , so that xj could be computed.
2Remark:
here r j is as in the Lanczos algorithm and generally r j = b Axj .
110
EIROLA
1 Problem 12.3. Denote: p1 . . . pj = v 1 . . . v j U j . Derive a recursion for the pj vectors. Show that from these we get: xj = xj 1 + (f j )j pj .
Using this we get the following:

Myopic
QMR:
0 = 0 = 0 , G1,0 = G0,1 = I , f 0 = r 0 for j = 1, 2, . . . j = w r j = Av j j v j j 1 v j 1 , sj = A w j j wj j 1wj 1 j Av j , 0 w j = Gj 1,j Gj 2,j 1 j 1 j uj wj = Gj,j +1 , 0 j f j = Gj,j +1 f j 1 0 pj = [v j (uj )j 1 pj 1 (uj )j 1 pj 2 ]/(uj )j , j = end Here the memory requirement is more reasonable. In step j we need to access only seven vectors from the previous steps v j 1 , v j , w j 1 , wj , pj 1 , pj and xj 1 . From vector f j it suces to remember only the last component. This quasiminimization strategy of the residual also xes the nervous behavior of the norm of the residual. In the following picture we have added (to the previous picture) the results (in black) of this QMR version:
log10 (krk k2 )
r 0 = b Ax0 , v 1 = w1 = r 0 / r0 , p1 = p0 = v 0 = w0 = 0
xj = xj 1 + (f j )j pj
s j r j , j = sj r j /j ,
j v j +1 = r j /j , wj +1 = sj /
-2
-4
-6
20
40
60
80
100
120
Notice that the slowing down of the convergence due to the loss of orthogonality is a problem also in this algorithm.
111
The other divisionbyzero problem of the biconjugate gradient method: r j rj = 0 with rj = 0 is the basic problem of the biorthogonal Lanczos and it can be xed only if we base the method on the look ahead Lanczos. This leads to the genuine QMR method, but we wont present it here. 12.4. Biconjugate gradient squared. Return to the biconjugate gradient method. Let us gure out for what the dual vectors r j and pj are needed: only to compute the numbers j = r and j = j /p j rj j Apj , i.e., in two inner products. It is easy to see, that the vectors of this method satisfy pj = j (A)r0 , r j = j (A )r 0 , pj = j (A) r 0 , r j = j (A ) r 0 ,
where j and j are polynomials of degree j . In particular, we get pj and pj (respectively r j and r j ) using the same polynomial. Let us dene an (indenite) inner product for polynomials
, = r 0 (A) (A)r 0
and the polynomials (t) = t , 1(t) = 1 . Then the biconjugate gradient method for the polynomials can be written as Polynomial biconjugate gradient method: 0 , 0 , 1 = 0 0 = 1 , 1 = for j = 0, 1, 2, . . . j , j , j = j /j 1 j = j = j + j j 1 j , j j = j / end j +1 = j j j
The biconjugate gradient squared method (BiCGS) starts by noticing (notice) that the required inner products can be computed also as follows: j , j = 1, 2 j , j = 1, 2 . j = and j j Let us form the iteration for the polynomials j = 2 j ,
2 j = j ,
j = j j 1 ,
j = j j .
112
EIROLA
Problem 12.4 (Polynomial BiCGS). Show that for these we get the iteration: 0 = 1 , 1 = 1, 0 , 0 = 1 = 0 for j = 0, 1, 2, . . . j = 1, j , j = j /j 1 j = j + j j , j = j / 1, j end j = j + j j + 2 j j 1 j +1 = j j (j + j +1)
j +1 = j j j ,
Returning back to the iteration of vectors we get an interesting method , where the transpose of A is not needed and which satises: if r BiCG = j (A)r 0 , then j 2 BiCGS = j (A) r 0 . In particular, if BiCG converges nicely, then j (A) has small rj norm and BiCGS converges almost twice that fast. Denote: rj = j (A)2 r 0 = j (A)r 0 , pj = j (A)2 r0 = j (A)r 0 ,
q j = j (A)j 1 (A)r0 = j (A)r0 , uj = j (A)j (A)r 0 = j (A)r 0 . Problem 12.5. Show that then we get the following Biconjugate gradient squared method (BiCGS): r 0 = b Ax0 , r 0 = r 0 , 1 = r 0 r 0 , p1 = q 0 = 0 for j = 0, 1, 2, . . . j = r 0 r j , j = j /j 1 uj = r j + j q j , v j = A pj , q j +1 = uj j v j , end In comparison with the BiCG method here the multiplication with A has been replaced by another multiplication with A . The example of the previous pictures looks now as follows. Here the dashed line corresponds to GMRes, the grey to BiCG, and the black to BiCGS. p j = uj + j q j + 2 j p j 1 r j +1 = r j j A (uj + q j +1)
j = j /r 0vj
xj +1 = xj + j (uj + q j +1 )

log10 (krk k2 ) 2
113
0 -2 -4 -6
20
40
60
80
100
120
Let us take another better:
100x100
problem, where BiCG works ne. Then BiCGS is even
log10 (krk k2 )
2 0
QMR
BiCG
-2
BiCGS
-4
GMRes
-6
10
20
30
40
50
12.5. On convergence. This was supposed to ll an entire chapter, but let us just state the following Theorem 12.1. Let A be diagonalizable : A = X X 1 . Then the GMRes residuals satisfy: rj 2 2 (X ) min max |p()| . pP j +1 (A) r0 2
p(0)=1
Problem 12.6. Proof. ... And mention that: Factum: If A = A + E , where the matrix E has rank k , then the corresponding GMRes iterates satisfy: r j +k r j . This is good to keep in mind when looking for preconditioners: if a small change in rank gives a system that is easily solvable then this is sure to give a good preconditioner.
114
EIROLA
13. Preconditioning In the previous chapter we saw that preconditioning may have a drastic eect on the convergence speed of an iterative solver. In this chapter we look at general strategies and ideas to build preconditioners. General preconditioning. Assume we are solving the system of equations (13.1) Ax = b and let K and M be matrices, one of them somehow close to A and such that the systems Ky = c and M z = d are easy to solve. Replace (13.1) by the equation (13.2) M 1 AK 1 y = M 1 b . Then M is called a left and K a right preconditioner. Form the iteration for the partition AK 1 = M N : i.e., M y k+1 = N y k + b = (M AK 1 )y k + b , y k+1 = y k + M 1 (b AK 1 y k ) .
This converges, if the spectral radius of the matrix I M 1 AK 1 is less than one. Notice, that the matrices K and M are not needed, only some routines are applied that perform the operations v K 1 v and v M 1 v . The iteration can then be accelerated with the Tshebyshev iteration, in the Hermitian case with the preconditioned conjugate gradient method, or in the general case with methods like GMRes, QMR, CGS, etc. Generally, the analysis of these iterations show that they are fast in cases where the iteration matrix is of the form where the norm of R is small and the rank of F is small. In the sequel we restrict ourselves to consider only left preconditioning, i.e., we assume, that K = I . The general preconditioning strategies can be devided in the following types: 1) Classical preconditioners: M is chosen as in the Jacobi, Gauss Seidel, SOR, or SSOR iterations. 2) ILU preconditioners (incomplete LU factorization): here M = LU , where L and U are (sparse) lower and upper triangular matrices such that LU A.
0Version:
I M 1 AK 1 = R + F ,
April 26, 2013
115
3) Polynomial preconditioners: here a direct approximation for the inverese of A is looked for in the polynomial form: H = M 1 = p(A) A1 . 4) Direct sparse approximation for the inverse: look for H = M 1 from some nice class such that for example I HA F is as small as possible. 5) The multigrid idea: an approximate inverse H is constructed from a solver of a smaller, but similar problem. 6) Other approaches (not considered here) we get, when we look for matrices somehow close to A and that have certain structure so that the corresponding systems can be easily solved with e.g. circulant or Toeplitz solvers or using fast Fourier transform. If the preconditioner is directly a matrix H = M 1 we talk about explicit preconditioning, otherwise implicit (i.e., when we solve systems M z = d ). 13.1. ILU preconditioners. Incomplete LU factorizations. Consider rst the LU factorization of a band matrix A having the width u of upper band and l of the lower band, in other words, ai,j = 0 , for j > i + u and i > j + l . The LU factorization of such a matrix is also banded (check): A = LU = u l
# # .. # # # # . . # # # ... # ... # # ... #
... # #
The work needed to form this factorization is 2nlu . It is a branch of art, how to get small l + u by permutations. This we however wont study here. In many sparse cases A has nonzero elements only on some diagonals:
l = # . . # . # .
# # # #
# .. #
1 # 1 . . # ... ... 1 # # 1 # # 1
u # . . # . # .
# # #
# # # # # # # ... ... #
A=
116
EIROLA
then, unfortunately, the LU factorization of A is not that sparse but all the grey areas of the following gure generally become nonzero
A=
=LU
The idea of the incomplete LU factorization (ILU) is to x some sparsity structure for L and U and require, that the equation A = LU holds for the corresponding elements. Usually the structure is chosen to be the same as that of A or somewhat more dense. So, we choose a set J {1, . . . , n}2 of index pairs such that (i, j ) ai,j = 0 J . Denote MJ = M Cnn mi,j = 0 , (i, j ) J and search for L, U MJ (lower and upper triangular matrix, respectively) such that (LU )i,j = ai,j A typical choice in the case of the gure above is for all (i, j ) J .
L=
U=
The computation of the incomplete LU factorization is done in principle similarly to that of the full factorization but considering only the index set J . The existence of the factorization (pivot elements are nonzero) is generally dicult to guarantee. Also pivoting techniques become complicated. Some theorems exist for the so called Mmatrices1 Instead of dening the sparsity pattern of L and U a priori, we can set it on the way: When performing the LU-decomposition then after the j :th step we set all elements |Li,j | < and |U j +1,i | < to zero. Often e.g. due to the lack of pivoting the computation of the ILU factorization can become very unstable. As a remedy, some modications have been designed (MILU
1A
is an Mmatrix, if ai,j 0 for all i = j and (A1 )i,j 0 for all i, j .
117
= modied ILU) for example such that we add scalars on the diagonal of U such that the row sums of LU and A become the same. Another way to try to stabilize the computation is to perform the ILU for the matrix A + D , where D is a positive diagonal matrix. When A is Hermitian and positive denite, then the ILU is searched in the form U = L . Then LL is the incomplete Cholesky factorization of A . A drawback of the ILU preconditioning is that it is not very suitable for parallel processing. Problem 13.1. Test the preconditioned conjugate gradient method in the Problem 10.9 using the incomplete Cholesky factorization as a preconditioner. 13.2. Polynomial preconditioning. In cases, where A (or already slightly preconditioned A ) is cheap to apply in a parallel architecture it often pays to try polynomial preconditioning. Let be A = M 0 N 0 some converging decomposition, in other words,
1 (E 0 ) = (M 0 N 0) < 1 .
Then
j =0
2 1 . Ej 0 = I + E 0 + E 0 + = (I E 0 )
This series converges, since in a suitable norm E 0 < 1 , and E j 0 We get: 1 1 1 A1 = [M 0 (I M = (I E 0 )1 M 0 N 0 )] 0 . A simple approximation for A1 we get from the truncated series:
m
E0
Hm =
j =0
1 Ej 0M 0 .
This proposes the preconditioner

m
Hm =
j =0
1 1 1 1 j (I M 0 A ) M 0 = pm ( M 0 A ) M 0
where pm is a polynomial of degree m .

m
Problem 13.2. Show, that above pm (x) =

m j k j =0 k =0
m+1 k +1
(x)k . Hint: show rst that
m+1 k +1
118
EIROLA
Another way to construct polynomial preconditioners is to start with the ideal goal pm (A)A = I . Then pm () = 1 for every eigenvalue of A . Now, if 1 , . . . , m are some approximate eigenvalues of A then it sounds reasonable to choose pm to be the polynomial that satises pm (j )j = 1 , j = 1, . . . , m . For example, if A is Hermitian and positive denite and we start with the conjugate gradient method, then the corresponding Lanczos tridiagonal matrix T j is also found with little extra work. From this we can get approximations for the eigenvalues and from those a polynomial preconditioner, so that we can continue using a preconditioned method. 13.3. Direct sparse approximations of the inverse. Explicit well parallelizable preconditioners can also be looked for in the following way: choose some sparsity structure J and look for the matrix H MJ (see subsection 13.1) for which I HA is the smallest possible. Since
n n F
I HA
2 F
= IA
H 2 F
=
i,j =1
|(I A H )i,j | =
j =1
A h j ej
where hj is the j th row of H , we see that the minimization reduces to n separate problems, each of which has its own sparsity requirement for hj . In the minimization we have the corresponding rows of A to which we perform for example the QR factorization. This seems to be expensive workwise, but it parallelizes completely. 13.4. The multigrid idea. This is to construct an approximation for the inverse from a solver H of a smaller but similar problem. Here we consider only the cases of Example 10.2 and Problem 10.5, i.e., the Poisson equation in one and two space dimensions. The simplest version of the one dimensional equation is uxx (x) = f (x) , For this we get the discretization uxx (ih) =
1 [u(ih h2
x (0, 1) , u(0) = u(1) = 0 . h) + 2u(ih) u(ih + h)]
where h = 1/(n + 1) . The matrix of the corresponding system Au = b , where u = (u(h), u(2h), . . . , u(nh)) , b = (f (h), f (2h), . . . , f (nh)) ,
119
becomes A=
1 (n+1)2
Let n = 2m + 1 be odd and assume, that we can easily solve the corresponding system A u = b (approximately) by u = H b , where the coecient matrix is similar but smaller A Rmm . The idea is now to use this solver as a preconditioner for A . Then we have to transform the ne grid problems of size n = 2m + 1 , to problems on the coarser grid, i.e., of size m and the solutions of the latter back to nvectors. In other words the ne grid functions have to be projected on the coarse grid and functions dened on the latter extended on the ne grid. The latter is done by interpolation. The grids are depicted in the following:
0
2 1 1 2 1 1 2 ... ... ... 1 1 2
Rn n .
x1 x2 x3
^ x 1 ^ x 2 ^ x m
xn
When vj v (jh) = v (xj ) and vj v (j 2h) = v (xj ) , we get using the linear interpolation: 1 v2j +1 = 2 (vj + vj +1 ) , v2j = vj . In the matrix language this is 1 v = Q1 v , where 1 1 2 Q1 =
2
1 2
1 1 2 2 1 2
. .
1
1 2
The projection could be done simply uj = u2j , but a weighted mean turns out to be nicer 1 uj = 4 [u2j 1 + 2u2j + u2j +1 ] ,
1 T Q1 u , which brings more symmetry in the iteration. since then u = 2
Rn m . 1 2
A naive iteration based purely on the multigrid preconditioning would be

1 1 uk+1 = uk + 2 Q1 HQT 1 (b Auk ) ,
where H A (a good symmetric approximation). Here k is the iteration (not component) index. This however does not work, since in the iteration the preconditioner has to be invertible. Here the rank of the matrix Q1 HQT 1 is only m . A working multigrid iteration we get by combining this with a cheap basic iteration on the ne grid. A good choice turns out to be underrelaxed Jacobi:
120
EIROLA
1 diag(A)1 . In order to get the full iteration to become based on a symmetric H=2 preconditioner, we perform the Jacobi twice, before and after the coarse grid step. 1 Denoting Q = Q we nally get: 2 1
Multigrid iteration:
uk+1 = w k + H (b Awk )
w k = v k + Q HQT (b Av k ) coarse grid step Jacobi step
v k = uk + H (b Auk )
Jacobi step
This, in fact, is anly a 2-grid iteration, but if we use the same iteration to build the solver H we recursively get the genuine multigrid iteration. Problem 13.3. Show, that the previous iteration can be written uk+1 = uk + H (b Auk ) , where
H = 2H + C HAH CAH HAC + HACAH and C = Q HQT . Hence the preconditioner is symmetric. Show: if (I HA) < 1 and if H is positive denite, then also H is positive denite. Problem 13.4. When the conditions of the previous problem are valid, this works also as a preconditioner for the conjugate gradient method. How do you write the preconditioning step?
In the following gure the one dimensional Poisson equation has been solved with grid sizes h = 0.1, 0.05, 0.025, 0.0125 and 0.00625 , i.e., for n values 9, 19, 39, 79 and 1 159 , darker ones corresponding to smaller h . Here H = A . The direct multigrid iteration is drawn in dots and the preconditioned conjugate gradient method in solid lines.. The remarkable thing here is that the speed does not seem to depend on h . How is this with the classic iterations?
121
log10 (krk k2 ) 0
-2 -4 -6 -8 -10 -12 -14 0 5 10 15 20
For the two dimensional Poisson equation uxx (x, y ) uyy (x, y ) = f (x, y ) , (x, y ) (0, 1) (0, 1) we get respectively the discretization: uxx (ih, jh) = 1 [u(ih h, jh) + 2u(ih, jh) u(ih + h, jh)] h2 1 uyy (ih, jh) = 2 [u(ih, jh h) + 2u(ih, jh) u(ih, jh + h)] , h
and the corresponding matrix

4 1 1 4 ... ... ... 1 1 4 ... 1 1 1
A=
1 (n+1)2
1 1
...
4 1 1 4 ... ... ...
1 ... ... . . . . . . 1 . . . ... ... 4 1 1 1 4
...
2 2 Rn n .
Let again n = 2m + 1 and A Rm grids are depicted in the following
2 m2
be the corresponding smaller matrix. The
122
EIROLA
n+1 1 2 3
2n n
Set vi+nj v (ih, jh) = v (xi , yj ) and vi+mj v (i2h, j 2h) = v (xi , yj ) . If Q1 denotes the interpolation matrix of the 1-dim case, then for the present case the linear interpolation matrix becomes: 1Q
2
1 1 Q 2 1
1 1Q 2 Q1 Q2 =
Q1 1 Q 1Q 2 1 2 1 . . .
1 Q 2 1
= Q1 Q1 Rn2 m2 . 1 Q 2 1
Q1 1 Q 2 1
Problem 13.5. Show this2.
1 QT v and for Q we take Q = 2 Q2 . The projection becomes respectively v = 1 4 2 Otherwise we proceed similarly as before and the corresponding gure, again with grid sizes h = 0.1, 0.05, 0.025, 0.0125 and 0.00625 (in the last case A is a 25 281 25 281 matrix, but solving this on a PC is no problem) is the following. The speed is not much worse than in the one dimensional case and it is essentially independent on h .
we used the Kronecker product: If A Rmn and B Rpq , then A B Rmpnq : (A B )(i1)p+k,(j 1)q+l = ai,j bk,l .
2Above
123
log10 (krk k2 )
-2 -4 -6 -8 -10 -12 0 5 10 15 20
124
EIROLA
References
1. A. M. Bruaset, A survey of preconditioned iterative methods, Longman, London, 1995. 2. G. Golub and C. Van Loan, Matrix computations, John Hopkins Univerity Press, Baltimore, 1989. 3. G. Golub and G. Meurant, R esolution num erique des grands syst` emes lin eaires, Editions Eyrolles, Paris, 1983. 4. R.A. Horn and C.R. Johnson, Matrix analysis, Cambridge University Press, New York, 1985. 5. O. Nevanlinna, Convergence of iterations for linear equations, Birkh auser, Basel, 1993. 6. J. H. Wilkinson, The algebraic eigenvalue problem, Clarendon Press, Oxford, 1965. 7. D. Young, Iterative solution of large linear systems, Academic Press, New York, 1971.

Matrix Computations

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Matrix Computations

Enviado por

Direitos autorais:

Formatos disponíveis

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

Mat-1.3651 Lecture notes

Aalto University School of Science timo.eirola@aalto.fi

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

51 51 55 56 60 60 61 62 63 66 68 72 74 80 82 84 89 91 96 96 100 101 104 104

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

April 26, 2013

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

The p norm of a vector x Cn ( 1 p < ) is dened by x

. Further, the Frobenius-norm is dened as

The standard inner product of vectors x, y Cn is denoted by x, y =

The special case p = q = 2 of this is the Cauchy (SchwartzBunyakovsky) inequality: | x, y | x y .

the case p = 1 , q = is included.

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

turns out to be handy.

a1,1 = 0 . Then . . . 0 a1,1 . . . 0 a2,1 . . . 0 a ,1 3 . .. . . . an,1 0 0 ... 1

a1,2 a1,3 a2,2 a2,3 a3,2 a3,3 . . . . . . an,2 an,3

. . . a1,n a1,1 a1,2 a1,3 . . . a1,n . . . a2,n 0 . . . a3,n = 0 . A . . . . . . . . an,n 0

April 26, 2013

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

Now each of the Gauss transforms M k is of the form . . . 0 k tk = k+1 , . . .

uv . Lemma 2.1. If v T u = 1 then I uv T is invertible and (I uv T )1 = I + 1 vT u

T 1 Since eT = I + tk eT k tk = 0 , we now get (I tk ek ) k . Generally: j k = T ej tk = 0 , so that T T L = (I + t1 eT 1 )(I + t2 e2 ) . . . (I + tn1 en1 )

T T T = (I + t1 eT 1 + t2 e2 )(I + t3 e3 ) . . . (I + tn1 en1 )

Assume, that the theorem holds for all A=

At the rst step of the algorithm:

2(|B | + |t||w|T ) + O (2) .

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

(1 + 2)(|B | + |t||w|T ) + O (2)

= 2(n 1)(|A| + |L||U |) + O (2) , i.e., the theorem holds for

f (x + d) f (x) / f (x) . d / x x / f (x) .

Again, if f is dierentiable, then = J

Example 2.2. f (x) = x1 x2 , J = [1 1] , J = 2 2 x2 1 + x2 = . |x1 x2 |

Hence D F (A)() = A1 A1 b and F = A1 x

>> >> >> >> >>

n=12; A=hilb(n); x0=ones(n,1); b=A*x0; cond(A)

ans = 1.6976e+16 >> x=A\b

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

contain some large elements.

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

We obtained the LDL decomposition of a Hermitian matrix:

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

(i) = (iii) : If P is orthogonal, then

(iii) = (iv) is clear.

= x, x x, y y , x x, y x, y + | x, y |2 y , y = 1 | x, y |2 < 1 , 1/ u > 1 , a contradiction. Hence R(P ) N (P ) .

(v) = (i) : Clear from R(I P ) N (P ) .

E F = {0} further implies

Similarly we obtain dim F dim E .

dim E n dim F = n (n dim F ) = dim F .

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

where the multiplication error satises Then E

Since the coecient of E not amplied.

is one, the errors accumulate only additively, they are

, e1 Choosing = | x x 2 ( = x 2 , if x, e1 = 0 ) we get: x, e1 x,e1 | 1 | x, e1 | x 2 , giving x, v = 2 v , v and H x = e1 .

Continuing this way we get:

r1,1 r1,2 0 r2,2 . . . 0 H n . . . H 2H 1A = 0 0 0 . . . . . . 0 0

. . . r1,n . . . r2,n . .. . . . R . . . rn,n = 0 ... 0 . . . ... 0

NUMERICAL LINEAR ALGEBRA; ITERATIVE METHODS

it is better to make it in the form Qk = I + W k V k , k = 1, . . . , n as follows: set: W 1 = w 1 = v2v1 v 1 , V 1 = v 1 . Then Q1 = H 1 = I + W 1 V 1 and

where wk = v2vk Qk1 v k and W k = [W k1 w k ] , V k = [V k1 v k ] .