Escolar Documentos
Profissional Documentos
Cultura Documentos
GOAL.
Understand the norm of vectors and matrix
Understand the conditional number of a matrix
Understand, implement and analyze iterative methods
KEY WORDS. Condition number, iterative method, Jacobi method, GaussSeidel method, successive over-relaxation (SOR) method
In the last Chapter, we have seen that Gaussian elimination is the most
general approach to solve a nonsingular linear system
Ax = b.
(1)
We first present the norm of vectors and matrix, because they are going to be
useful in the discussion of stability of the algorithm and in the stopping criteria,
convergence analysis of the iterative methods.
Definition 1.1. (Vector Norm) A vector norm kxk is any mapping from Rn to
R with the following three properties.
1. kxk > 0, if x 6= 0
2. kxk = ||kxk, for any R
3. kx + yk kxk + kyk
for any vector x, y Rn .
Example 1.2. One of the most commonly used vector norms is the Euclidean
norm (or called l2 norm)
n
X
kxk2 = (
x2i )1/2 = xT x,
l2 norm,
i=1
n
X
|xi |,
l1 norm,
i=1
l norm,
It can be checked by Definition 1.1 that the l1 , l2 , l norm defined above are
vector norms. Below we use the l1 norm as an example.
Example 1.3. The l1 norm is a vector norm.
It suffices to check that
Pn
1. kxk1 = i=1 |xi | > 0, if x 6= 0
Pn
Pn
2. kxk = i=1 |xi | = i=1 |xi | = ||kxk, for any R
Pn
Pn
3. kx + yk = i=1 |xi + yi | i=1 (|xi | + |yi |) = kxk + kyk.
Example 1.4. Let x = (1, 1, , 1)1n , then
kxk1 = n,
kxk2 = n
kxk = 1.
Definition 1.5. (Matrix Norm) A matrix norm of a matrix kAk is any mapping
from Rnn to R with the following three properties.
1. kAk > 0, if A 6= 0
2. kAk = ||kAk, for any R
3. kA + Bk kAk + kBk (triangular inequality)
for any matrix A, B Rnn .
We usually prefer matrix norms that are related to a vector norm.
Definition 1.6. (subordinate matrix norm) The subordinate matrix norm based
on a vector norm k k is given by
kAk
=
=
sup{kAxk : x Rn
sup {kAxk}
kxk=1
and kxk = 1}
(2)
kxk=1
kxk=1
3.
kA + Bk = sup {k(A + B)xk} = sup {kAx + Bxk}
kxk=1
kxk=1
kxk=1
= kAk + kBk
Why introduce subordinate matrix norms? Because of some additional properties that they enjoy,
kIk = 1
kAxk kAkkxk
kABk kAkkBk
To derive them,
x
,
kAk1 = max
1jn
|aij |,
kAk = max
n
X
1in
l1 norm
i=1
l2 norm
|aij |,
l norm
j=1
where i are the square root of eigenvalues of AT A, which are called the singular
values of A. max is the largest in absolute value among i .
Example 1.7. Let the matrix A be
1
3
2
4
then
kAk1 = 6,
kAk2 = 5.4650,
kAk = 7.
To get kAk2 , use eig(A*A) in matlab to obtain 12 , , n2 , then pick the
largest one among all i s, max .
The formulas for l1 , l2 and l subordinate matrix norms can be derived by
using Definition 1.6. For example,
kAk2
sup kAxk2
kxk2 =1
y=Qx
sup (yT y) 2 ,
kyk2 =1
max .
(3)
.
kxk
kbk
(4)
(5)
(6)
(7)
To see that the iterative procedure eq.(7) actually is consistent with the original
Ax = b, we let k and presume that the approximate sequence converges
to x, then we have
Qx = (Q A)x + b
which leads to Ax = b. Thus, if the sequence converge, its limit is the solution
to the Ax = b.
To have a method that is efficient, we hope to have the Q satisfying the
following properties for the general iterative procedure (from eq. (7)),
1. Q is easy to invert.
2. The sequence x(k) will converge to x, no matter what the initial guess is.
3. The sequence x(k) converges to x as rapidly as possible.
In the following, we will introduce three iterative methods: Jacobi method,
the Gauss-Seidel method and the successive over-relaxation (SOR) method.
Jacobi method. Lets first write the system of equations Ax = b in its
detailed form
n
X
aij xj = bi , 1 i n.
(8)
j=1
(k)
In the kth iteration, we solve the ith equation for the ith unknown xi , assum(k1)
ing that the other xj comes from the previous iteration xj
, we obtain an
equation that describes the Jacobi method:
i1
X
(k1)
aij xj
(k)
+ aii xi
j=1
or
n
X
(k1)
aij xj
= bi ,
1 i n.
(9)
j=i+1
n
X
(k1)
aij xj
(k)
+ aii xi
j=1,j6=i
= bi ,
1 i n.
(10)
n
X
(bi
(k1)
aij xj
j=1,j6=i
n
X
bi
aii
j=1,j6=i
)/aii
aij (k1)
x
.
aii j
(11)
Here we assume that all diagonal entries are nonzero. If this is not the case, we
can usually rearrange the equation so that it is.
The equation (9) could be written in the following matrix form
0
a12 a1n
a11 0
0
a2n
a22 0
(k1)
(k) a21 0
=b
x
+
x
..
.
.
.
..
.
..
..
..
..
. ..
. ..
.
0
an1
ann
an2
CL = (aij )i>j ,
CU = (aij )i<j ,
2
1 0
1
1 , b = 8 .
A = 1 3
0
1 2
5
Carry out a number of Jacobi iteration, starting with zero initial vector.
Solution: Rewriting the equation, we have
(k)
(k)
(k)
x1
x2
x3
1 (k1)
x
+
2 2
1 (k1)
+
x
3 1
1 (k1)
x
2 2
7
1
2
1 (k1) 8
+
x
3 3
3
5
.
2
(12)
(13)
(21)
After 21 iterations, the actual solution is obtained within some fixed precision.
In the Jacobi method, the matrix Q is taken to be the diagonal part of A,
2 0 0
0 3 0
0 0 2
With this Q, we know that the Jacobi method could also be implemented as
x(k) = Bx(k1) + h
with the Jacobi iterative matrix B and constant vector h are
0 21 0
1/2
B = 13 0 31 , h = 8/3 .
5/2
0 12 0
Gauss-Seidel method. Lets first write the system of equations Ax = b in its
detailed form
n
X
aij xj = bi , 1 i n.
j=1
In the Jacobi method, the equations are solved in order. When solving the ith
(k)
equation, the component xj (1 j < i) can be immediately in their place, and
(k1)
(k)
aij xj
(k)
aii xi
n
X
j=1
(k1)
aij xj
= bi ,
1 i n.
(14)
j=i+1
xi
(bi
i1
X
(k)
aij xj
j=1
n
X
(k1)
aij xj
)/aii
j=i+1
i1
n
X
X
bi
aij (k)
aij (k1)
xj
x
aii j=1 aii
a j
j=i+1 ii
(15)
Here we assume that all diagonal entries are nonzero. If this is not the case,
we can usually rearrange the equation so that it is. The equation (14) could be
8
a11 0
a21 a22
..
..
..
.
.
.
an1
an2
matrix form
0
(k)
x +
..
0
0
..
.
a12
0
..
.
..
.
a1n
a2n
..
.
ann
(k1)
=b
x
(k)
(k)
x1
x2
x3
1 (k1) 1
x
+
2 2
2
1 (k) 1 (k1) 8
x + x3
+
3 1
3
3
1 (k) 5
x .
2 2
2
(16)
(17)
x(2)
(9)
After 9 iterations, the actual solution is obtained within some fixed precision.
In the GS method, the matrix Q is taken to be the lower triangular part of
A,
2
0
0
1 3
0
0
1 2
With this Q, we know that the GS method could also be implemented as
x(k) = Bx(k1) + h
with the GS iterative matrix B and constant vector h are
0 12
0
1/2
1
.
, h = 17/6
B = 0 16
3
1
1
13/12
0 12 6
9
(18)
1 i n.
aij xj = bi ,
j=1
The idea of the SOR method is essentially the same as the GS method, except
(k1)
(k)
that it also use xi
to solve for xi . The algorithm is the following
i1
X
(k)
aij xj +aii (
j=1
n
X
1 (k1)
1 (k)
(k1)
x +(1 )xi
)+
aij xj
= bi ,
w i
w
j=i+1
1 i n. (19)
xi
w(bi
i1
X
(k)
aij xj
j=1
w(
n
X
(k1)
aij xj
(k1)
)/aii + (1 w)xi
j=i+1
i1
X
n
X
aij (k)
aij (k1)
bi
(k1)
xj
xj
) + (1 w)xi
aii j=1 aii
a
ii
j=i+1
(20)
Again we assume that all diagonal entries are nonzero. The equation (19) could
be written in the following matrix form
(1 w1 )a11 a12
a1n
a11 /w 0
0
1
a21
a
/w
0
)a
a2n
0
(1
22
22
w
(k)
x + .
..
..
.
.
.
.
. . ..
..
. . ...
.
..
.
an1
an2
ann /w
(1
x1
(k)
x2
(k)
x3
1 (k1) 1
(k1)
= w( x2
+ ) + (1 w)x1
2
2
1 (k) 1 (k1) 8
(k1)
= w( x1 + x3
+ ) + (1 w)x2
3
3
3
1 (k) 5
(k1)
= w( x2 ) + (1 w)x3
.
2
2
10
(21)
(22)
1
w )ann
(k1)
=b
x
(2)
(7)
After 7 iterations, the actual solution is obtained within some fixed precision.
In the SOR method, the matrix Q is taken to be
2/w 0
0
1 3/w 0
0
1 2/w
With this Q, we know that the SOR(w) method could also be implemented as
x(k) = Bx(k1) + h
with the SOR iterative
1/10
B = 11/300
121/6000
11/20
11/20
0
.
61/600
11/30 , h = 627/200
4103/4000
671/12000 61/600
(23)
with w = 1.1.
Remark 3.4. (Complexity analysis) In each of the iterative step in the methods
described above, the complexity is of order O(n2 ).
Remark 3.5. (Stopping criteria) The stopping criteria in the iterative methods
for solving Ax = b is to make sure that the distance, measure in norms, between
approximations are bounded by some prescribed tolerance,
kx(k) x(k1) k2 < ,
where is the tolerance of the error.
11
(24)
Let the error at the kth iteration be e(k) = x x(k) . Let x mines both sides of
eq.(24), we have
e(k)
x Q1 [(Q A)x(k1) + b]
x (I Q1 A)x(k1) Q1 b
x x(k1) + Q1 Ax(k1) Q1 b
e(k1) + Q1 Ax(k1) Q1 Ax
e(k1) + Q1 A(x(k1) x)
e(k1) Q1 Ae(k1)
(I Q1 A)e(k1) .
(25)
= k(I Q1 A)e(k1) k
k(I Q1 A)kke(k1) k
(26)
As can be seen from eq.(26), if k(I Q1 A)k < 1, the error becomes smaller
and smaller as the iteration goes on, therefore the iterative method converges.
What is more, the smaller the k(I Q1 A)k is, the faster convergence we would
expect. A very classical theorem about the convergence of the iterative method
is the following
Theorem 4.1. (Spectral Radius Theorem) In order that the sequence generated
by eq.(7) to converge, no matter what the starting point x(0) is selected, it is
necessary and sufficient that all eigenvalues of the matrix I Q1 A lies in the
open unit disc, |z| < 1, in the complex plane.
Proof. Let B = I Q1 A, then
e(k) = B k e(0) ,
hence
ke(k) k kB k kke(0) k.
By the spectral radius Theorem,
(B) < 1 lim B k = 0.
k
12
In Example 3.1, we have use Jacobi, GS and SOR method to iteratively solve
it. We have observed that they take 21, 9 and 7 iterations respectively to obtain
solutions within the same tolerance. Actually, this behavior could be predicted
by the eigenvalues of I Q1 A.
Example 4.2. Determine whether the Jacobi, GS and SOR method will converge for the matrix A and b in Example 3.1, no matter what the initial condition
is.
Solution: For the Jacobi method, we can easily compute the eigenvalues of the
relevant matrix I Q1 A (the matrix B in Example 3.1). The steps are
1/2 0
1
det(B I) = det 1/3 1/3 = 3 + = 0.
3
0
1/2
Solving for gives us the three eigenvalues are 0, 0.5774, all of which lies in
the open unit disk. Thus, the Jacobi method converges.
Similarly, for the GS method, the eigenvalues of the relevant matrix I Q1 A
(the B from Example 3.2 eq.(18)) are determined by
11/20
0
= (1/6 )2 + 1 = 0.
1/6 1/3
det(B I) = det 0
36
0
1/12
1/6
Solving for gives us the three eigenvalues are 0, 0, 0.3333. Thus, the GS method
converges.
Similarly, for the SOR method with w = 1.1, the eigenvalues of the relevant
matrix I Q1 A (the B from Example 3.3 eq.(23)) are determined by
1/10 11/20
0
61/600 11/30
det(B I) = det 11/300
121/6000 671/12000 61/600
= 1/1000 + 31/3000 + 31/30002 3 = 0.
Solving for gives us the three eigenvalues are 0.1200, 0.0833, 0.1000. Thus,
the SOR method converges.
Also from the magnitude of those eigenvalues, it is not surprise that the SOR
performs better than GS, then Jacobi, in terms of the efficiency.
13