CHAPTER 3 (4 LECTURES)

NUMERICAL SOLUTION OF SYSTEM OF LINEAR

EQUATIONS

1. Introduction
System of simultaneous linear equations are associated with many problems in engineering and science, as well as with applications to the social
sciences and quantitative study of business and economic problems. These
problems occur in wide variety of disciplines, directly in real world problems
as well as in the solution process for other problems.
The principal objective of this Chapter is to discuss the numerical aspects
of solving linear system of equations having the form

a x + a12 x2 + .........a1n xn = b1

11 1
a21 x1 + a22 x2 + .........a2n xn = b2
(1.1)
................................................

an1 x1 + an2 x2 + .........ann xn = bn .

This is a linear system of n equation in n unknowns x1 , x2 ......xn . This system
can simply be written in the matrix equation form
Ax=b

a11
a21

..
.

a12
a22
..
.

..
.

an1 an2

a1n
x1
b1
x2 b2
a2n

.. .. = ..
. . .
ann

xn

(1.2)

bn

This equations has a unique solution x = A1 b, when the coefficient matrix

A is non-singular. Unless otherwise stated, we shall assume that this is the
case under discussion. If A1 is already available, then x = A1 b provides
a good method of computing the solution x.
If A1 is not available, then in general A1 should not be computed solely
for the purpose of obtaining x. More efficient numerical procedures will
be developed in this chapter. We study broadly two categories Direct and
Iterative methods. We start with direct method to solve the linear system.
2. Gaussian Elimination
Direct methods, which are technique that give a solution in a fixed number
of steps, subject only to round-off errors, are considered in this chapter.
Gaussian elimination is the principal tool in the direct solution of system
1

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(1.2). The method is named after Carl Friedrich Gauss (1777-1855). To solve
larger system of linear equation we use a method of introductory Algebra.
For example, consider
x1 + 2x2 + x3 = 0
2x1 + 2x2 + 3x3 = 3
x1 3x2 = 2.
Eliminating x1 from second and third equations, we obtain
x1 + 2x2 + x3 = 0
2x2 + x3 = 3
x2 + x3 = 2.
Now eliminate x2 from last equation with the help of second equation
x1 + 2x2 + x3 = 0
2x2 + x3 = 3
1
x3 = 2.
2
The last equation gives x3 = 1.
Therefore 2x2 + 1 = 3, = x2 = 1.
x1 + 2(1) + 1 = 0, = x1 = 1.

Formal structure of Gaussian elimination is the following.

We consider a following 3 3 system
a11 x1 + a12 x2 + a13 x3 = b1
a21 x1 + a22 x2 + a23 x3 = b2
a31 x1 + a32 x2 + a33 x3 = b3

(R1 )
(R2 )
(R3 ).

Let a11 6= 0 and eliminate x1 from R2 and R3 .

a21
a31
Define multipliers m21 =
and m31 =
.
a11
a11
We write each entry in R2 as R2 m21 R1 and in R3 as R3 m31 R1 .
We obtain a 2 2 system
(2)

(2)

(2)

(R2 )

(2)

(2)

(2)

(R3 ).

a22 x2 + a23 x3 = b2

a32 x2 + a33 x3 = b3
Here
(2)

aij = aij mi1 a1j , i, j = 2, 3

(2)

bi

= bi mi1 b1 , i = 2, 3.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(2)

Let a22 6= 0 and eliminate x2 from R3 .

(2)
a33
Define multipliers m32 = (2)
. Subtract m32 times R2 from R3 . This yields
a22
(3)

(3)

a33 x3 = b3

(R3 ).

Here
(3)

(2)

(2)

(3)

(2)

(2)

a33 = a32 m32 a23 ,

b3 = b3 m32 b2 .
Using back substitution, we can find the values
(3)

x3 =
(2)

x2 =

b3

(3)

a33

(2)

b3 a23 x3
(2)

a22
b1 a12 x2 a13 x3
x1 =
.
a11
Note that every step of the elimination procedure can be obtained through
elementary row operations on the Augmented matrix (A | b).
Algorithm [Gauss Elimination]
(1) Start
(2) Declare the variables and read the order of the matrix n.
(3) Input the coefficients of the linear equation with right side as:
Do for i = 1 to n
Do for j = 1 to n + 1
Read a[i][j] End for j
End for i
(4) Do for k = 1 to n 1
Do for i = k + 1 to n
Do for j = k + 1 to n + 1
a[i][j] = a[i][j] a[i][k]/a[k][k] a[k][j] End for j
End for i
End for k
(5) Compute x[n] = a[n][n + 1]/a[n][n]
(6) Do for i = n 1 to 1
sum = 0
Do for j = i + 1 to n
sum = sum +a[i][j] x[j] End for j
x[i] = 1/a[i][i] (a[i][n + 1] sum)
End for i
(7) Display the result x[i]
(8) Stop

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Partial Pivoting: In the elimination process, it is assumed that pivot element aii 6= 0, i = 1, 2, . . . , n. If at any stage of elimination, one of the pivot
becomes small (or zero) then we bring other element as pivot by interchanging rows.
Remark 2.1. Unique Solution, No Solution, or Infinite Solutions.
Here are some tips that will allow us to determine what type of solutions
we have based on either the reduced echelon form.
1. If we have a leading one in every column, then we will have a unique
solution.
2. If we have a row of zeros equal to a non-zero number in right side, then
the system has no solution.
3. If we dont have a leading one in every column in a homogeneous system,
i.e. a system where all the equations equal zero, or a row of zeros, then
system have infinite solutions.
Example 1. Solve the system of equations. This system has solution x1 =
2.6, x2 = 3.8, x3 = 5.0.
6x1 + 2x2 + 2x3 = 2
1
2
2x1 + x2 + x3 = 1
3
3
x1 + 2x2 x3 = 0.
Sol. Let us use a floating-point representation with 4digits and all operations will be rounded. Augmented matrix is given by

6.000 2.000 2.000 2.000

2.000 1.667 0.3333 1.000
1.000 2.000 1.000
0.0
1
2
= 0.3333 and m31 = = 0.1667.
6
6
(2)
= a21 m21 a11 , a22 = a22 m21 a12 etc.

6.000
2.000
2.000
2.000
0.0 0.0001000 0.3333 1.667
0.0
1.667
1.333 0.3334

Multipliers are m21 =

(2)

a21

1.667
Multiplier is m32 =
= 16670
0.0001

6.000
2.000
2.000
2.000
0.0 0.0001000 0.3333 1.667
0.0
0.0
5555
27790
Using back substitution, we obtain
x3 = 5.003
x2 = 0.0

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

x1 = 1.335.
We observe that computed solution is not compatible with the exact solution.
The difficulty is in a22 . This coefficient is very small (almost zero). This
means that the coefficient in this position had essentially infinite relative error and this was carried through into computation involving this coefficient.
To avoid this, we interchange second and third rows and then continue the
elimination.
In this case (after interchanging) multipliers is m32 = 0.00005999.

6.000 2.000
2.000
2.000
0.0 1.667 1.337 0.3334
0.0
0.0 0.3332 1.667
Using back substitution, we obtain
x3 = 5.003
x2 = 3.801
x1 = 2.602.
We see that after partial pivoting, we get the desired solution.
Complete Pivoting: In the first stage of elimination, we search the largest
element in magnitude from the entire matrix and bring it at the position of
first pivot. We repeat the same process at every step of elimination. This
process require interchange of both rows and columns.
Scaled Partial Pivoting: In this approach, the algorithm selects as the
pivot element the entry that has the largest relative entries in its rows.
At the beginning, a scale factor must be computed for each equation in the
system. We define
si = max |aij | (1 i n)
1jn

These numbers are recored in the scaled vector s = [s1 , s2 , , sn ]. Note

that the scale vector does not change throughout the procedure. In starting
the forward elimination process, we do not arbitrarily use the first equation
ai,1
as the pivot equation. Instead, we use the equation for which the ratio
si
is greatest. We repeat the process by taking same scaling factors.
Example 2. Solve the system
3x1 13x2 + 9x3 + 3x4
6x1 + 4x2 + x3 18x4
6x1 2x2 + 2x3 + 4x4
12x1 8x2 + 6x3 + 10x4

=
=
=
=

19
34
16
26

by hand using scaled partial pivoting. Justify all row interchanges and write
out the transformed matrix after you finish working on each column.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Sol. The augmented matrix is

3 13
6
4

6
2
12 8

9
3
19
1 18 34

2
4
16
6 10
26.

and the scale factors are s1 = 13, s2 = 18, s3 = 6, & s4 = 12. We need to pick
the largest (3/13, 6/18, 6/6, 12/12), which is the third entry, and interchange
row 1 and row 3 and interchange s1 and s3 to get

6
2 2
4
16
6
4
1 18 34

3 13 9
3
19
12 8 6 10
26.
with s1 = 6, s2 = 18, s3 = 13, s4 = 12. Performing R2 (6/6)R1 R2,
R3 (3/6)R1 R3 , and R4 (12/6)R1 R4 , we obtain

6 2 2
4
16
0
2
3 14 18

0 12 8
1
27
0 4 2
2
6
Comparing (a22 /s2 = 2/18, a32 /s3 = 12/13, a42 /s4 = 4/12), the largest is
the third entry so we need to interchange row 2 and row 3 and interchange
s2 and s3 to get

6 2 2
4
16
0 12 8
1
27

0
2
3 14 18
0 4 2
2
6
with s1 = 6, s2 = 13, s3 = 18, s4 = 12. Performing R3 (2/12)R2 R3 and
R4 (4/12)R2 R4, we get

6 2
2
4
16
0 12
8
1
27

0
0
13/3 83/6 45/2
0
0
2/3
5/3
3
Comparing (a33 /s3 = (13/3)/18, a43 /s4 = (2/3)/12), the largest is the first
entry so we do not interchange rows. Performing R4 (2/13)R3 R4 , we
get the final reduced matrix

6 2
2
4
16
0 12
8
1
27

0
0
13/3 83/6 45/2
0
0
0
6/13 6/13
Backward substitution gives x1 = 3, x2 = 1, x3 = 2, x4 = 1.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Example 3. Solve this system of linear equations:

0.0001x + y = 1
x+y =2
using no pivoting, partial pivoting, and scaled partial pivoting. Carry at
most five significant digits of precision (rounding) to see how finite precision
computations and roundoff errors can affect the calculations.
Sol. By direct substitution, it is easy to verify that the true solution is
x = 1.0001 and y = 0.99990 to five significant digits.
For no pivoting, the first equation in the original system is the pivot equation, and the multiplier is 1/0.0001 = 10000. The new system of equations
is
0.0001x + y = 1
9999y = 9998
We obtain y = 9998/9999 0.99990 and x = 1. Notice that we have lost
the last significant digit in the correct value of x.
We repeat the solution process using partial pivoting for the original system.
We see that the second entry is larger, so the second equation is used as the
pivot equation. We can interchange the two equations, obtaining
x+y =2
0.0001x + y = 1
which gives y = 0.99980/0.99990 0.99990 and x = 2 y = 2 0.99990 =
1.0001.
Both computed values of x and y are correct to five significant digits.
We repeat the solution process using scaled partial pivoting for the original
system. Since the scaling constants are s = (1, 1) and the ratios for determining the pivot equation are (0.0001/1, 1/1), the second equation is now
the pivot equation. We do not actually interchange the equations and use
the second equation as the first pivot equation. The rest of the calculations
are as above for partial pivoting. The computed values of x and y are correct
to five significant digits.
Operations count for Gauss elimination We consider the number of
floating point operations (flops) required to solve the system Ax = b.
Gaussian Elimination first uses row operations to transform the problem
into an equivalent problem of the form U x = b, where U is upper triangular.
Then back substitution is used to solve for x. First we look at how many
floating point operations are required to reduce into upper triangular matrix.
First a multiplier is computed for each row. Then in each row the algorithm
performs n multiplies and n adds. This gives a total of (n 1) + (n 1)n
multiplies (counting in the computing of the multiplier in each of the (n 1)
rows) and (n 1)n adds.
In total this is 2n2 n 1 floating point operations to do a single pivot on

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

the n by n system.
Then this has to be done recursively on the lower right subsystem, which
is an (n 1) by (n 1) system. This requires 2(n 1)2 (n 1) 1
operations. Then this has to be done on the next subsystem, requiring
2(n 2)2 (n 2) 1 operations, and so on.
In total, then, we use In total floating point operations, with
In =

n
X

(2k 2 k 1) =

k=1

2
n(n + 1)(4n 1)
n n3 .
6
3

Counts for back substitution: To find xn we just requires one division. Then
to solve for xn1 we requires 3 flops. Similarly, solving for xn2 requires
5 flops. Thus in total back substitution requires Bn total floating point
operations with
Bn =

n
X
(2k 1) = n(n 1) n = n(n 2) n2 .
k=1

The LU Factorization: When we use matrix multiplication, another

meaning can be given to the Gauss elimination. The matrix A can be
factored into the product of the two triangular matrices.
Let AX = b is the system to be solved, A is n n coefficient matrix. The
linear system can be reduced to the upper triangular system U X = g with

u11 u12 u1n

0 u22 u2n

U = ..

..
.
.
.

.
.
0
0 unn
Here uij = aij . Introduce an auxiliary lower triangular matrix L based on
the multipliers mij as following

1
m21

L = m31
..
.
mn,1

0
1
m32
..
.

0
0
1
..
.

mn,n1

0
0

Theorem 2.1. Let A be a non-singular matrix and let L and U be defined

as above. If U is produced without pivoting then
LU = A.
This is called LU factorization of A.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Example 4. Solve the system

4x1 + 3x2 + 2x3 + x4
3x1 + 4x2 + 3x3 + 2x4
2x1 + 3x2 + 4x3 + 3x4
x1 + 2x2 + 3x3 + 4x4

=
=
=
=

1
1
1
1.

Also write the LU decomposition and verify that LU = A.

Sol. We write augmented matrix and

4 3 2
3 4 3

2 3 4
1 2 3

solve the system

1 1
2 1

3 1
4 1

1
1
3
Multipliers are m21 = , m31 = , and m41 = .
4
2
4
Replace R2 with R2 m21 R1 , R3 with R3 m31 R1 and R4 with R4 m41 R1 .

4 3
2
1
1
0 7/4 3/2 5/4
1/4

0 3/2 3
5/2 3/2
0 5/4 5/2 15/4 5/4
5
6
Multipliers are m32 = and m42 = .
7
7
Replace R3 with R3 m32 R2 and R4 with R4 m42 R2 , we obtain

4 3
2
1
1
0 7/4 3/2 5/4
1/4

0 0 12/7 10/7 12/7

0 0 10/7 20/7 10/7
Multiplier is m43 =

5
and we replace
6

4 3
2
0 7/4 3/2

0 0 12/7
0 0
0

R4 with R4 m43 R3 .

1
1
5/4
1/4

10/7 12/7
5/3
0

Using back substitution successively for x4 ,

x3 = 1, x2 = 1, x1 = 0.
Here

4 3
2
0 7/4 3/2
U =
0 0 12/7
0 0
0

x3 , x2 , x1 , we obtain x4 = 0,

1
5/4

10/7
5/3

10

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

1
0
0
3/4 1
0
L=
1/2 6/7 1
1/4 5/7 5/6
It can be verified that LU = A.

0
0

0
1

3. Iterative Method
The linear system Ax = b may have a large order. For such systems Gauss
elimination is often too expensive in either computation time or computer
memory requirements or both.
In an iterative method, a sequence of progressively iterates is produced to
approximate the solution.
Jacobi and Gauss-Seidel Method: We start with an example and let us
consider a system of equations
9x1 + x2 + x3 = 10
2x1 + 10x2 + 3x3 = 19
3x1 + 4x2 + 11x3 = 0.
One class of iterative method for solving this system as follows.
We write
1
x1 = (10 x2 x3 )
9
1
x2 = (19 2x1 3x3 )
10
1
x3 = (0 3x1 4x2 ).
11
(0) (0) (0)
(0)
Let x = [x1 x2 x3 ] be an initial approximation of solution x. Then
define an iteration of sequence
1
(k+1)
(k)
(k)
x1
= (10 x2 x3 )
9
1
(k+1)
(k)
(k)
x2
= (19 2x1 3x3 )
10
1
(k+1)
(k)
(k)
x3
= (0 3x1 4x2 ), k = 0, 1, 2, . . . .
11
This is called Jacobi or method of simultaneous replacements. The method
is named after German mathematician Carl Gustav Jacob Jacobi.
We start with [0 0 0] and obtain
(1)

(1)

(1)

x1 = 1.1111, x2 = 1.900, x3 = 0.0.

(2)

(2)

(2)

x1 = 0.9000, x2 = 1.6778, x3 = 0.9939

etc.
An another approach to solve the same system will be following.
1
(k+1)
(k)
(k)
x1
= (10 x2 x3 )
9

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(k+1)

x2

11

1
(k+1)
(k)
(19 2x1
3x3 )
10

1
(k+1)
(k+1)
(0 3x1
4x2
), k = 0, 1, 2, . . . .
11
This method is called Gauss-Seidel or method of successive replacements.
It is named after the German mathematicians Carl Friedrich Gauss and
Philipp Ludwig von Seidel. Starting with [0 0 0], we obtain
(k+1)

x3

(1)

(1)

(1)

(2)

(2)

(2)

x1 = 1.1111, x2 = 1.6778, x3 = 0.9131.

x1 = 1.0262, x2 = 1.9687, x3 = 0.9588.
General approach: We rewrite the system Ax = b as
(k+1)

a11 x1
(k+1)

(k)

= a12 x2 + b1
(k+1)

a21 x1

+ a22 x2

(k)

= a23 x3 + b2

..................................
(k+1)

an1 x1

(k+1)

+ an2 x2

+ + ann x(k+1)
= bn
n

or (D + L)x(k+1) = U x(k) + b
where D, L and U are diagonal, strictly lower triangle and upper triangle
matrices, respectively.
x(k+1) = (D + L)1 U x(k) + (D + L)1 b
x(k+1) = T x(k) + B,
1

Here T = (D + L)

k = 0, 1, 2,

U and this matrix is called iteration matrix.

Algorithm[Gauss-Seidel]
(1) Input matrix A = [aij ], b, XO = x(0) , tolerance TOL, maximum
number of iterations
(2) Set k = 1
(3) while (k N ) do step 4-7
(4) For i = 1, 2, , n

i1
n
X
X
1
xi =

(aij xj )
(aij XOj ) + bi )
aii
j=1

j=i+1

(5) If ||x XO|| < T OL, then OUTPUT (x1 , x2 , , xn )

STOP
(6) k = k + 1
(7) For i = 1, 2, , n
Set XOi = xi
(8) OUTPUT (x1 , x2 , , xn )
STOP.

12

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

3.1. Convergence analysis of iterative methods.

Definition 3.1 (Eigenvalue and eigenvector). Let A be a square matrix then
number is called an eigenvalue of A if there exists a nonzero vector x such
that Ax = x. Here x is called the corresponding eigenvector.
Definition 3.2 (Characteristic polynomial). Characteristic polynomial is
defined as
P () = det(I A).
is an eigenvalue of matrix A if and only if is a root of the characteristic
polynomial, i.e., P () = 0.
Definition 3.3 (Spectrum and spectral radius). The set of all eigenvalues
of matrix A is called the spectrum of A and is denoted by (A).
The spectrum radius of A is defined as
(A) = max{ ||, (A) }.
Theorem 3.4 (Necessary and sufficient condition). A necessary and sufficient condition for the convergence of an iterative method is that the eigenvalue of iteration matrix T satisfy the inequality
(T ) < 1.
Proof. Let
The sequence of vector

x(k)

(T ) < 1.
by iterative method (Gauss-Seidel) are given by
x(1) = T x(0) + B.

x(2) = T x(1) + B = T (T x(0) + B) + B = T 2 x(0) + (T + I)B.

........................
(k)
k (0)
x = T x + (T k1 + T k2 + ... + T + I)B
Since (T ) < 1, this implies
lim T k x(0) = 0

Therefore

lim x(k) = (I T )1 B

as k

Therefore, x(k) converges to unique solution x = T x + B.

Conversely, assume that the sequence x(k) converges to x. Now
x x(k) = T x + B T x(k1) B = T (x x(k1) )
= T 2 (x x(k2) )
= T k (x x(0 ).
Let z = x x(0) ) then
lim T k z = lim (x x(k) ) = x lim x(k) = x x = 0.

= (T ) < 1.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

13

Theorem 3.5. If A is strictly diagonally dominant in Ax = b, then GaussSeidel iteration always converges for any initial starting vector.
Proof. We assume that A is strictly diagonally dominant, hence aii 6= 0
and
n
X
|aii | >
|aij |, i = 1, 2, , n
j=1,j6=i

Gauss-Seidel iterations are given by

x(k) = (D + L)1 U x(k1) + (D + L)1
x(k) = T x(k1) + B.
Method is convergent iff (T ) < 1, where T = (D + L)1 U .
Let be an eigenvalue of matrix A and x be an eigenvector then
T x = x
(D + L)1 U x = x
U x = (D + L)x

i
X
naij = [
aij xj ],

X
j=i+1

i = 1, 2, . . . , n

j=1

naij = aii xi +

j=i+1

aij xj

j=1

aii xi =

i1
X

aij xj

j=1

|aii xi | ||

i1
X

i1
X

n
X

|aij | |xj | + ||

j=1

aij xj

j=i+1
n
X

|aij | |xj |

j=i+1

Since x is an eigenvector, x 6= 0, so we can take norm ||x|| = 1.

Hence

i1
n
X
X
|| |aii |
|aij |
|aij |
j=1

Pn

= ||

j=i+1 |aij |
Pi1
|aii | j=1
|aij |

which implies spectral radius (T ) < 1.

This implies Gauss-Seidel is convergent.

j=i+1

Pn

j=i+1
Pn

|aij |

j=i+1 |aij |

=1

14

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Example 5. Given the matrix

1 2 2
A = 1 1 1
2 2 1
Decide whether Gauss-Seidel converge to the solution of Ax = b.
Sol. The iteration matrix of the Gauss-Seidel method is
T

= (D + L)1 U

1
1 0 0
0
= 1 1 0 0
2 2 1
0

1
0 0
0
= 1 1 0 0
0 2 1
0

0 2 2
= 0 2 3
0 0 2

0 2 2
= 0 2 3
0 0 2

2 2
0 1
0 0

2 2
0 1
0 0

The eigenvalues of iteration matrix T are = 0, 2, 2 and therefore spectral

radius > 1. The iteration diverges.
4. Power method for approximating eigenvalues
The eigenvalues of an n n of matrix A are obtained by solving its
characteristic equation
det(a I) = 0
n + cn1 n1 cn2 n2 + + c0 = 0.
For large values of n, the polynomial equations like this one are difficult,
time-consuming to solve and sensitive to rounding errors. In this section we
look at an alternative method for approximating eigenvalues. The method
can be used only to find the eigenvalue of A that is largest in absolute value.
We call this eigenvalue the dominant eigenvalue of A.
Definition 4.1 (Dominant Eigenvalue and Dominant Eigenvector). Let 1 ,
2 , , and n be the eigenvalues of an n n matrix A. 1 is called the
dominant eigenvalue of A if
|1 | > |i |,

i = 2, 3, . . . , n.

The eigenvectors corresponding to 1 are called dominant eigenvectors of A.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

15

4.1. The Power Method. The power method for approximating eigenvalues is iterative. First we assume that the matrix A has a dominant
eigenvalue with corresponding dominant eigenvectors. Then we choose an
initial approximation x0 of one of the dominant eigenvectors of A. This
initial approximation must be a nonzero vector in R. Finally we form the
sequence given by
x1 = Ax0
x2 = Ax1 = A2 x0
..............
xk = Axk1 = A(Ak1 x0 ) = Ak x0 .
For large powers of k, and by properly scaling this sequence, we will see
that we obtain a good approximation of the dominant eigenvector of A.
This procedure is illustrated in the following example.
Example 6. Complete six iterations of the power method to approximate a
dominant eigenvector of


2 12
A=
.
1 5
Sol. We begin with an initial non-zero approximation of
 
1
x0 =
1
We then obtain the following approximations by power method

  



2 12 1
10
1.00
y1 = Ax0 =
=
= 10
= 10x1
1 5 1
4
0.40


 



2 12 1.00
2.8
1.000
y2 = Ax1 =
=
= 2.8
= 2.8x2
1 5 0.40
1
0.357


 



2 12 1.000
2.284
1.000
y3 = Ax2 =
=
= 2.284
1 5 0.357
0.785
0.3436
After several iterations, we note that the approximations appears to be approaching the dominant eigenvalue = 2.
Theorem 4.2. If A is an n n diagonalizable matrix with a dominant
eigenvalue, then there exists a nonzero vector x0 such that the sequence of
vectors given by
Ax0 , A2 x0 , A3 x0 , . . . , Ak x0 , . . .
approaches a multiple of the dominant eigenvector of A.
Proof. Let A is diagonalizable, which implies that it has n linearly
independent eigenvectors x1 , x2 , . . . , xn with corresponding eigenvalues of
1 , 2 , , n .
We assume that these eigenvalues are ordered so that 1 is the dominant
eigenvalue (with a corresponding eigenvector of x1 ).

16

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Because the n eigenvectors x1 , x2 , . . . , xn are linearly independent, they must

form a basis for Rn .
For the initial approximation x0 , we choose a nonzero vector such that the
linear combination
x0 = c1 x1 + c2 x2 + + cn xn
has nonzero leading coefficients. (If c1 = 0, the power method may not
converge, and a different x0 must be used as the initial approximation.
Now, operating both sides of this equation by A produces
Ax0 = Ac1 x1 + Ac2 x2 + + Acn xn
Ax0 = c1 (Ax1 ) + c2 (Ax2 ) + + cn (Axn )
Ax0 = c1 (1 x1 ) + c2 (2 x2 ) + + cn (n 2xn )
As i are eigenvalues, so Axi = i xi .
Repeated multiplication of both sides of this equation by A produces
Ak x0 = c1 (k1 x1 ) + c2 (k2 x2 ) + + cn (kn xn )
which implies that
"
k

A x0 =

k1


c1 x1 + c2

2
1

k


x2 + + cn

n
1

k
xn

Now, from our original assumption that 1 is larger in absolute value than
the other eigenvalues it follows that each of the fractions
n
2 3
, , ,
< 1.
1 1
1
Therefore each of the factors
 k
 k  k
3
n
2
,
, ,
1
1
1
must approach 0 as k approaches infinity. This implies that the approximation
Ak x0 k1 c1 x1 , c1 6= 0
improves as k increases. Since x1 is a dominant eigenvector, it follows that
any scalar multiple of x1 is also a dominant eigenvector. Thus we have
shown that Ak x0 approaches a multiple of the dominant eigenvector of A.
Algorithm
(1) Start
(2) Define matrix A and initial guess x
(3) Calculate y = Ax
(4) Find the largest element in magnitude of matrix y and assign it to
K.
(5) Calculate fresh value x = (1/K) y
(6) If [K(n) K(n 1)] > error, goto step 3.
(7) Stop

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

17

Example 7. Calculate seven iterations of the power method with scaling to

approximate a dominant eigenvector of the matrix

1 2 0
2 1 2
1 3 1
Sol. Using x0 = [1, 1, 1]T as initial approximation, we obtain

1 2 0
1
3
y1 = Ax0 = 2 1 2 1 = 1
1 3 1
1
5
and by scaling we obtain the approximation

3
0.60
x1 = 1/5 1 = 0.20
5
1.00
Similarly we get

1.00
0.45
y2 = Ax1 = 1.00 = 2.20 0.45 = 2.20x2
2.20
1.00

1.35
0.48
y3 = Ax2 = 1.55 = 2.8 0.55 = 2.8x3
2.8
1.00

0.51
y4 = Ax3 = 3.1 0.51
1.00
etc.
After several iterations, we observe that dominant eigenvector is

0.50
x = 0.50
1.00
Scaling factors are approaching to dominant eigenvalue = 3.
Remark 4.1. The power method is useful to compute the eigenvalue but it
gives only dominant eigenvalue. To find other eigenvalue we use properties
of matrix such as sum of all eigenvalue is equal to the trace of matrix. Also
if is an eigenvalue of A then 1 is the eigenvalue of A1 . Hence the
smallest eigenvalue of A is the dominant eigenvalue of A1 .
Remark 4.2. We consider A I then its eigenvalues are (1 , 2
1
1
, . . . ). Now the eigenvalues of (A I)1 are (
,
, . . . ).
1 2
The eigenvalues of the original matrix A that is the closest to corresponds
to the eigenvalue of largest magnitude of the shifted and inverted of matrix
(A I)1 .

18

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

To find the eigenvalue closest to , we apply the Power method to obtain

the eigenvalue of (A I)1 . Then we recover the eigenvalue of the
original problem by = 1/ + . This method is shifted and inverted. We
solve y = (AI)1 x which implies (AI)y = x. We need not to compute
the inverse of the matrix.
Example 8. Find the eigenvalue of matrix nearest to 3

2 1 0
1 2 1
0 1 2
using power method.
Sol. The eigenvalue of matrix A which is nearest to 3 is the smallest eigenvalue in magnitude of A3I. Hence it is the largest eigenvalue of (A3I)1
in magnitude. Now

1 1 0
A 3I = 1 1 1
0 1 1

0 1 1
B = (A 3I)1 = 1 1 1 .
1 1 0
Starting with

1
x0 = 1
1
we obtain

0 1 1
1
0
y1 = Bx0 = 1 1 1 1 = 1 = 1.x1
1 1 0
1
0

1
y2 = Bx1 = 1 = 1.x2
1

2
0.6667
y3 = Bx2 = 3 = 3 1 = 3x3
0.6667
2

1.6667
0.7143
y4 = Bx3 = 2.3334 = 2.3334 1 = 2.3334x4 .
1.6667
0.7143
After six iterations, we obtain the dominant eigenvalue of matrix B and
which is 2.4 and the dominant eigenvector is

0.7143
1 .
0.7143

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

19

1
Now the eigenvalue of matrix A is 3 2.4
= 3 0.42 = 3.42, 2.58. Since 2.58
does not satisfy |A 2.58I| = 0, therefore the correct eigenvalue of matrix
A nearest to 3 is 3.42.
Although the power method worked well in these examples, we must say
something about cases in which the power method may fail. There are basically three such cases :
1. Using the power method when A is not diagonalizable. Recall that A
has n linearly Independent eigenvector if and only if A is diagonalizable. Of
course, it is not easy to tell by just looking at A whether is is diagonalizable.
2. Using the power method when A does not have a dominant eigenvalue or
when the dominant eigenvalue is such that 1 = 2 .
3. If the entries of A contains significant error. Powers Ak will have significant roundoff error in their entires.

Exercises
(1) Using the four-decimal-place computer solve the following system of
equation without and with pivoting
0.729x1 + 0.81x2 + 0.9x3 = 0.6867
x1 + x2 + x3 = 0.8338
1.331x1 + 1.21x2 + 1.1x3 = 1.000
This system has exact solution, rounded to four places x1 = 0.2245,
x2 = 0.2814, x3 = 0.3279.
(2) Solve the following system of equations by Gaussian elimination with
partial and scaled partial pivoting
x1 + 2x2 + x3 = 3
3x1 + 4x2 = 3
2x1 + 10x2 + 4x3 = 10.
(3) Consider the linear system
x1 + 4x2 = 1
4x1 + x2 = 0.
The true solution is x1 = 1/5 and x2 = 4/15. Apply the Jacobi
and Gauss-Seidel methods with x(0) = [0, 0]T to the system and
find out which methods diverges rapidly. Next, interchange the two
equations to write the system as
4x1 + x2 = 0
x1 + 4x2 = 1
and apply both methods with x(0) = [0, 0]T .
Iterate until ||x x(k) || 105 . Which method converges faster?

20

SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(4) Solve the system of equations by Jacobi and Gauss-Seidel method

8x1 + x2 + 2x3 = 1
x1 5x2 + x3 = 16
x1 + x2 4x3 = 7.
(5) Solve this system of equations by Gauss-Seidel, starting with the
initial vector [0,0,0]:
4.63x1 1.21x2 + 3.22x3 = 2.22
3.07x1 + 5.48x2 + 2.11x3 = 3.17
1.26x1 + 3.11x2 + 4.57x3 = 5.11.
(6) Show that Gauss-Seidel method does not converge for the following
system of equations
2x1 + 3x2 + x3 = 1
3x1 + 2x2 + 2x3 = 1
x1 + 2x2 + 2x3 = 1.
(7) Consider the iteration
x

(k+1)

2 1
=b+
1 2

x(k) ,

k0

where is a real constant. For some values of , the iteration

method converges for any choice of initial guess x(0) , and for some
other values of , the method diverges. Find the values of for
which the method converges.
(8) Determine the largest eigenvalue and the corresponding eigenvector
of the matrix

4 1 0
A = 1 20 1
0 1 4
correct to three decimals using the power method.
(9) Find the smallest eigenvalue in magnitude of the matrix

2 1 0
1 2 1
0 1 2
using four iterations of the inverse power method.
Bibliography
[Gerald]

Curtis F. Gerald and Patrick O. Wheatley Applied Numerical

Analysis, 7th edition, Pearson, 2003.
[Atkinson] K. Atkinson and W. Han. Elementary Numerical Analysis,
3rd edition, John Willey and Sons, 2004.