0 Votos favoráveis0 Votos desfavoráveis

94 visualizações142 páginasNumerical Analysis Complex Matrix

Aug 15, 2015

© © All Rights Reserved

PDF, TXT ou leia online no Scribd

Numerical Analysis Complex Matrix

© All Rights Reserved

94 visualizações

Numerical Analysis Complex Matrix

© All Rights Reserved

- Numerical Methods for Large Eigenvalue Problems
- Python
- 11 Planetary Science Data Dictionary
- E&G – Quaternary Science Journal Vol. 59 No 1- 2
- Introduction to Python Basics
- E&G - Quaternary Science Journal Vol. 60 No 1: Loess in Europe
- Numerical Model Turner
- Numerical Diffusion
- Fire History and Climate Change:A Synthesis of Knowledge
- A Critique of Piketty's "Capital in the Twenty-first Century"
- Longitudinal and Panel Data Analysis and Applications in the Social Sciences
- Introduction to Scientific Computing Using Python
- Python Man
- Kirghiz and Wakhi of Afghanistan(2002) by Shahrani
- Extreme Fire Behavior: A Synthesis of Knowledge for Fire Managers,Vol.1
- The Bible, The Qur'an and Science - Maurice Bucaille - XKP
- Iop - Ng G W - Intelligent Systems; Fusion, Tracking and Control (2003)
- NumericalIsingModel
- Theory and Applications of Numerical Analysis
- Numerical Solution ADR Equations_Hundsdorfer

Você está na página 1de 142

This electronic version is for personal use and may not be duplicated or distributed.

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

A

pp

lie

M

at

he

m

at

ics

Numerical

Matrix Analysis

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

A

pp

lie

M

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

This electronic version is for personal use and may not be duplicated or distributed.

M

at

he

m

at

ics

Numerical

Matrix Analysis

la

nd

A

pp

lie

us

tr

ia

Ilse C. F. Ipsen

So

cie

ty

fo

rI

nd

Raleigh, North Carolina

Philadelphia

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

M

at

10 9 8 7 6 5 4 3 2 1

A

pp

lie

All rights reserved. Printed in the United States of America. No part of this book may be

reproduced, stored, or transmitted in any manner without the written permission of the

publisher. For information, write to the Society for Industrial and Applied Mathematics,

3600 Market Street, 6th Floor, Philadelphia, PA, 19104-2688 USA.

us

tr

ia

la

nd

Trademarked names may be used in this book without the inclusion of a trademark

symbol. These names are used in an editorial context only; no infringement of trademark

is intended.

rI

nd

So

cie

ty

fo

Ipsen, Ilse C. F.

Numerical matrix analysis : linear systems and least squares / Ilse C. F. Ipsen.

p. cm.

Includes index.

ISBN 978-0-898716-76-4

1. Least squares. 2. Linear systems. 3. Matrices. I. Title.

QA275.I67 2009

511'.42--dc22

2009008514

is a registered trademark.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

A

pp

lie

M

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

A

pp

lie

M

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

book

2009/5/27

page vii

i

This electronic version is for personal use and may not be duplicated or distributed.

he

m

at

ics

Contents

Preface

M

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

1

1

4

4

5

6

8

9

12

14

15

16

19

20

22

2.1

Sensitivity and Conditioning . . . . . . . . . . . .

2.2

Absolute and Relative Errors . . . . . . . . . . . .

2.3

Floating Point Arithmetic . . . . . . . . . . . . . .

2.4

Conditioning of Subtraction . . . . . . . . . . . . .

2.5

Vector Norms . . . . . . . . . . . . . . . . . . . .

2.6

Matrix Norms . . . . . . . . . . . . . . . . . . . .

2.7

Conditioning of Matrix Addition and Multiplication

2.8

Conditioning of Matrix Inversion . . . . . . . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

23

23

25

26

27

29

33

37

38

Linear Systems

3.1

The Meaning of Ax = b . . . .

3.2

Conditioning of Linear Systems

3.3

Solution of Triangular Systems

3.4

Stability of Direct Methods . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

43

43

44

51

52

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

nd

la

ia

tr

us

nd

rI

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

cie

ty

fo

.

.

.

.

.

.

.

.

.

.

.

.

.

.

So

.

.

.

.

.

.

.

.

.

.

.

.

.

.

lie

Matrices

1.1

What Is a Matrix? . . . . . . . . . .

1.2

Scalar Matrix Multiplication . . . . .

1.3

Matrix Addition . . . . . . . . . . .

1.4

Inner Product (Dot Product) . . . . .

1.5

Matrix Vector Multiplication . . . .

1.6

Outer Product . . . . . . . . . . . .

1.7

Matrix Multiplication . . . . . . . .

1.8

Transpose and Conjugate Transpose .

1.9

Inner and Outer Products, Again . .

1.10

Symmetric and Hermitian Matrices .

1.11

Inverse . . . . . . . . . . . . . . . .

1.12

Unitary and Orthogonal Matrices . .

1.13

Triangular Matrices . . . . . . . . .

1.14

Diagonal Matrices . . . . . . . . . .

pp

xiii

at

Introduction

ix

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

vii

i

i

i

book

2009/5/27

page viii

i

This electronic version is for personal use and may not be duplicated or distributed.

viii

Contents

LU Factorization . . . . . . . . . . . . . . . .

Cholesky Factorization . . . . . . . . . . . .

QR Factorization . . . . . . . . . . . . . . . .

QR Factorization of Tall and Skinny Matrices

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

58

63

68

73

4.1

Extreme Singular Values . . . . . . . . . . . . . . . . . . . .

4.2

Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3

Singular Vectors . . . . . . . . . . . . . . . . . . . . . . . .

77

79

81

86

91

5.1

Solutions of Least Squares Problems . . . . . . . . . . . . . 91

5.2

Conditioning of Least Squares Problems . . . . . . . . . . . 95

5.3

Computation of Full Rank Least Squares Problems . . . . . . 101

Subspaces

6.1

Spaces of Matrix Products . . . . . . .

6.2

Dimension . . . . . . . . . . . . . . .

6.3

Intersection and Sum of Subspaces . .

6.4

Direct Sums and Orthogonal Subspaces

6.5

Bases . . . . . . . . . . . . . . . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

pp

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

lie

.

.

.

.

.

at

he

m

at

ics

3.5

3.6

3.7

3.8

121

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

Index

105

106

109

111

115

118

i

i

book

2009/5/27

page ix

i

This electronic version is for personal use and may not be duplicated or distributed.

at

ics

Preface

pp

lie

at

he

m

This book was written for a first-semester graduate course in matrix theory at North

Carolina State University. The students come from applied and pure mathematics,

all areas of engineering, and operations research. The book is self-contained. The

main topics covered in detail are linear system solution, least squares problems,

and singular value decomposition.

My objective was to present matrix analysis in the context of numerical

computation, with numerical conditioning of problems, and numerical stability of

algorithms at the forefront. I tried to present the material at a basic level, but in a

mathematically rigorous fashion.

la

nd

Main Features. This book differs in several regards from other numerical linear

algebra textbooks.

So

cie

ty

fo

rI

nd

us

tr

ia

Perturbation theory is used to determine sensitivity of problems as well as

numerical stability of algorithms, and the perturbation results built on each

other.

For instance, a condition number for matrix multiplication is used to derive a

residual bound for linear system solution (Fact 3.5), as well as a least squares

bound for perturbations on the right-hand side (Fact 5.11).

No floating point arithmetic.

There is hardly any mention of floating point arithmetic, for three main

reasons. First, sensitivity of numerical problems is, in general, not caused

by arithmetic in finite precision. Second, many special-purpose devices in

engineering applications perform fixed point arithmetic. Third, sensitivity

is an issue even in symbolic computation, when input data are not known

exactly.

Numerical stability in exact arithmetic.

A simplified concept of numerical stability is introduced to give quantitative

intuition, while avoiding tedious roundoff error analyses. The message is

that unstable algorithms come about if one decomposes a problem into illconditioned subproblems.

Two bounds for this simpler type of stability are presented for general direct solvers (Facts 3.14 and 3.17). These bounds imply, in turn, stability

bounds for solvers based on the following factorizations: LU (Corollary

3.22), Cholesky (Corollary 3.31), and QR (Corollary 3.33).

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

ix

i

i

i

book

2009/5/27

page x

i

This electronic version is for personal use and may not be duplicated or distributed.

Preface

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

Simple derivations.

The existence of a QR factorization for nonsingular matrices is deduced very

simply from the existence of a Cholesky factorization (Fact 3.32), without

any commitment to a particular algorithm such as Householder or Gram

Schmidt.

A new intuitive proof is given for the optimality of the singular value decomposition (Fact 4.13), based on the distance of a matrix from singularity.

I derive many relative perturbation bounds with regard to the perturbed solution, rather than the exact solution. Such bounds have several advantages:

They are computable; they give rise to intermediate absolute bounds (which

are useful in the context of fixed point arithmetic); and they are easy to

derive.

Especially for full rank least squares problems (Fact 5.14), such a perturbation bound can be derived fast, because it avoids the MoorePenrose inverse

of the perturbed matrix.

High-level view of algorithms.

Due to widely available high-quality mathematical software for small dense

matrices, I believe that it is not necessary anymore to present detailed implementations of direct methods in an introductory graduate text. This frees

up time for analyzing the accuracy of the output.

Complex arithmetic.

Results are presented for complex rather than real matrices, because engineering problems can give rise to complex matrices. Moreover, limiting

ones focus to real matrices makes it difficult to switch to complex matrices

later on. Many properties that are often taken for granted in the real case no

longer hold in the complex case.

Exercises.

The exercises contain many useful facts. A separate category of easier exercises, labeled with roman numerals, is appropriate for use in class.

So

cie

ty

and Teresa Selee for providing helpful suggestions and all MA523 students for

giving me the opportunity to develop the material in this book. It has been a

pleasure working with the SIAM publishing staff, in particular with Sara Murphy,

who made possible the publication of this book, and Lou Primus, who patiently

and competently dealt with all my typesetting requests.

Ilse Ipsen

Raleigh, NC, USA

December 2008

i

i

Preface

book

2009/5/27

page xi

i

This electronic version is for personal use and may not be duplicated or distributed.

xi

References

Gene H. Golub and Charles F. Van Loan: Matrix Computations, Third Edition,

Johns Hopkins Press, 1996

Nicholas J. Higham: Accuracy and Stability of Numerical Algorithms, Second

Edition, SIAM, 2002

Roger A. Horn and Charles A. Johnson: Matrix Analysis, Cambridge University

Press, 1985

he

m

at

Carl D. Meyer: Matrix Analysis and Applied Linear Algebra, SIAM, 2000

ics

Peter Lancaster and Miron Tismenetsky: The Theory of Matrices, Second Edition,

Academic Press, 1985

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

Gilbert Strang: Linear Algebra and Its Applications, Third Edition, Harcourt Brace

Jovanovich, 1988

i

i

book

2009/5/27

page xii

i

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

i

i

book

2009/5/27

page xiii

i

This electronic version is for personal use and may not be duplicated or distributed.

at

ics

Introduction

at

he

m

The goal of this book is to help you understand the sensitivity of matrix computations to errors in the input data. There are two important reasons for such

errors.

ia

la

nd

pp

lie

For instance, your weight on the scale tends to be 125 pounds, but may

change to 126 or 124 depending where you stand on the scale. So, you are

sure that the leading digits are 12, but you are not sure about the third digit.

Therefore the third digit is considered to be in error.

(ii) Arithmetic operations can produce errors.

Arithmetic operations may not give the exact result when they are carried

out in finite precision, e.g., in floating point arithmetic or in fixed point

arithmetic. This happens, for instance, when 1/3 is computed as .33333333.

ty

fo

rI

nd

us

tr

There are matrix computations that are sensitive to errors in the input. Consider the system of linear equations

1

1

x1 + x2 = 1,

3

3

1

x1 + .3x2 = 0,

3

So

cie

which has the solution x1 = 27 and x2 = 30. Suppose we make a small change

in the second equation and change the coefficient from .3 to 13 . The resulting linear

system

1

1

x1 + x2 = 1,

3

3

1

1

x 1 + x2 = 0

3

3

has no solution. A small change in the input causes a drastic change in the output,

i.e., the total loss of the solution. Why did this happen? How can we predict that

something like this can happen? That is the topic of this book.

xiii

i

i

i

book

2009/5/27

page xiv

i

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

i

i

book

2009/5/27

page 1

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

1. Matrices

pp

lie

nd

An array of numbers

a11

..

A= .

...

a1n

..

.

am1

...

amn

us

tr

ia

la

So

cie

ty

fo

rI

nd

(i, j ). The elements aij are scalars, namely, real or complex numbers. The set of

real numbers is R, and the set of complex numbers is C.

We write A Rmn if A is an mn matrix whose elements are real numbers,

and A Cmn if A is an m n matrix whose elements are complex numbers. Of

course, Rmn Cmn . If m = n, then we say that A is a square matrix of order n.

For instance,

1 2 3 4

A=

5 6 7 8

is a 2 4 matrix with elements a13 = 3 and a24 = 8.

Vectors. A row vector y = y1

column vector

...

x1

x = ...

xn

1

i

i

i

book

2009/5/27

page 2

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

then x Rn .

Submatrices. Sometimes we need only those elements of a matrix that are situated in particular rows and columns.

he

m

at

ics

Definition 1.1. Let A Cmn have elements aij . If 1 i1 < i2 < < ik m

and 1 j1 < j2 < < jl n, then the k l matrix

ai2 ,j1 ai2 ,j2 . . . ai2 ,jl

..

..

..

.

.

.

aik ,j1 aik ,j2 . . . aik ,jl

3

6 ,

9

2

5

8

nd

1

A = 4

7

pp

Example. If

lie

at

and its diagonal elements are diagonal elements of A, that is, k = l and i1 = j1 ,

i2 = j 2 , . . . , i k = j k .

a21

fo

rI

The submatrix

ia

3

,

6

a23 = 4

6 ,

1

a13

=

7

a33

3

9

tr

1

a13

=

4

a23

us

a11

a21

nd

la

a11

a31

a12

a22

a32

a13

2

a23 = 5

8

a33

3

6 .

9

So

cie

ty

is a principal matrix of A, as are the diagonal elements a11 , a22 , a33 , and A itself.

Notation. Most of the time we will use the following notation:

Matrices: uppercase Roman or Greek letters, e.g., A, .

Vectors: lowercase Roman letters, e.g., x, y.

Scalars: lowercase Greek letters, e.g., ;

or lowercase Roman with subscripts, e.g., xi , aij .

Running variables: i, j , k, l, m, and n.

The elements of the matrix A are called aij or Aij , and the elements of the vector

x are called xi .

Zero Matrices. The zero matrix 0mn is the m n matrix all of whose elements

are zero. When m and n are clear from the context, we also write 0. We say A = 0

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 3

i

This electronic version is for personal use and may not be duplicated or distributed.

if all elements of the matrix A are equal to zero. The matrix A is nonzero, A = 0,

if at least one element of A is nonzero.

Identity Matrices. The identity matrix of order n is the real square matrix

..

In =

.

1

pp

nd

lie

at

he

m

at

ics

with ones on the diagonal and zeros everywhere else (instead of writing many

zeros, we often write blanks). In particular, I1 = 1. When n is clear from the

context, we also write I .

The

matrix are also called canonical vectors ei . That

columns of the identity

is, In = e1 e2 . . . en , where

1

0

0

0

1

0

e1 = . ,

e2 = . ,

...,

en = . .

..

..

..

la

Exercises

So

cie

ty

fo

rI

nd

us

tr

ia

A square matrix of order n whose element in position (i, j ) is i+j11 ,

1 i, j n, is called a Hilbert matrix.

Write down a Hilbert matrix for n = 5.

(ii) Toeplitz Matrix.

Given 2n 1 scalars k , n + 1 k n 1, a matrix of order n whose

element in position (i, j ) is j i , 1 i, j n, is called a Toeplitz matrix.

Write down the Toeplitz matrix of order 3 when i = i, 2 i 2.

(iii) Hankel Matrix.

Given 2n 1 scalars k , 0 k 2n 2, a matrix of order n whose element

in position (i, j ) is i+j 2 , 1 i, j n, is called a Hankel matrix.

Write down the Hankel matrix of order 4 for i = i, 0 i 6.

(iv) Vandermonde Matrix.

Given n scalars i , 1 i n, a matrix of order n whose element in position

j 1

(i, j ) is i , 1 i, j n, is called a Vandermonde matrix. Here we interpret i0 = 1 even for i = 0. The numbers i are also called nodes of the

Vandermonde matrix.

Write down the Vandermonde matrix of order 4 when i = i, 1 i 3, and

4 = 0.

(v) Is a square zero matrix a Hilbert, Toeplitz, Hankel, or Vandermonde matrix?

(vi) Is the identity matrix a Hilbert, Toeplitz, Hankel, or Vandermonde matrix?

(vii) Is a Hilbert matrix a Hankel matrix or a Toeplitz matrix?

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 4

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

1.2

then the elements of the scalar matrix product A Cmn are

(A)ij aij .

Multiplying the matrix A Cmn by the scalar zero produces a zero matrix,

0 A = 0mn ,

at

ics

where the first zero is a scalar, while the second zero is a matrix with the same

number of rows and columns as A. Scalar matrix multiplication is associative,

A (1) A.

at

he

m

() A = (A).

pp

lie

Exercise

Matrix Addition

la

1.3

nd

us

tr

ia

Corresponding elements of two matrices are added. The matrices must have the

same number of rows and the same number of columns. If A and B Cmn , then

the elements of the sum A + B Cmn are

rI

nd

fo

So

cie

ty

Adding the zero matrix does not change anything. That is, for any m n

matrix A,

0mn + A = A + 0mn = A.

Matrix addition is commutative,

A + B = B + A.

Matrix addition is associative,

(A + B) + C = A + (B + C).

Matrix addition and scalar multiplication are distributive,

(A + B) = A + B,

( + ) A = A + A.

One can use the above properties to save computations. For instance, computing A + B requires twice as many operations as computing (A + B). In

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 5

i

This electronic version is for personal use and may not be duplicated or distributed.

while A + (B + C) = A + 0 = A requires no work.

A special type of addition is the sum of scalar vector products.

Definition 1.2. A linear combination of m column (or row) vectors v1 , . . . , vm ,

m 1, is

1 v1 + + m vm ,

ics

pp

1.4

lie

at

he

m

at

canonical vectors,

x1

x2

.. = x1 e1 + x2 e2 + + xn en .

.

xn

ia

la

nd

The product of a row vector times an equally long column vector produces a single

number. If

y1

..

x = x1 . . . xn ,

y = . ,

tr

yn

nd

us

fo

rI

xy = x1 y1 + + xn yn .

So

cie

ty

of two vectors with n elements each,

1

a1

n

a2

1

aj = a1 a2 . . . an . = 1 1 . . . 1 . .

..

..

j =1

an

Example. A polynomial p() = nj=0 j j of degree n can be represented as an

inner product of two vectors with n + 1 elements each,

0

1

p() = 1 . . . n . = 0 1 . . . n . .

..

..

n

n

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 6

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

Exercise

(i) Let n 1 be an integer. Represent n(n + 1)/2 as an inner product of two

vectors with n elements each.

1.5

at

ics

The product of a matrix and a vector is again a vector. There are two types of

matrix vector multiplications: matrix times column vector and row vector times

matrix.

nd

pp

lie

at

he

m

Matrix Times Column Vector. The product of matrix times column vector is

again a column vector. We present two ways to describe the operations that are

involved in a matrix vector product. Let A Cmn with rows rj and columns cj ,

and let x Cn with elements xj ,

r1

x1

..

..

A = . = c1 . . . cn ,

x = . .

rm

xn

us

tr

ia

la

inner product of row rj with x,

r1 x

Ax = ... .

rI

nd

rm x

Ax = c1 x1 + + cn xn .

cie

ty

fo

So

The vectors in the linear combination are the columns cj of A, and the

coefficients are the elements xj of x.

Example. Let

0

A = 0

1

0

0

2

0

0 .

3

The first view shows that Ae2 is equal to column 2 of A. That is,

0

0

0

0

Ae2 = 0 0 + 1 0 + 0 0 = 0 .

1

2

3

2

i

i

book

2009/5/27

page 7

i

This electronic version is for personal use and may not be duplicated or distributed.

The second view shows that the first and second elements of Ae2 are equal to zero.

That is,

0 0 0 e2

0

Ax = 0 0 0 e2 = 0 .

2

1 2 3 e2

at

ics

x1

x2

x = .

x3

x4

he

m

0 1 0 0

0 0 1 0

A=

,

0 0 0 1

0 0 0 0

nd

pp

lie

at

The first view shows that the last element of Ax is equal to zero. That is,

0 1 0 0

x2

0 0 1 0

x = x3

.

Ax =

x4

0 0 0 1

0

0 0 0 0

fo

rI

nd

us

tr

ia

la

Row Vector Times Matrix. The product of a row vector times a matrix is a row

vector. There are again two ways to think about this operation. Let A Cmn

with rows rj and columns cj , and let y C1m with elements yj ,

r1

A = ... = c1 . . . cn ,

y = y1 . . . ym .

rm

So

cie

ty

product of y with the column cj ,

yA = yc1 . . . ycn .

View 2: yA is a linear combination of rows of A,

yA = y1 r1 + + ym rm .

The vectors in the linear combination are the rows rj of A, and the coefficients

are the elements yj of y.

Exercises

(i) Show that Aej is the j th column of the matrix A.

(ii) Let A be an m n matrix and e the n 1 vector of all ones. What does Ae

do?

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 8

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

Prove: If one of the coefficients j is nonzero, then one of the vectors can

be represented as a linear combination of the other vectors.

1. Let A, B Cmn . Prove: A = B if and only if Ax = Bx for all x Cn .

1.6

Outer Product

x m y1

M

d

xm yn

...

pp

lie

x1 y1 . . . x1 yn

.. .

xy = ...

.

at

he

m

at

ics

The product of a column vector times a row vector gives a matrix (this is not to be

confused with an inner product which produces a single number). If

x1

..

y = y1 . . . yn ,

x = . ,

xm

tr

ia

la

nd

The vectors in an outer product are allowed to have different lengths. The columns

of xy are multiples of each other, and so are the rows. That is, each column of xy

is a multiple of x,

xy = xy1 . . . xyn ,

and each row of xy is a multiple of y,

xm y

ty

fo

rI

nd

us

x1 y

xy = ... .

So

cie

Example. A Vandermonde matrix of order n all of whose nodes are the same, e.g.,

equal to , can be represented as the outer product

1

..

. 1 . . . n1 .

1

Exercise

(i) Write the matrix below as an outer product:

4

5

8 10 .

12 15

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 9

i

This electronic version is for personal use and may not be duplicated or distributed.

1.7

Matrix Multiplication

at

ics

equal to the number of rows in B. Specifically, if A Cmn and B Cnp , then

AB Cmp . We can describe matrix multiplication in four different ways. Let

A Cmn with rows aj , and let B Cnp with columns bj :

a1

..

B = b1 . . . bp .

A = . ,

am

at

he

m

are matrix vector products of A with columns of B,

AB = Ab1 . . . Abp .

nd

pp

lie

View 2: AB is a block column vector of matrix vector products, where the rows

of AB are matrix vector products of the rows of A with B,

a1 B

AB = ... .

la

am B

us

tr

ia

an inner product of row i of A with column j of B,

1 i m,

1 j p.

rI

nd

(AB)ij = ai bj ,

So

cie

ty

fo

r1

A = c1 . . . cn ,

B = ... ,

rn

then AB is a sum of outer products, AB = c1 r1 + + cn rn .

Multiplying by the identity matrix does not change anything. That is, for an

m n matrix A,

Im A = A In = A.

Matrix multiplication is associative,

A (BC) = (AB) C.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

10

book

2009/5/27

page 10

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

(A + B) C = AC + BC.

For instance, if

0 1

1

A=

,

B=

0 0

0

2

4

6

8

10

he

m

at

us

3

6 3

9 2

12 1

15

tr

ia

la

1

2

(AB) C = 3

4

5

3 ,

nd

3

C = 2 ,

1

1

2

A = 3 ,

B= 1 2

4

5

1

= BA.

0

at

2

0

=

0

0

lie

0

0

pp

AB =

then

0

,

2

ics

A (B + C) = AB + AC,

cie

ty

fo

rI

nd

1

2

A (BC) = 3 10.

4

5

So

1 1

3

2 2

A = 3 3 ,

B= 1 2 3 ,

C = 2 ,

4 4

1

5 5

it looks as if we could compute

1

2

A (BC) = 3

4

5

1

2

3 10.

4

5

i

i

book

2009/5/27

page 11

i

This electronic version is for personal use and may not be duplicated or distributed.

11

However, the product ABC is not defined because AB is not defined (here we

have to view BC as a 1 1 matrix rather than just a scalar). In a product ABC, all

adjacent products AB and BC have to be defined. Hence the above option A (BC)

is not defined either.

Matrix Powers. A special case of matrix multiplication is the repeated multiplication of a square matrix by itself. If A is a nonzero square matrix, we define

A0 I , and for any integer k > 0,

k times

ics

A = A . . . A = Ak1 A = A Ak1 .

he

m

at

lie

at

involutory if A2 = I ,

idempotent (or a projector) if A2 = A,

nilpotent if Ak = 0 for some integer k > 0.

pp

tr

ia

la

nd

1

is involutory,

0 1

1

is idempotent,

0 0

us

and

is nilpotent.

cie

Which is the only matrix that is both idempotent and nilpotent?

Let x Cn1 , y C1n . When is xy idempotent? When is it nilpotent?

Prove: If A is idempotent, then I A is also idempotent.

Prove: If A and B are idempotent and AB = BA, then AB is also idempotent.

Prove: A is involutory if and only if (I A)(I + A) = 0.

Prove: If A is involutory and B = 12 (I + A), then B is idempotent.

So

(i)

(ii)

(iii)

(iv)

(v)

(vi)

(vii)

ty

Exercises

fo

rI

nd

0

0

(viii) Let x Cn1 , y C1n . Compute (xy)3 x using only inner products and

scalar multiplication.

1. Fast Matrix Multiplication.

One can multiply two complex numbers with only three real multiplications

instead of four. Let = 1 + 2 and = 1 + 2 be two complex numbers,

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 12

i

This electronic version is for personal use and may not be duplicated or distributed.

12

1. Matrices

= 1 1 2 2 + [(1 + 2 )(1 + 2 ) 1 1 2 2 ]

at

ics

shows that the complex product can be computed with three real multiplications: 1 1 , 2 2 , and (1 + 1 )(2 + 2 ).

Show that this approach can be extended to the multiplication AB of two

complex matrices A = A1 + A2 and B = B1 + B2 , where A1 , A2 Rmn

and B1 , B2 Rnp . In particular, show that no commutativity laws are

violated.

he

m

pp

lie

at

Transposing a matrix amounts to turning rows into columns and vice versa. If

a21 a22 . . . a2n

A= .

..

.. ,

..

.

.

am1 am2 . . . amn

tr

ia

la

nd

a12 a22 . . . am2

AT = .

..

.. .

..

.

.

us

a1n

a2n

...

amn

So

cie

ty

fo

rI

nd

the matrix elements are complex numbers. A complex number is written

= 1 + 2 , where 2 = 1 and 1 , 2 R. The complex conjugate of the

scalar is = 1 2 .

If A Cmn is a matrix, its conjugate transpose A Cnm is obtained by

converting rows to columns and, in addition, taking the complex conjugates of the

elements,

a 11 a 21 . . . a m1

a 12 a 22 . . . a m2

A = .

..

.. .

..

.

.

a 1n a 2n . . . a mn

Example. If

A=

then

1 + 2

A =

5

T

1 + 2

3

3

,

6

5

,

6

1 2

A =

5

3+

.

6

i

i

book

2009/5/27

page 13

i

This electronic version is for personal use and may not be duplicated or distributed.

13

Example. We can express the rows of the identity matrix in terms of canonical

vectors,

T

e1

e1

In = ... = ... .

en

enT

Fact 1.4 (Properties of Transposition).

(A ) = A.

at

(AT )T = A,

he

m

at

ics

For real matrices, the conjugate transpose and the transpose are identical.

That is, if A Rmn , then A = AT .

Transposing a matrix twice gives back the original,

(A) = A .

pp

(A)T = AT ,

lie

Transposition does not affect a scalar, while conjugate transposition conjugates the scalar,

nd

la

(A + B)T = AT + B T ,

(A + B) = A + B .

us

tr

ia

The transpose of a product is the product of the transposes with the factors

in reverse order,

(AB) = B A .

fo

rI

nd

(AB)T = B T AT ,

So

cie

ty

Example. Why do we have to reverse the order of the factors when the transpose

is pulled inside the product AB? Why isnt (AB)T = AT B T ? One of the reasons

is that one of the products may not be defined. If

1 1

1

A=

,

B=

,

1 1

1

then

(AB)T = 2

2 ,

Exercise

(i) Let A be an n n matrix, and let Z be the matrix with zj ,j +1 = 1,

1 j n 1, and all other elements zero. What does ZAZ T do?

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 14

i

This electronic version is for personal use and may not be duplicated or distributed.

14

1. Matrices

1.9

Transposition comes in handy for the representation of inner and outer products. If

x1

y1

..

..

x = . ,

y = . ,

xn

yn

y x = y 1 x1 + + y n xn .

at

x y = x 1 y1 + + x n yn ,

ics

then

lie

at

he

m

With

x= 1

2

la

nd

y T x = x T y.

x x = 0 if and only if x = 0.

If x is real, then x T x = 0 if and only if x = 0.

ia

1.

2.

3.

4.

pp

So

cie

ty

fo

rI

nd

us

tr

T

and y = y1 . . . yn . For the first equality

Proof. Let x

= x1 . . . x

n

write y x = nj=1 y j xj = nj=1 xj y j . Since complex conjugating twice gives

back the original, we get nj=1 xj y j = nj=1 x j y j = nj=1 x j yj = nj=1 x j yj =

x y, where the long overbar denotes complex

over the whole sum.

conjugation

As for the third statement, 0 = x x = nj=1 x j xj = nj=1 |xj |2 if and only

if xj = 0, 1 j n, if and only if x = 0.

Example. The identity matrix can be represented as the outer product

In = e1 e1T + e2 e2T + + en enT .

Exercises

(i) Let x be a column vector. Give an example to show that x T x = 0 can happen

for x = 0.

(ii) Let x Cn and x x = 1. Show that In 2xx is involutory.

(iii) Let A be a square matrix with aj ,j +1 = 1 and all other elements zero. Represent A as a sum of outer products.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 15

i

This electronic version is for personal use and may not be duplicated or distributed.

1.10

15

Definition 1.6. A matrix A Cnn is

ics

symmetric if AT = A,

Hermitian if A = A,

skew-symmetric if AT = A,

skew-Hermitian if A = A.

at

he

m

The identity matrix In is symmetric and Hermitian. The square zero matrix

0nn is symmetric, skew-symmetric, Hermitian, and skew-Hermitian.

is skew-symmetric,

1

2

M

d

is Hermitian,

lie

2

4

pp

2

0

1

2

2

4

is skew-Hermitian.

la

nd

0

2

at

Example. Let 2 = 1.

1 2

is symmetric,

2 4

Example. Let 2 = 1.

ia

us

tr

ty

fo

rI

nd

0

0

cie

So

Fact 1.7. If A Cmn , then AAT and AT A are symmetric, while AA and A A

are Hermitian.

If A Cnn , then A + AT is symmetric, and A + A is Hermitian.

Exercises

(i) Is a Hankel matrix symmetric, Hermitian, skew-symmetric, or skewHermitian?

(ii) Which matrix is both symmetric and skew-symmetric?

(iii) Prove: If A is a square matrix, then A AT is skew-symmetric and A A

is skew-Hermitian.

(iv) Which elements of a Hermitian matrix cannot be complex?

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 16

i

This electronic version is for personal use and may not be duplicated or distributed.

16

1. Matrices

at

he

m

at

ics

(v) What can you say about the diagonal elements of a skew-symmetric matrix?

(vi) What can you say about the diagonal elements of a skew-Hermitian matrix?

(vii) If A is symmetric and is a scalar, does this imply that A is symmetric? If

yes, give a proof. If no, give an example.

(viii) If A is Hermitian and is a scalar, does this imply that A is Hermitian? If

yes, give a proof. If no, give an example.

(ix) Prove: If A is skew-symmetric and is a scalar, then A is skew-symmetric.

(x) Prove: If A is skew-Hermitian and is a scalar, then A is, in general, not

skew-Hermitian.

(xi) Prove: If A is Hermitian, then A is skew-Hermitian, where 2 = 1.

(xii) Prove: If A is skew-Hermitian, then A is Hermitian, where 2 = 1.

(xiii) Prove: If A is a square matrix, then (A A ) is Hermitian, where 2 = 1.

pp

lie

Hermitian and A2 is skew-Hermitian.

2. Prove: Every square matrix A can be written A = A1 + A2 , where A1 and

A2 are Hermitian and 2 = 1.

nd

Inverse

la

1.11

us

tr

ia

of matrices is more complicated than inversion of scalars. There is only one scalar

that does not have an inverse: 0. But there are many matrices without inverses.

cie

Example.

ty

fo

rI

nd

Definition 1.8.

A matrix A Cnn is nonsingular (or invertible) if A has an

inverse, that is, if there is a matrix A1 so that AA1 = I = A1 A. If A does not

have an inverse, it is singular.

So

An involutory matrix is its own inverse: A2 = I .

Proof. Let A Cnn , and let AB = BA = In and AC = CA = In for matrices

B, C Cnn . Then

B = BIn = B(AC) = (BA)C = In C = C.

It is often easier to determine that a matrix is singular than it is to determine

that a matrix is nonsingular. The fact below illustrates this.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 17

i

This electronic version is for personal use and may not be duplicated or distributed.

1.11. Inverse

17

If x = 0 and Ax = 0, then A is singular.

If x = 0 and A is nonsingular, then Ax = 0.

If Ax = b, where A is nonsingular and b = 0, then x = 0.

at

ics

Proof. To prove the first statement, assume to the contrary that A is nonsingular

and has an inverse A1 . Then 0 = Ax implies 0 = A1 Ax = In x = x, hence x = 0,

which contradicts the assumption x = 0. Therefore A must be singular.

The proofs for the other two statements are similar.

he

m

at

I A = 0, in which case A is the identity, or else I A = 0, in which case it has

a nonzero column and Fact 1.10 implies that A is singular.

lie

(AT )1 = (A1 )T .

nd

(A )1 = (A1 ) ,

pp

ia

la

us

tr

A (A1 ) = (A1 A) = I = I

nd

and

rI

(A1 ) A = (AA1 ) = I = I .

fo

So

cie

ty

and AT .

The expression below is useful because it can break apart the inverse of a

sum.

Fact 1.13 (ShermanMorrison Formula). If A Cnn is nonsingular, and

V Cmn , U Cnm are such that I + V A1 U is nonsingular, then

1

(A + U V )1 = A1 A1 U I + V A1 U

V A1 .

Here is an explicit expression for the inverse of a partitioned matrix.

Fact 1.14. Let A Cnn and

A11

A=

A21

A12

.

A22

i

i

18

book

2009/5/27

page 18

i

This electronic version is for personal use and may not be duplicated or distributed.

1. Matrices

S11

A1 =

1

A22 A21 S11

1

A1

11 A12 S2

,

S21

1

where S1 = A11 A12 A1

22 A21 and S2 = A22 A21 A11 A12 .

ics

Exercises

pp

lie

at

he

m

at

(ii) Prove: If A, B Cnn are nonsingular, then B 1 = A1 B 1 (B A)A1 .

(iii) Let A Cmn , B Cnm be such that I + BA is invertible. Show that

(I + BA)1 = I B(I + AB)1 A.

(iv) Let A Cnn be nonsingular, u Cn1 , v C1n , and vA1 u = 1. Show

that

A1 uvA1

(A + uv)1 = A1

.

1 + vA1 u

ia

la

nd

(v) The following expression for the partitioned inverse requires only A11 to be

nonsingular but not A22 .

Let A Cnn and

A11 A12

.

A=

A21 A22

rI

nd

us

tr

11 A12 , then

1

1

1

1

A1

A11 + A1

11 A12 S A21 A11

11 A12 S

.

A1 =

S 1 A21 A1

S 1

11

So

cie

ty

fo

(vii) Prove: The inverse, if it exists, of a Hermitian (symmetric) matrix is also

Hermitian (symmetric).

(viii) Prove: If A is involutory, then I A or I + A must be singular.

(ix) Let A be a square matrix so that A + A2 = I . Prove: A is invertible.

(x) Prove: A nilpotent matrix is always singular.

1. Let S Rnn . Show: If S is skew-symmetric, then I S is nonsingular.

Give an example to illustrate that I S can be singular if S Cnn .

2. Let x be a nonzero column vector. Determine a row vector y so that yx = 1.

3. Let A be a square matrix and let j be scalars, at least two of which are

nonzero, such that kj =0 j Aj = 0. Prove: If 0 = 0, then A is nonsingular.

4. Prove: If (I A)1 = kj =0 Aj for some integer k 0, then A is nilpotent.

5. Let A, B Cnn . Prove: If I + BA is invertible, then I + AB is also

invertible.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 19

i

This electronic version is for personal use and may not be duplicated or distributed.

1.12

19

Definition 1.15. A matrix A Cnn is

unitary if AA = A A = I ,

at

he

m

at

Example 1.16. Let c and s be scalars with |c|2 + |s|2 = 1. The matrices

c s

c s

,

s c

s c

ics

orthogonal if AAT = AT A = I .

are unitary.

lie

la

nd

pp

c s

s c

us

tr

ia

is

calleda Givens rotation. If c and s are also real, then the Givens rotation

c s

is orthogonal.

s c

cie

ty

fo

rI

nd

When a Givens rotation is real, then both diagonal elements are the same.

When a Givens rotation is complex, then the diagonal elements are complex conjugates of each other. A unitary matrix of the form

c s

,

s c

So

where the real parts of the diagonal elements have different signs, is a reflection;

it is not a Givens rotation.

An orthogonal matrix that can reorder the rows or columns of a matrix is

called a permutation matrix. It is an identity matrix whose rows have been reordered (permuted). One can also think of a permutation matrix as an identity

matrix whose columns have been reordered. Here is the official definition.

Definition 1.18 (Permutation Matrix). A square matrix is a permutation matrix

if it contains a single one in each column and in each row, and zeros everywhere

else.

Example. The following are permutation matrices.

The identity matrix I .

i

i

book

2009/5/27

page 20

i

This electronic version is for personal use and may not be duplicated or distributed.

20

1. Matrices

J =

1

..

x1

xn

x2 ..

J . = . .

.. x

2

xn

ics

lie

at

he

m

at

xn

x1

.. x1

Z . = . .

xn1 ..

xn1

xn

0 1

0 1

.

.

.

.

,

.

.

Z=

..

. 1

1

0

x1

la

nd

pp

That is, if P is a permutation matrix, then P P T = P T P = P P = P P = I .

2. The product of permutation matrices is again a permutation matrix.

ia

Exercises

So

cie

ty

fo

rI

nd

us

tr

What can you say about an involutory matrix that is also unitary (orthogonal)?

Which idempotent matrix is unitary and orthogonal?

Prove: If A is unitary, so is A, where 2 = 1.

Prove: The product of unitary matrices is unitary.

Partitioned Unitary Matrices.

columns, and A2 has n k columns. Show that A1 A1 = Ik , A2 A2 = Ink ,

and A1 A2 = 0.

(vii) Let x Cn and x x = 1. Prove: In 2xx is Hermitian and unitary. Conclude that In 2xx is involutory.

(viii) Show: If P is a permutation matrix, then P T and P are also permutation

matrices.

(ix) Show: If P1 P2 is a permutation matrix, then P2 P1 is also a permutation matrix.

(i)

(ii)

(iii)

(iv)

(v)

(vi)

Triangular matrices occur frequently during the solution of systems of linear equations, because linear systems with triangular matrices are easy to solve.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 21

i

This electronic version is for personal use and may not be duplicated or distributed.

21

Definition 1.20. A matrix A Cnn is upper triangular if aij = 0 for i > j . That is,

a11 . . . a1n

.. .

..

A=

.

.

ann

A matrix A Cnn is lower triangular if AT is upper triangular.

ics

Fact 1.21. Let A and B be upper triangular, with diagonal elements ajj and bjj ,

respectively.

at

he

m

at

The diagonal elements of AB are aii bii .

If ajj = 0 for all j , then A is invertible, and the diagonal elements of A1

are 1/ajj .

pp

lie

diagonal, and strictly triangular if it has zeros on the diagonal.

la

nd

Example. The identity matrix is unit upper triangular and unit lower triangular.

The square zero matrix is strictly lower triangular and strictly upper triangular.

tr

ia

Exercises

rI

nd

us

(i) What does an idempotent triangular matrix look like? What does an involutory triangular matrix look like?

So

cie

ty

fo

If A and B are unit triangular, then so is the product AB.

2. Show that a strictly triangular matrix is nilpotent.

3. Explain why the matrix I ei ejT is triangular. When does it have an inverse? Determine the inverse in those cases where it exists.

4. Prove:

1 2 . . . n

1

.

.

. . ..

1

.

.

..

..

=

.. ..

2

.

.

1

1

5. Uniqueness of LU Factorization.

Let L1 , L2 be unit lower triangular, and U1 , U2 nonsingular upper triangular.

Prove: If L1 U1 = L2 U2 , then L1 = L2 and U1 = U2 .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 22

i

This electronic version is for personal use and may not be duplicated or distributed.

22

1. Matrices

6. Uniqueness of QR Factorization.

Let Q1 , Q2 be unitary (or orthogonal), and R1 , R2 upper triangular with

positive diagonal elements. Prove: If Q1 R1 = Q2 R2 , then Q1 = Q2 and

R1 = R 2 .

1.14

Diagonal Matrices

ics

Diagonal matrices are special cases of triangular matrices; they are upper and lower

triangular at the same time.

at

he

m

at

a11

..

A=

.

.

lie

ann

pp

The identity matrix and the square zero matrix are diagonal.

nd

Exercises

fo

rI

nd

us

tr

ia

la

(i) Prove: Diagonal matrices are symmetric. Are they also Hermitian?

(ii) Diagonal matrices commute.

Prove: If A and B are diagonal, then AB is diagonal, and AB = BA.

(iii) Represent a diagonal matrix as a sum of outer products.

(iv) Which diagonal matrices are involutory, idempotent, or nilpotent?

(v) Prove: If a matrix is unitary and triangular, it must be diagonal. What are its

diagonal elements?

So

cie

ty

i

i

book

2009/5/27

page 23

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

Norms

pp

lie

Two difficulties arise when we solve systems of linear equations or perform other

matrix computations.

cie

ty

fo

rI

nd

us

tr

ia

la

nd

Matrix elements may be contaminated with errors from measurements or

previous computations, or they may simply not be known exactly. Merely

inputting numbers into a computer or calculator can cause errors (e.g., when

1/3 is stored as .33333333). To account for all these situations, we say that

the matrix elements are afflicted with uncertainties or are perturbed . In

general, perturbations of the inputs cause difficulties when the outputs are

sensitive to changes in the inputs.

(ii) Errors in algorithms.

Algorithms may not compute an exact solution, because computing the exact

solution may not be necessary, may take too long, may require too much

storage, or may not be practical. Furthermore, arithmetic operations in finite

precision may not be performed exactly.

So

affect the outputs.

2.1

In real life, sensitive means1 acutely affected by external stimuli, easily offended

or emotionally hurt, or responsive to slight changes. A sensitive person can be

easily upset by small events, such as having to wait in line for a few minutes.

Hardware can be sensitive: A very slight turn of a faucet may change the water

from freezing cold to scalding hot. The slightest turn of the steering wheel when

driving on an icy surface can send the car careening into a spin. Organs can be

1 The

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

23

i

i

i

24

book

2009/5/27

page 24

i

This electronic version is for personal use and may not be duplicated or distributed.

sensitive: Healthy skin may not even feel the prick of a needle, while it may cause

extreme pain on burnt skin.

It is no different in mathematics. Steep functions, for instance, can be sensitive to small perturbations in the input.

Example. Let f (x) = 9x and consider the effect of a small perturbation to the

input of f (50) = 950 , such as

at

ics

Here a 1 percent change in the input causes a 300 percent change of the output.

27

x=

.

30

at

M

nd

pp

lie

1/3 1/3

1

A=

,

b=

1/3 .3

0

he

m

matrix or the right-hand side causes a large change in the solution.

us

tr

ia

la

However, a small change of the (2, 2) element from .3 to 1/3 results in the total

= b with

loss of the solution, because the system Ax

1/3 1/3

A =

1/3 1/3

rI

nd

has no solution.

cie

ty

fo

A linear system like the one above whose solution is sensitive to small perturbations in the matrix is called ill-conditioned . Here is another example of

ill-conditioning.

So

1

1

1

A=

,

b=

,

1 1+

1

has the solution

x=

0 < 1,

1 2

.

2

But changing the (2, 2) element of A from 1 + to 1 results in the loss of the

= b with

solution, because the linear system Ax

1 1

A =

1 1

has no solution. This happens regardless of how small is.

i

i

book

2009/5/27

page 25

i

This electronic version is for personal use and may not be duplicated or distributed.

25

in the right-hand side, as the next example shows.

Example 2.2. The linear system Ax = b with

1

1

2

A=

,

b=

,

1 1+

2

0 < 1,

at

T

1 , which is completely different from x.

has the solution x = 1

he

m

at

ics

T

has the solution x = 2 0 . Changing the leading element in the right-hand side

from 2 to 2 + alters the solution radically. That is, the system Ax = b with

2

b =

2+

tr

ia

la

nd

pp

lie

compute the solution. Ill-conditioning is a property of the linear system. Hence

there is, in general, nothing you can do about ill-conditioning.

In an ill-conditioned linear system, errors in the matrix or in the right-hand

side can be amplified so that the errors in the solution are much larger. Our aim is to

determine which properties of a linear system are responsible for ill-conditioning,

and how one can quantify ill-conditioning.

nd

us

rI

So

cie

ty

fo

Example. Suppose you have y = 10 dollars in your bank account. But the bank

makes a mistake and subtracts 20 dollars from your account, so that your account

now has a negative balance of y = 10 dollars. The account is overdrawn, and all

kinds of bad consequences ensue.

Now imagine this happens to Bill Gatez. He has g = 1011 dollars in his

account, and if the bank subtracts by mistake 20 dollars from his balance, he still

has g = 1011 20 dollars.

In both cases, the bank makes the same error,

y y = g g = 20.

But you are much worse off than Bill Gatez. You are now in debt, while Bill Gatez

has so much money, he may not even notice the error. In your case, the error is

larger than your credit; while in Bill Gatezs case, the error is only a tiny part of

his fortune.

How can we express mathematically that the banks error is much worse for

you than for Bill Gatez? We can compare the error to the balance in your account:

yy

y = 2. This shows that the error is twice as large as your original balance.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 26

i

This electronic version is for personal use and may not be duplicated or distributed.

26

g

10 , so that the error is only a tiny fraction

For Bill Gatez we obtain g

g = 2 10

of his balance. Now its clear that the banks error is much more serious for you

than it is for Bill Gatez.

y

yy

A difference like y y measures an absolute error, while y

y and y

measure relative errors. We use relative errors if we want to know how large the

error is when compared to the original quantity. Often we are not interested in the

y|

|yy|

|y| and |y|

.

at

|xx|

|x|

he

m

then

ics

x|

|x x|

an absolute error. If x = 0, then we call |x

|x| a relative error. If x = 0,

nd

pp

lie

at

x|

|x|, which

totally inaccurate. To see this, suppose that |x

|x| 1. Then |x x|

means that the absolute error is larger than the quantity we are trying to compute.

If we approximate x = 0 by x = 0, however small, then the relative error is always

|0x|

|x|

= 1. Thus, the only approximation to 0 that has a small relative error is 0

itself.

In contrast to an absolute error, a relative error can give information about

how many digits two numbers have in common. As a rule of thumb, if

tr

ia

la

|x x|

5 10d ,

|x|

nd

us

So

2.3

cie

ty

fo

rI

x|

Example. If x = 1 and x = 1.003, then |x

|x| = 3 10

x agree to three decimal digits.

According to the above definition, the numbers x = 1 and x = .997 also agree

x|

3 5 103 .

to three decimal digits because |x

|x| = 3 10

Many computations in science and engineering are carried out in floating point

arithmetic, where all real numbers are represented by a finite set of floating point

numbers. All floating point numbers are stored in the same, fixed number of bits

regardless of how small or how large they are. Many computers are based on IEEE

double precision arithmetic where a floating point number is stored in 64 bits.

The floating point representation x of a real number x differs from x by a

factor close to one, and satisfies2

x = x(1 + x ),

where

|x | u.

2 We assume that x lies in the range of normalized floating point numbers, so that no underflow

or overflow occurs.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 27

i

This electronic version is for personal use and may not be duplicated or distributed.

27

Here u is the unit roundoff that specifies the accuracy of floating point arithmetic.

In IEEE double precision arithmetic u = 253 1016 . If x = 0, then

|x x|

|x |.

|x|

Conditioning of Subtraction

lie

2.4

at

he

m

at

ics

This means that conversion to floating point representation causes relative errors.

We say that a floating point number x is a relative perturbation of the exact number x.

Since floating point arithmetic causes relative perturbations in the inputs, it

makes sense to determine relativerather than absoluteerrors in the output. As

a consequence, we will pay more attention to relative errors than to absolute errors.

The question now is how elementary arithmetic operations are affected when

they are performed on numbers contaminated with small relative perturbations,

such as floating point numbers. We start with subtraction.

nd

pp

Subtraction is the only elementary operation that is sensitive to relative perturbations. The analogy below of the captain and the battleship can help us understand

why.

x = 1122339,

y = 1122337.

ty

fo

rI

nd

us

tr

ia

la

Example. To find out how much he weighs, the captain first weighs the battleship

with himself on it, and then he steps off the battleship and weighs it without himself

on it. At the end he subtracts the two weights. Intuitively we have a vague feeling

for why this should not give an accurate estimate for the captains weight. Below

we explain why.

Let x represent the weight of the battleship plus captain, and y the weight of

the battleship without the captain, where

So

cie

Due to the limited precision of the scale, the underlined digits are uncertain and

may be in error. The captain computes as his weight x y = 2. This difference

is totally inaccurate because it is derived from uncertainties, while all the accurate

digits have cancelled out. This is an example of catastrophic cancellation.

Catastrophic cancellation occurs when we subtract two numbers that are

uncertain, and when the difference between these two numbers is as small as

the uncertainties. We will now show that catastrophic cancellation occurs when

subtraction is ill-conditioned with regard to relative errors.

Let x be a perturbation of the scalar x and y a perturbation of the scalar y.

We bound the error in x y in terms of the errors in x and y.

Absolute Error.

From

|(x y)

(x y)| |x x| + |y y|,

i

i

book

2009/5/27

page 28

i

This electronic version is for personal use and may not be duplicated or distributed.

28

we see that the absolute error in the difference is bounded by the absolute errors in

the inputs. Therefore we say that subtraction is well-conditioned in the absolute

sense. In the above example, the last digit of x and y is uncertain, so that |x x| 9

and |y y| 9, and the absolute error is bounded by |(x y)

(x y)| 18.

Relative Error. However, the relative error in the difference can be much larger

than the relative error in the inputs. In the above example we can estimate the

relative error from

at

ics

|(x y)

(x y)| 18

= 9,

|x y|

lie

at

he

m

In general, this severe loss of accuracy can occur when we subtract two

nearly equal numbers that are in error. The bound in Fact 2.4 below shows that

subtraction can be ill-conditioned in the relative sense if the difference is much

smaller in magnitude than the inputs.

tr

ia

la

nd

pp

and y be scalars.

If x = 0, y = 0, and x = y, then

|(x y)

(x y)|

|x x| |y y|

max

,

,

|x y|

|x|

|y|

rI

nd

us

where

|x| + |y|

.

|x y|

So

cie

ty

fo

it quantifies how relative errors in the input can be amplified, and how sensitive

subtraction can be to relative errors in the input. When 1, subtraction is

ill-conditioned in the relative sense and is called catastrophic cancellation.

If we do not know x, y, or x y, but want an estimate of the condition

number, we can use instead the bound

|x x| |y y|

|x|

+ |y|

|(x y)

(x y)|

max

,

,

=

,

|x y|

|x|

|y|

|x y|

provided x = 0, y = 0, and x = y,

Remark 2.5. Catastrophic cancellation does not occur when we subtract two

numbers that are exact.

Catastrophic cancellation can only occur when we subtract two numbers

that have relative errors. It is the amplification of these relative errors that leads

to catastrophe.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 29

i

This electronic version is for personal use and may not be duplicated or distributed.

29

Exercises

1. Relative Conditioning of Multiplication.

Let x, y, x,

y be nonzero scalars. Show:

xy x y

x x y y

(2 + ),

,

where = max

,

xy

x y

and if 1, then

at

ics

xy x y

xy 3.

nd

pp

lie

at

he

m

Therefore, if the relative error in the inputs is not too large, then the condition

number of multiplication is at most 3. We can conclude that multiplication is

well-conditioned in the relative sense, provided the inputs have small relative

perturbations.

2. Relative Conditioning of Division.

Let x, y, x,

y be nonzero scalars, and let

x x y y

,

.

= max

x y

Show: If < 1, then

rI

nd

us

tr

ia

la

x/y x/

y

2

x/y

1,

x/y x/

y

x/y

4.

So

cie

ty

fo

Therefore, if the relative error in the operands is not too large, then the

condition number of division is at most 4. We can conclude that division is

well-conditioned in the relative sense, provided the inputs have small relative

perturbations.

In the context of linear system solution, the error in the solution constitutes a

vector. If we do not want to pay attention to individual components of the error,

perhaps because there are too many components, then we can combine all errors

into a single number. This is akin to a grade point average which combines all

grades into a single number. Mathematically, this combining is accomplished

by norms. We start with vector norms, which measure the length of a vector.

Definition 2.6. A vector norm is a function from Cn to R with three properties:

N1: x 0 for all x Cn , and x = 0 if and only if x = 0.

i

i

30

book

2009/5/27

page 30

i

This electronic version is for personal use and may not be duplicated or distributed.

N3: x = || x for all C, x Cn .

at

ics

The vector p-norms below are useful for computational purposes, as well as

analysis.

T

Fact 2.7 (Vector p-Norms). Let x Cn with elements x = x1 . . . xn . The

p-norm

1/p

n

|xj |p ,

p 1,

xp =

he

m

j =1

at

is a vector norm.

Example.

e = 1,

ep = n1/p ,

1 < p < .

e1 = n,

pp

lie

If e = 1 1 1 T Rn , then

us

tr

ia

la

nd

The three p-norms below are the most popular, because they are easy to

compute.

One norm: x1 = nj=1 |xj |.

n

2

j =1 |xj | = x x.

fo

x = n.

cie

ty

1

x1 = n(n + 1),

2

Rn , then

1

x2 =

n(n + 1)(2n + 1),

6

rI

Example. If x = 1

nd

So

Hlder inequality: |x y| x1 y

CauchySchwarz inequality: |x y| x2 y2 .

Moreover, |x y| = x2 y2 if and only if x and y are multiples of each other.

T

Example. Let x Cn with elements x = x1 xn . The Hlder inequality

and CauchySchwarz inequality imply, respectively,

n

n

xi n max |xi |,

xi n x2 .

1in

i=1

i=1

i

i

book

2009/5/27

page 31

i

This electronic version is for personal use and may not be duplicated or distributed.

31

if x = 1. The vector x/x has unit norm.

Example. Let e be the n 1 vector of all ones. Then

1

1

1 = e = e = e

.

n 1

n 2

ics

he

m

at

gives about individual, componentwise errors.

at

is a

x

xx

and

are

normwise

normwise absolute error. If x = 0 or x = 0, then x

x

x

relative errors.

pp

lie

errors? For vectors x, x Cn , the infinity norm is equal to the largest absolute

error,

x x

= max |xj xj |.

la

nd

1j n

ia

max |xj xj | x x

1 n max |xj xj |

1j n

us

tr

1j n

nd

and

rI

max |xj xj | x x

2

n max |xj xj |.

1j n

fo

1j n

So

cie

ty

Hence absolute errors in the one and two norms can overestimate the worst componentwise error by a factor that depends on the vector length n.

Unfortunately, normwise relative errors give much less information about

componentwise relative errors.

Example. Let x be an approximation to a vector x where

1

1

x=

, 0 < 1,

x =

.

0

The normwise relative error

xx

x

inaccurate approximation to x2 in the relative sense.

The preceding example illustrates that a normwise relative error can be

small, even if individual vector elements have a large relative error. In the infinity norm, for example, the normwise relative error only bounds the relative

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

32

book

2009/5/27

page 32

i

This electronic version is for personal use and may not be duplicated or distributed.

let |xk | = x . Then

max1j n |xj xj | |xk xk |

x x

=

.

x

|xk |

|xk |

1 |xk xk |

x x

1

,

x1

n |xk |

x x

2

1 |xk xk |

.

x2

n |xk |

ics

For the normwise relative errors in the one and two norms we incur additional

factors that depend on the vector length n,

at

he

m

at

components of smaller magnitude. If relative errors in individual vector components are important, then do not use normwise errors.

x x

,

x

x x

.

x

=

nd

=

pp

lie

x

xx

x,

x or x

? If x

then the two errors are about the same. In general, the two errors are related as

follows. Let x = 0, x = 0, and

If < 1, then

nd

us

tr

ia

la

.

1+

1

This follows from = x/x

and 1 x/x

1 + .

Exercises

cie

ty

fo

rI

(ii) For each equality below, determine a class of vectors that satisfy the equality:

x1 = x ,

x1 = nx ,

x2 = x ,

x2 = nx .

So

|x y| = x1 y . Also find examples for |x y| = x2 y2 .

(iv) The p-norm of a vector does not change when the vector is permuted.

Prove: If P is a permutation matrix, then P xp = xp .

(v) The two norm of a vector does not change when the vector is multiplied by

a unitary matrix.

Prove: If the matrix V Cnn is unitary, then V x2 = x2 for any vector

x Cn .

(vi) Prove: If Q Cnn is unitary and x Cn is a nonzero vector with Qx = x,

where is a scalar, then || = 1.

1. Verify that the vector p-norms do indeed satisfy the three properties of a

vector norm in Definition 2.6.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 33

i

This electronic version is for personal use and may not be duplicated or distributed.

33

at

he

m

at

ics

Let x, y Cn and let be a vector norm. Prove: x y x y.

3. Theorem of Pythagoras.

Prove: If x, y Cn and x y = 0, then x y22 = x22 + y22 .

4. Parallelogram Equality.

Let x, y Cn . Prove: x + y22 + x y22 = 2(x22 + y22 ).

5. Polarization Identity.

Let x, y Cn . Prove: (x y) = 14 (x + y22 x y22 ), where () is the

real part of a complex number .

6. Let x Cn . Prove:

x x1 nx .

Matrix Norms

pp

2.6

lie

tr

ia

la

nd

We need to separate matrices from vectors inside the norms. To see this, let

Ax = b be a nonsingular linear system, and let Ax = b be a perturbed system.

= A1 (b b).

In order to isolate the

we have to define a

perturbation and derive a bound of the form A1 b b,

norm for matrices.

nd

us

properties:

rI

ty

fo

So

cie

Because of the triangle inequality, matrix norms are well-conditioned, in the

absolute sense and in the relative sense.

Fact 2.13. If A, E Cmn , then A + E A E.

Proof.

The triangle inequality implies A + E A + E, hence

A + E A E. Similarly A = (A + E) E A + E + E,

so that E A + E A. The result follows from

E A + E A E.

The matrix p-norms below are based on the vector p-norms and measure

how much a matrix can stretch a unit-norm vector.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 34

i

This electronic version is for personal use and may not be duplicated or distributed.

34

Ap = max

x=0

Axp

= max Axp

xp =1

xp

is a matrix norm.

ics

Remark 2.15. The matrix p-norms are extremely useful because they satisfy the

following submultiplicative inequality.

Let A Cmn and y Cn . Then

at

at

x=0

Axp

Ayp

.

xp

yp

Ap = max

he

m

pp

lie

The matrix one norm is equal to the maximal absolute column sum.

nd

1j n

la

1j n

|aij |.

i=1

tr

ia

Proof.

us

1 j n.

nd

rI

x1 =1

So

cie

ty

fo

Viewing the matrix vector product Ay as a linear combination of columns

of A, see Section 1.5, and applying the triangle inequality for vector norms

gives

A1 = Ay1 = y1 Ae1 + + yn Aen 1 |y1 |Ae1 1 + + |yn |Aen 1

(|y1 | + + |yn |) max Aej 1 .

1j n

The matrix infinity norm is equal to the maximal absolute row sum.

Fact 2.17 (Infinity Norm). Let A Cmn . Then

A = max A ei 1 = max

1im

1im

|aij |.

j =1

i

i

book

2009/5/27

page 35

i

This electronic version is for personal use and may not be duplicated or distributed.

35

Proof. Denote the rows of A by ri = ei A, and let rk have the largest one norm,

rk 1 = max1im ri 1 .

Let y be a vector with A = Ay and y = 1. Then

A = Ay = max |ri y| max ri 1 y = rk 1 ,

1im

1im

at

he

m

at

ics

where the inequality follows from Fact 2.8. Hence A max1i ri 1 .

For any vector y with y = 1 we have A Ay |rk y|. Now

we show

the elements of y such that rk y = rk 1 . Let

how to choose

= 0, then yj =0, and otherwise yj = |j |/j .

that j yj = |j |. That is, if j

Then y = 1 and |rk y| = nj=1 j yj = nj=1 |j | = rk 1 . Hence

1im

lie

pp

nd

tr

ia

la

Proof. Let x Cp such that ABp = ABxp and xp = 1. Applying Remark

2.15 twice gives

nd

us

cie

ty

fo

rI

until later. However, even without knowing how to compute it, we can still derive

several useful properties of the two norm. If x is a column vector, then x22 = x x.

We show below that an analogous property holds for matrices. We also show that

a matrix and its transpose have the same two norm.

So

A 2 = A2 ,

A A2 = A22 .

Proof. The definition of the two norm implies that for some x Cn with x2 = 1

we have A2 = Ax2 . The definition of the vector two norm implies

A22 = Ax22 = x A Ax x2 A Ax2 A A2 ,

where the first inequality follows from the CauchySchwarz inequality in Fact 2.8

and the second inequality from the two norm of A A. Hence A22 A A2 .

Fact 2.18 implies A A2 A 2 A2 . As a consequence,

A22 A A2 A 2 A2 ,

A2 A 2 .

i

i

36

book

2009/5/27

page 36

i

This electronic version is for personal use and may not be duplicated or distributed.

A 22 AA 2 A2 A 2 ,

A 2 A2 .

If we omit a piece of a matrix, the norm does not increase but it can decrease.

ics

Bp Ap .

he

m

at

Exercises

tr

ia

la

nd

pp

lie

at

(i) Let D Cnn be a diagonal matrix with diagonal elements djj . Show that

Dp = max1j n |djj |.

(ii) Let A Cnn be nonsingular. Show: Ap A1 p 1.

(iii) Show: If P is a permutation matrix, then P p = 1.

(iv) Let P Rmm , Q Rnn be permutation matrices and let A Cmn . Show:

P AQp = Ap .

(v) Let U Cmm and V Cnn be unitary. Show: U 2 = V 2 = 1, and

U BV 2 = B2 for any B Cmn .

(vi) Let x Cn . Show: x 2 = x2 without using Fact 2.19.

(vii) Let x Cn . Is x1 = x 1 , and x = x ? Why or why not?

(viii) Let x Cn be the vector of all ones. Determine

x 1 ,

x ,

us

x1 ,

x ,

x2 ,

x 2 .

fo

rI

nd

(ix) For each of the two equalities, determine a class of matrices A that satisfy

the equality A = A1 , and A = A1 = A2 .

(x) Let A Cmn . Then A = A 1 .

So

cie

ty

1. Verify that the matrix p-norms do indeed satisfy the three properties of a

matrix norm in Definition 2.12.

2. Let A Cmn . Prove:

i,j

i,j

1

A A2 mA ,

n

1

A1 A2 nA1 .

m

3. Norms of Outer Products.

Let x Cm and y Cn . Show:

xy 2 = x2 y2 ,

i

i

book

2009/5/27

page 37

i

This electronic version is for personal use and may not be duplicated or distributed.

37

Multiplication

at

2.7

he

m

at

ics

two norm that realizes z, in the sense that the perturbed system has z as a

solution.

Let A Cnn , Ax = b, and z = 0. Show: Among all matrices E with

(A + E)z = b the matrix E0 = (b Az)z has the smallest two norm, where

z = (z z)1 z .

5. Norms of Idempotent Matrices.

Show: If A = 0 is idempotent, then Ap 1. If A is also Hermitian, then

A2 = 1.

6. Let A Cnn . Show: Among all Hermitian matrices, 12 (A + A ) is the

matrix that is closest to A in the two norm.

lie

We derive normwise relative bounds for matrix addition and subtraction, as well

as for matrix multiplication.

la

nd

pp

Then

U + V (U + V )p

U p + V p

max{U , V },

U + V p

U + V p

ia

U U p

,

U p

V =

V V p

.

V p

us

U =

tr

where

nd

ty

fo

rI

U + V (U + V )p U U p + V V p = U p U + V p V

(U p + V p ) max{U , V }.

So

cie

V is (U p + V p )/U + V p . It is analogous to the condition number for

scalar subtraction in Fact 2.4. If U p + V p U + V p , then the matrix

addition U + V is well-conditioned in the normwise relative sense. But if

U p + V p U + V p , then the matrix addition U + V is ill-conditioned

in the normwise relative sense.

Fact 2.22 (Matrix Multiplication). Let U , U Cmn and V , V Cnp such that

U , V , U V = 0. Then

U V U V p

U p V p

(U + V + U V ) ,

U V p

U V p

where

U =

U U p

,

U p

V =

V V p

.

V p

i

i

book

2009/5/27

page 38

i

This electronic version is for personal use and may not be duplicated or distributed.

38

U V U V = (U + E)(V + F ) U V = U F + EV + EF .

Now take norms, apply the triangle inequality, and divide by U V p .

at

ics

Fact 2.22 shows that the normwise relative condition number for multiplying matrices U and V is U p V p /U V p . If U p V p U V p , then

the matrix multiplication U V is well-conditioned in the normwise relative sense.

However, if U p V p U V p , then the matrix multiplication U V is illconditioned in the normwise relative sense.

he

m

Exercises

pp

lie

at

(i) What is the two-norm condition number of a product where one of the matrices is unitary?

(ii) Normwise absolute condition number for matrix multiplication when one of

the matrices is perturbed.

Let U , V Cnn , and U be nonsingular. Show:

nd

F p

U (V + F ) U V p U p F p .

U 1 p

tr

ia

la

(iii) Here is a bound on the normwise relative error for matrix multiplication with

regard to the perturbed product.

Let U Cmn and V Cnm . Show: If (U + E)(V + F ) = 0, then

rI

nd

us

(U + E)(V + F ) U V p

U + Ep V + F p

(U + V + U V ) ,

(U + E)(V + F )p

(U + E)(V + F )p

fo

where

V =

F p

.

V + F p

So

2.8

Ep

,

U + Ep

cie

ty

U =

We start by bounding the inverse of a perturbed identity matrix. If the

norm of the perturbation is sufficiently small, then the perturbed identity matrix is

nonsingular.

Fact 2.23 (Inverse of Perturbed Identity). If A Cnn and Ap < 1, then

I + A is nonsingular and

1

1

(I + A)1 p

.

1 + Ap

1 Ap

If also Ap 1/2, then (I + A)1 p 2.

i

i

book

2009/5/27

page 39

i

This electronic version is for personal use and may not be duplicated or distributed.

39

Proof. Suppose, to the contrary, that Ap < 1 and I + A is singular. Then there

is a vector x = 0 such that (I + A)x = 0. Hence xp = Axp Ap xp

implies Ap 1, a contradiction.

Lower bound: I = (I + A)(I + A)1 implies

1 = I p I + Ap (I + A)1 p (1 + Ap )(I + A)1 p .

Upper bound: From

ics

follows

he

m

at

at

A1 p

.

1 A1 Ep

nd

(A + E)1 p

pp

lie

and A1 Ep < 1. Then A + E is nonsingular and

la

fo

rI

nd

us

tr

ia

A1 Ep < 1 follows with Fact 2.23 that I + A1 E is nonsingular. Hence A + E

is nonsingular. Its inverse can be written as (A + E)1 = (I + A1 E)1 A1 . Now

take norms and apply Fact 2.23.

The second assertion follows from A1 Ep A1 p Ep 1/2.

So

cie

ty

(A + E)1 p exceeds A1 p by a factor of at most two.

We use the above bounds to derive normwise condition numbers for the

inverses of general nonsingular matrices. A perturbation of a nonsingular matrix

remains nonsingular if the perturbation is small enough in the normwise relative

sense.

Fact 2.25. If A Cnn is nonsingular and A1 Ep < 1, then

(A + E)1 A1 p A1 p

A1 Ep

.

1 A1 Ep

(A + E)1 A1 p

Ep

2p (A)

,

Ap

A1 p

where p (A) = Ap A1 p 1.

i

i

book

2009/5/27

page 40

i

This electronic version is for personal use and may not be duplicated or distributed.

40

we obtain for the absolute difference

(A + E)1 A1 = (I + F )1 A1 A1

= (I + F )1 I A1 = (I + F )1 F A1 ,

where the last equation follows from (I + F )1 (I + F ) = I . Taking norms and

applying the first bound in Fact 2.23 yields

where

at

he

m

If A1 p Ep 1/2, then the second bound in Fact 2.23 implies

ics

F

.

1 F p

at

lie

Ep

Ep

= p (A)

.

Ap

Ap

pp

la

nd

tr

ia

ty

fo

rI

nd

us

small. In particular, the perturbed matrix is nonsingular if the perturbation has small enough norm.

The inverse of A is well-conditioned in the relative sense if p (A) is close

to 1. Note that p (A) 1.

So

cie

Definition 2.27. Let A Cnn be nonsingular. The number p (A) = Ap A1 p

is a normwise relative condition number of A with respect to inversion.

According to Fact 2.25, a perturbed matrix A + E is nonsingular if

A1 Ep < 1. Is this bound pessimistic, or is it tight? Does it imply that if

A1 Ep = 1, then A + E can be singular? The answer is yes. We illustrate

this now for the two norm.

Example 2.28. Let A Cnn be nonsingular. We show how to construct an outer

product E such that A1 E2 = 1 and A + E is singular.

Set E = yx /x22 , where x = 0 and y = 0 are vectors we still need to

choose. Since E is an outer product, Exercise 3 in Section 2.6 implies

A1 E2 =

(A1 y)x 2

A1 y2 x2

A1 y2

=

=

.

2

2

x2

x2

x2

i

i

book

2009/5/27

page 41

i

This electronic version is for personal use and may not be duplicated or distributed.

41

Since (A + E)x = 0 for x = 0, the matrix A + E must be singular.

Therefore, if A is nonsingular, y = 0 is any vector, x = A1 y, and E =

Exercise 3 in Section 2.6 implies that the two norm of the perturbation in

Example 2.28 is E2 = y2 /x2 = y2 /A1 y2 . What is the smallest two

norm a matrix E can have that makes A + E singular? We show that the smallest

norm such an E can have is equal to 1/A1 2 .

he

m

at

ics

Then

1

min {E2 : A + E is singular} =

.

A1 2

la

nd

pp

lie

at

Proof. Let E Cnn be any matrix such that A + E is singular. Then there is a

vector x = 0 so that (A+E)x = 0. Hence x2 = A1 Ex2 A1 2 E2 x2

implies E2 1/A1 2 . Since this is true for any E that makes A + E singular,

1/A1 2 is a lower bound for the absolute distance of A to singularity.

Now we show that there is a matrix E0 that achieves equality. Construct E0

as in Example 2.28, and choose the vector y such that A1 2 = A1 y2 and

y2 = 1. Then E0 2 = y2 A1 y2 = 1/A1 2 .

nd

us

tr

ia

Then

E2

1

min

: A + E is singular =

,

A2

2 (A)

fo

rI

So

cie

ty

Therefore, matrices that are ill-conditioned with respect to inversion are close

to singular, and vice versa. In other words, matrices that are close to being singular

have sensitive inverses.

The example below illustrates that absolute and relative distance to singularity are not the same.

Example. Just because a matrix is close to singularity in the absolute sense does

not imply that it is also close to singularity in the relative sense. To see this, let

1

1

1

A=

,

0 < 1,

A =

.

0

0 1

Exercise 2 in Section 2.6 implies for an n n matrix B that B2 n maxij |bij |.

Hence A2 2 and 1 A1 2 2 . Therefore,

1

,

1

2 A 2

1

1

1,

4 2 (A)

i

i

42

book

2009/5/27

page 42

i

This electronic version is for personal use and may not be duplicated or distributed.

so that A is close to singularity in the absolute sense, but far from singularity in

the relative sense.

Exercises

he

m

at

ics

(ii) Let A, B Cnn be nonsingular. Show: p (AB) p (A)p (B).

(iii) Residuals for Matrix Inversion.

Let A, A + E Cnn be nonsingular, and let Z = (A + E)1 . Show:

at

respect to inversion, in the normwise absolute and relative sense.

Show: If A Cnn and Ap < 1, then

lie

Ap

,

1 Ap

pp

(I + A)1 I p

la

nd

us

tr

ia

Let A Cnn and Ap 1/2. Show:

rI

nd

(A + E)1 A1 p

Ep

p (A)

.

1

Ap

(A + E) p

So

cie

ty

fo

3. One can also bound the relative error with regard to (A + E)1 .

Let A and A + E be nonsingular. Show:

n

1 j n.

i=1,i=j

5. Let A Cnn be nonsingular. Show: p (A) Ap /A Bp for any

singular matrix B Cnn .

i

i

book

2009/5/27

page 43

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

3. Linear Systems

nd

pp

lie

matrix is nonsingular, and we discuss the accuracy of these algorithms.

la

tr

ia

Fact 3.1 (Two Views of a Linear System). Let A Cmn and b Cm1 .

rI

nd

us

1. The linear system Ax = b has a solution if and only if there is a vector x that

solves the m equations

So

cie

ty

where

fo

r1 x = b1 ,

...,

r1

A = ... ,

rm

rm x = bm ,

b1

b = ... .

bm

of the columns of A,

b = a1 x1 + + an xn ,

where

A = a1

...

an ,

x1

..

x = . .

xn

When the matrix is nonsingular, the linear system has a solution for any

right-hand side, and the solution can be represented in terms of the inverse of A.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

43

i

i

i

book

2009/5/27

page 44

i

This electronic version is for personal use and may not be duplicated or distributed.

44

3. Linear Systems

Ax = b has the unique solution x = A1 b for every b Cn .

where

E=

rz

,

z22

at

(A + E)z = b,

he

m

at

ics

Before we discuss algorithms for solving linear systems we need to take into

account, as discussed in Chapter 2, that the matrix and right-hand side may be

contaminated by uncertainties. This means, instead of solving Ax = b, we solve

a perturbed system (A + E)z = b + f . We want to determine how sensitive the

solution is to the perturbations f and E.

Even if we dont know the perturbations E and f , we can estimate them

from the approximate solution z. To this end, define the residual r = Az b. We

can view z as the solution to a system with perturbed right-hand side, Az = b + r.

If z = 0, then we can also view z as the solution to a system with perturbed matrix,

pp

lie

Exercises

cie

ty

fo

rI

nd

us

tr

ia

la

nd

(ii) Determine the solution to Ax = b when A is involutory.

(iii) Let A consist of several columns of a unitary matrix, and let b be such that

the linear system Ax = b has a solution. Determine a solution to Ax = b.

(iv) Let A be idempotent. When does the linear system Ax = b have a solution

for every b?

(v) Let A be a triangular matrix. When does the linear system Ax = b have a

solution for any right-hand side b?

(vi) Let A = uv be an outer product, where u and v are column vectors. For

which b does the linear system Ax = b have a solution?

x1

(vii) Determine a solution to the linear system A B

= 0 when A is nonx2

singular. Is the solution unique?

So

This problem shows how to construct a matrix perturbation from the residual.

Let A Cnn be nonsingular, Ax = b, and z Cn a nonzero approximation

to x. Show that (A + E0 )z = b, where E0 = (b Az)z and z = (z z)1 z ;

and that (A + E)z = b, where E = E0 + G(I zz ) and G Cnn is any

matrix.

2. In Problem 1 above show that, among all matrices F that satisfy(A + F )z = b,

the matrix E0 is one with smallest two norm, i.e., E0 2 F 2 .

3.2

We derive normwise bounds for the conditioning of linear systems. The following

two examples demonstrate that it is not obvious how to estimate the accuracy of

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 45

i

This electronic version is for personal use and may not be duplicated or distributed.

45

that the residual r = Az b may give misleading information about how close z

is to x.

0

at

r = Az b =

. The residual

at

he

m

whose solution x is approximated by z = 2

ics

Example 3.3. We illustrate that a totally wrong approximate solution can have a

small residual norm.

Consider the linear system Ax = b with

1

1

2

1

A=

,

b=

,

0 < 1,

x=

,

1 1+

2+

1

nd

pp

lie

has a small norm, rp = , because is small. This appears to suggest that z

does a good job of solving the linear system. However, comparing z to the exact

solution,

1

zx =

,

1

ia

la

not imply that z is close to x.

us

tr

The same thing can happen even for triangular matrices, as the next example

shows.

rI

nd

1 108

1 + 108

A=

,

b=

,

0 1

1

1

,

1

ty

fo

x=

So

cie

0

z=

,

1 + 108

0

r = Az b =

.

108

As in the previous example, the residual has small norm, i.e., rp = 108 , but z

is totally inaccurate,

1

zx =

.

108

Again, the residual norm is deceptive. It is small even though z is a bad approximation to x.

The bound below explains why inaccurate approximations can have residuals

with small norm.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

46

book

2009/5/27

page 46

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

r = Az b, then

z xp

rp

p (A)

.

xp

Ap xp

at

he

m

rp

A1 p bp rp

z xp

= Ap A1 p

.

1

xp

Ap xp

A bp bp

ics

Proof. If b = 0 and A is nonsingular, then x = 0; see Fact 1.10. The desired bound

follows immediately from the perturbation bound for matrix multiplication: Apply

Fact 2.22 to U = U = A1 , V = b, V = b + r, U = 0, and V = rp /bp to

obtain

nd

pp

lie

at

respect to inversion; see Definition 2.27. The bound in Fact 3.5 implies that the

linear system Ax = b is well-conditioned if p (A) is small. In particular, if p (A)

rp

is also small, then the approximate

is small and the relative residual norm Ap x

p

solution z has a small error (in the normwise relative sense). However, if p (A) is

large, then the linear system is ill-conditioned. We return to Examples 3.3 and 3.4

to illustrate the bound in Fact 3.5.

us

tr

ia

la

1

1

2

A=

,

b=

,

0 < 1,

1 1+

2+

0

1

,

1

with residual

r = Az b =

0

.

ty

fo

rI

nd

and has an approximate solution z = 2

x=

So

cie

The relative error in the infinity norm is z x /x = 1, indicating that z has

no accuracy whatsoever. To see what the bound in Fact 3.5 predicts, we determine

the inverse

1 1 + 1

1

A =

,

1

1

the matrix norms

A = 2 + ,

A1 =

2+

,

(A) =

(2 + )2

,

r = ,

x = 1,

r

.

=

A x

2+

i

i

book

2009/5/27

page 47

i

This electronic version is for personal use and may not be duplicated or distributed.

47

Since (A) 4/, the system Ax = b is ill-conditioned. The bound in Fact 3.5

equals

z x

r

(A)

= 2 + ,

x

A x

and so it correctly predicts the total inaccuracy of z. The small relative residual

norm of about /2 here is deceptive because the linear system is ill-conditioned.

at

pp

lie

at

he

m

1

1 108

1 + 108

A=

,

b=

,

x=

,

1

0 1

1

T

and has an approximate solution z = 0 1 + 108 with residual

0

r = Az b =

.

108

ics

ia

la

nd

The normwise relative error in the infinity norm is z x /x = 1 and indicates that z has no accuracy. From

1 108

1

A =

0

1

ty

fo

rI

nd

us

tr

Note that conditioning of triangular systems cannot be detected by merely looking

at the diagonal elements; the diagonal elements of A are equal to 1 and far from

zero, but nevertheless A is ill-conditioned with respect to inversion.

The relative residual norm is

108

r

=

1016 .

A x

1 + 108

So

cie

z x

r

(A)

= (1 + 108 )108 1,

x

A x

The residual bound below does not require knowledge of the exact solution.

The bound is analogous to the one in Fact 3.5 but bounds the relative error with

regard to the perturbed solution.

Fact 3.7 (Computable Residual Bound). Let A Cnn be nonsingular and

Ax = b. If z = 0 and r = Az b, then

z xp

rp

p (A)

.

zp

Ap zp

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 48

i

This electronic version is for personal use and may not be duplicated or distributed.

48

3. Linear Systems

We will now derive bounds that separate the perturbations in the matrix from

those in the right-hand side. We first present a bound with regard to the relative

error in the perturbed solution because it is easier to derive.

Fact 3.8 (Matrix and Right-Hand Side Perturbation). Let A Cnn be nonsingular and let Ax = b. If (A + E)z = b + f with z = 0, then

z xp

p (A) A + f ,

zp

f =

f p

.

Ap zp

at

Ep

,

Ap

he

m

A =

ics

where

at

Proof. In the bound in Fact 3.7, the residual r accounts for both perturbations, because if (A + E)z = b + f , then r = Az b = f Ez. Replacing

rp Ep zp + f p in Fact 3.7 gives the desired bound.

pp

lie

Below is an analogous bound for the error with regard to the exact solution. In contrast to Fact 3.8, the bound below requires the perturbed matrix to be

nonsingular.

tr

ia

la

nd

Fact 3.9 (Matrix and Right-Hand Side Perturbation). Let A Cnn be nonsingular, and let Ax = b with b = 0. If (A+E)z = b +f with A1 p Ep 1/2,

then

z xp

2p (A) A + f ,

xp

us

where

Ep

,

Ap

rI

nd

A =

f =

f p

.

Ap xp

So

cie

ty

fo

Proof. We could derive the desired bound from the perturbation bound for matrix

multiplication in Fact 2.22 and matrix inversion in Fact 2.25. However, the resulting bound would not be tight, because it does not exploit any relation between

matrix and right-hand side. This is why we start from scratch.

Subtracting (A + E)x = b + Ex from (A + E)z = b + f gives (A + E)

(z x) = f Ex. Corollary 2.24 implies that A + E is nonsingular. Hence we

can write z x = (A + E)1 (Ex + f ). Taking norms and applying Corollary

2.24 yields

z xp 2A1 p (Ep xp + f p ) = 2p (A)(A + f ) xp .

We can simplify the bound in Fact 3.9 and obtain a weaker version.

Corollary 3.10. Let Ax = b with A Cnn nonsingular and b = 0. If (A + E)

z = b + f with A1 p Ep < 1/2, then

z xp

2p (A) (A + b ) ,

xp

where A =

Ep

,

Ap

b =

f p

.

bp

i

i

book

2009/5/27

page 49

i

This electronic version is for personal use and may not be duplicated or distributed.

49

Effect of the Right-Hand Side. So far we have focused almost exclusively on

the effect that the matrix has on the conditioning of the linear system, and we have

ignored the right-hand side. The advantage of this approach is that the resulting

perturbation bounds hold for all right-hand sides. However, the bounds can be too

pessimistic for some right-hand sides, as the following example demonstrates.

108 9

,

z=

1 + 107

0

r = Az b =

.

107

pp

lie

at

he

m

at

ics

Example 3.11. We illustrate that a favorable right-hand side can improve the conditioning of a linear system. Lets change the right-hand side in Example 3.6 and

consider the linear system Ax = b with

1

1 108

1 108

,

b=

,

x=

A=

1

0 1

1

la

nd

the relative error in z is surprisingly small,

tr

ia

z x

10

=

107 .

x

1 108

nd

us

ty

we obtain

r

107

=

,

A x

(108 1)(108 + 1)

fo

rI

(A) = (1 + 108 )2 ,

So

cie

r

108 + 1 7

z x

(A)

= 8

10 107 .

x

A x

10 1

So, what is happening here? Observe that the relative residual norm is extremely

small, Ar

1023 , and that the norms of the matrix and solution are large

x

compared to the norm of the right-hand side; i.e., A x 1016 b = 1.

We can represent this situation by writing the bound in Fact 3.5 as

z x A1 b r

.

x

A1 b b

Because A1 b /A1 b 1, the matrix multiplication of A1 with b

is well-conditioned with regard to changes in b. Hence the linear system Ax = b

is well-conditioned for this very particular right-hand side b.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

50

book

2009/5/27

page 50

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

Exercises

(i) Absolute Residual Bounds.

Let A Cnn be nonsingular, Ax = b, and r = Az b for some z Cn .

Show:

rp

z xp A1 p rp .

Ap

z xp

1 rp

.

p (A) bp

xp

at

he

m

rp

z xp

,

Ap xp

xp

at

ics

Let A Cnn be nonsingular, Ax = b, b = 0, and r = Az b for some

z Cn . Show:

lie

Let A Cnn be nonsingular, Ax = b, b = 0, and r = Az b for some

z Cn . Show:

pp

rp

rp

rp

p (A)

.

Ap xp

bp

Ap xp

ia

la

nd

(iv) If a linear system is well-conditioned, and the relative residual norm is small,

then the approximation has about the same norm as the solution.

Let A Cnn be nonsingular and b = 0. Prove: If

where

= p (A),

us

tr

< 1,

zp

1 + .

xp

ty

fo

rI

nd

then

b Azp

,

bp

So

cie

(v) For this special right-hand side, the linear system is well-conditioned with

regard to changes in the right-hand side.

Let A Cnn be nonsingular, Ax = b, and Az = b + f . Show: If A1 p =

A1 bp /bp , then

z xp

f p

.

xp

bp

1. Let A Cnn be the bidiagonal matrix

1

1

..

A=

.

..

.

1

i

i

book

2009/5/27

page 51

i

This electronic version is for personal use and may not be duplicated or distributed.

51

(a) Show:

(A) =

||+1

||1

(||n 1)

if || = 1,

if || = 1.

2n

at

ics

Ax = en when = 2 and n = 100. How small, approximately, must

the residual norm be so that the normwise relative error bound is less

than .1?

j =

xp 1

ej A p Ap .

|x|j

where

at

b Azp

|zj xj |

,

j

|xj |

bp

he

m

Let A Cnn be nonsingular, b = 0, and Ax = b. Prove: If xj = 0, then

ia

tr

3.3

la

nd

pp

lie

would you expect to be sensitive to perturbations?

3. Condition Estimation.

Let A be nonsingular. Show how to determine a lower bound for p (A) with

one linear system solution involving A.

nd

us

Linear systems with triangular matrices are easy to solve. In the algorithm below

we use the symbol to represent an assignment of a value.

fo

rI

cie

ty

Output: x = A1 b

So

1. If n = 1, then x b/A.

2. If n > 1, partition

A=

n1

1

n1

A

0

1

a

,

ann

x=

n1

1

x

,

xn

b=

n1

1

b

.

bn

(ii) Repeat the process on the smaller system A x = b xn a.

The process of solving an upper triangular system is also called backsubstitution, and the process of solving a lower triangular system is called forward

elimination.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 52

i

This electronic version is for personal use and may not be duplicated or distributed.

52

3. Linear Systems

Exercises

(i) Describe an algorithm to solve a nonsingular lower triangular system.

(ii) Solution of Block Upper Triangular Systems.

Even if A is not triangular, it may have a coarser triangular structure of which

one can take advantage. For instance, let

A11 A12

A=

,

0

A22

at

he

m

at

ics

where A11 and A22 are nonsingular. Show how to solve Ax = b by solving

two smaller systems.

(iii) Conditioning of Triangular Systems.

This problem illustrates that a nonsingular triangular matrix is ill-conditioned

if a diagonal element is small in magnitude compared to the other nonzero

matrix elements.

Let A Cnn be upper triangular and nonsingular. Show:

lie

A

.

min1j n |ajj |

nd

3.4

pp

(A)

rI

nd

us

tr

ia

la

then multiplying by b (likewise, you would not compute 2/4 by first forming 1/4

and then multiplying by 2). It is too expensive and numerically less accurate; see

Exercise 4 below.

A more efficient approach factors A into a product of simpler matrices and

then solves a sequence of simpler linear systems. Examples of such factorizations

include:

So

cie

ty

fo

U is upper triangular.

Cholesky factorization: A = LL (if it exists), where L is lower triangular.

QR factorization: A = QR, where Q is unitary and R is upper triangular. If

A is real, then Q is real orthogonal.

Methods that solve linear systems by first factoring a matrix are called direct

methods. In general, a direct method factors A = S1 S2 (where S stands for

simpler matrix) and then computes the solution x = A1 b = S21 S11 b by solving

two linear systems.

Algorithm 3.2. Direct Method.

Input: Nonsingular matrix A Cnn , vector b Cn

Output: Solution of Ax = b

1. Factor A = S1 S2 .

i

i

book

2009/5/27

page 53

i

This electronic version is for personal use and may not be duplicated or distributed.

53

3. Solve the system S2 x = y.

at

ics

Each step of the above algorithm is itself a computational problem that may

be sensitive to perturbations. We need to make sure that the algorithm does not introduce additional sensitivity by containing unnecessary ill-conditioned steps. For

a direct method, this means that the factors S1 and S2 should be well-conditioned

with respect to inversion. The example below illustrates that this cannot be taken

for granted. That is, even if A is well-conditioned with respect to inversion, S1 or

S2 can be ill-conditioned.

he

m

1

1+

A=

,

b=

,

1 0

1

at

0 < 1/2,

T

1 . The linear system is well-conditioned because

0 1

1

A =

,

(A) = (1 + )2 9/4.

1

nd

la

S2 =

0

1

1

tr

0

,

1

ia

1

S1 = 1

pp

lie

has the solution x = 1

fo

rI

nd

us

compute the factorization and the first linear system solution exactly, i.e.,

1+

A = S1 S2 ,

,

S1 y = b,

y=

1

So

cie

ty

and that we make errors only in the solution of the second system, i.e.,

1

S2 z = y + r 2 =

,

r2 =

.

0

1

Then the computed solution satisfies

0

z=

,

1

z x

= 1.

x

wrongalthough A is very well-conditioned. What happened? The triangular

matrices S1 and S2 contain elements that are much larger in magnitude than the

elements of A,

1+

1

,

S2 = ,

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

A = 1 + ,

S1 =

i

i

54

book

2009/5/27

page 54

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

S11 = S21 =

A1 = 1 + ,

1+ 2

1

(S1 ) =

2,

1+

.

1+

1

2.

2

(S2 ) =

he

m

at

ics

the original linear system Ax = b is well-conditioned, the algorithm contains steps

that are ill-conditioned, namely, the solution of the linear systems S1 y = b and

S2 x = y.

at

We want to avoid methods, like the one above, that factor a well-conditioned

matrix into two ill-conditioned matrices. Such methods are called numerically

unstable.

la

nd

pp

lie

arithmetic if each step in the algorithm is not much worse conditioned than the

original problem.

If an algorithm contains steps that are much worse conditioned than the

original problem, the algorithm is called numerically unstable.

ty

fo

rI

nd

us

tr

ia

this book we do not take into account errors caused by floating arithmetic operations

(analyses that estimate such errors can be rather tedious). However, if a problem

is numerically unstable in exact arithmetic, then it is also numerically unstable in

finite precision arithmetic, so that a distinction is not necessary in this case.

Below we analyze how the conditioning of the factors S1 and S2 affects the

stability of Algorithm 3.2. The bounds are expressed in terms of relative residual

norms from the linear systems.

So

cie

nonsingular, Ax = b, b = 0, and

A + E = S 1 S2 ,

S1 y = b + r1 ,

S 2 z = y + r2 ,

Ep

,

Ap

r1 p

1 =

,

bp

r2 p

2 =

.

yp

A =

z xp

2p (A) (A + 1 + ) ,

xp

condition

i

i

book

2009/5/27

page 55

i

This electronic version is for personal use and may not be duplicated or distributed.

where

=

55

S21 p S11 p

2 (1 + 1 ).

(A + E)1 p

stability

(A + E)z = S1 S2 z = S1 (y + r2 ) = S1 y + S1 r2 = b + r1 + S1 r2 .

he

m

at

ics

The obvious approach would be to apply Fact 3.9 to the perturbed linear system

(A + E)z = b + r1 + S1 r2 . However, the resulting bound would be too pessimistic,

because we did not exploit the relation between the matrix and the right-hand side.

Instead, we can exploit this relation by subtracting (A + E)x = b + Ex to obtain

at

(A + E)(z x) = Ex + r1 + S1 r2 .

lie

pp

nd

la

tr

ia

nd

us

rI

fo

So

cie

ty

We substitute this bound for r2 p into the above bound for z xp ,

z xp Ap xp (A + E)1 p (A + 1 ) + S21 p S11 p 2 (1 + 1 ) .

Factoring out (A + E)1 p and applying Corollary 2.24 gives the desired

bound.

Remark 3.15.

The numerical stability in exact arithmetic of a direct method can be represented by the condition number for multiplying the two matrices S21 and

S11 , see Fact 2.22, since

S21 p S11 p

S21 p S11 p

=

.

(A + E)1 p

S21 S11 p

i

i

book

2009/5/27

page 56

i

This electronic version is for personal use and may not be duplicated or distributed.

56

3. Linear Systems

If S21 p S11 p (A + E)1 p , then the matrix multiplication S21 S11

is well-conditioned. In this case the bound in Fact 3.14 is approximately

2p (A)(A + 1 + 2 (1 + 1 )), and Algorithm 3.2 is numerically stable in

exact arithmetic.

If S21 p S11 p (A + E)1 p , then Algorithm 3.2 is unstable.

Example 3.16. Returning to Example 3.12 we see that

S11 S21

1+

= 2 ,

1

A

ics

r2

= 2.

y

at

(A) = (1 + )2 ,

he

m

Hence the bound in Fact 3.14 equals 2(1 + )3 , and it correctly indicates the inaccuracy of z.

at

The following bound is similar to the one in Fact 3.14, but it bounds the

relative error with regard to the computed solution.

A + E = S1 S2 ,

Ep

,

Ap

r1 p

1 =

,

S1 p yp

r2 p

2 =

,

S2 p zp

nd

pp

A =

la

S 1 y = b + r1 ,

lie

us

tr

ia

S 2 z = y + r2 ,

fo

rI

nd

z xp

p (A) (A + ) ,

zp

So

cie

ty

where

=

condition

S1 p S2 p

(2 + 1 (1 + 2 )) .

Ap

stability

Proof. As in the proof of Fact 3.14 we start by expanding the right-hand side,

(A + E)z = S1 S2 z = S1 (y + r2 ) = S1 y + S1 r2 = b + r1 + S1 r2 .

The residual is r = Az b = Ez + S1 y + S1 r2 = b + r1 + S1 r2 . Take norms and

substitute the expressions for r1 p and r2 p to obtain

rp Ep zp + 1 S1 p yp + 2 S1 p S2 p zp .

To bound yp write y = S2 z r2 , take norms, and replace r2 p = 2 S2 p zp

to get

yp S2 p yp + r2 p = S2 p zp (1 + 2 ).

i

i

book

2009/5/27

page 57

i

This electronic version is for personal use and may not be duplicated or distributed.

57

= Ap zp (A + ).

The relative error bound now follows from Fact 3.7.

ics

S1 p S2 p /Ap . If S1 p S2 Ap , then Algorithm 3.2 is unstable.

he

m

at

Exercises

at

1. The following bound is slightly tighter than the one in Fact 3.14.

Under the conditions of Fact 3.14 show that

pp

lie

z xp

2p (A) A + p (A, b) ,

xp

S21 p S11 p

2 (1 + 1 ) + 1 .

(A + E)1 p

nd

=

la

bp

,

Ap xp

ia

p (A, b) =

where

nd

us

tr

2. The following bound suggests that Algorithm 3.2 is unstable if the first factor

is ill-conditioned with respect to inversion.

Under the conditions of Fact 3.14 show that

fo

rI

z xp

2p (A) A + 1 + p (S1 ) 2 (1 + 1 ) .

xp

So

cie

ty

3. The following bound suggests that Algorithm 3.2 is unstable if the second

factor is ill-conditioned with respect to inversion.

Let Ax = b where A is nonsingular. Also let

A = S 1 S2 ,

S1 y = b,

S 2 z = y + r2 ,

where

2 =

r2 p

S2 p zp

z xp

p (S2 ) 2 .

zp

4. How Not to Solve Linear Systems.

One could solve a linear system Ax = b by forming A1 , and then multiplying A1 by b. The bound below suggests that this approach is likely to

be numerically less accurate than a direct solver.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 58

i

This electronic version is for personal use and may not be duplicated or distributed.

58

3. Linear Systems

with A1 p Ep 1/2. Compute Z = (A + E)1 and z = Z(b + f ).

Show that

z xp

A1 p bp

p (A) 2

A + f ,

xp

A1 bp

where

f =

f p

,

Ap xp

ics

Ep

,

Ap

at

A =

LU Factorization

3.5

at

he

m

Hint: Use the perturbation bounds for matrix multiplication and matrix inversion in Facts 2.22 and 2.25.

pp

lie

nd

triangular and U is upper triangular, is called an LU factorization of A.

us

tr

ia

la

Exercise 5 in Section 1.13. Unfortunately, there are matrices that do not have an

LU factorization, as the example below illustrates.

1

0

fo

rI

nd

0

A=

1

So

cie

ty

triangular. Suppose to the contrary that it could. Then

0 1

1 0 u 1 u1

=

.

1 0

l 1

0 u3

The first column of the equality implies that u1 = 0, and lu1 = 1 so u1 = 0, a

contradiction.

Example 3.12 illustrates that a matrix A that is well-conditioned with respect to inversion can have LU factors that are ill-conditioned with respect to

inversion. Algorithm 3.3 below shows how to permute the rows of a nonsingular matrix so that the permuted matrix has an LU factorization. Permuting the

rows of A is called partial pivotingas opposed to complete pivoting where both

rows and columns are permuted. In order to prevent the factors from being too

ill-conditioned, Algorithm 3.3 chooses a permutation matrix so that the elements

of L are bounded.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 59

i

This electronic version is for personal use and may not be duplicated or distributed.

3.5. LU Factorization

59

Input: Nonsingular matrix A Cnn

Output: Permutation matrix P , unit lower triangular matrix L,

upper triangular matrix U such that P A = LU

1. If n = 1, then P 1, L 1, and U A.

2. If n > 1, then choose a permutation matrix Pn such that

ics

at

1

n1

n1

a

,

An1

he

m

Pn A =

at

where has the largest magnitude among all elements in the leading column,

i.e., || d , and factor

a

1

0

,

Pn A =

0 S

l In1

us

tr

ia

la

nd

pp

lie

3. Compute Pn1 S = Ln1 Un1 , where Pn1 is a permutation matrix, Ln1 is

unit lower triangular, and Un1 is upper triangular.

4. Then

a

1

0

1

0

Pn ,

,

U

.

L

P

0 Un1

Pn1 l Ln1

0 Pn1

nd

Remark 3.20.

So

cie

ty

fo

rI

one row of U .

Partial pivoting ensures that the magnitude of the multipliers is bounded by

one; i.e., l 1 in step 2 of Algorithm 3.3. Therefore, all elements of L

have magnitude less than or equal to one.

The scalar is called a pivot, and the matrix S = An1 d 1 a is a Schur

complement. We already encountered Schur complements in Fact 1.14, as

part of the inverse of a partitioned matrix. In this particular Schur complement S the matrix d 1 a is an outer product.

The multipliers can be easily recovered from L, because they are elements

of L. Step 4 of Algorithm 3.3 shows that the first column of L contains

the multipliers Pn1 l that zero out elements in the first column. Similarly,

column i of L contains the multipliers that zero out elements in column i.

However, the multipliers cannot be easily recovered from L1 .

T L

Step 4 of Algorithm 3.3 follows from S = Pn1

n1 Un1 , extracting the

permutation matrix,

1

0

1

0

a

Pn A =

T

Pn1 l In1

0 Ln1 Un1

0 Pn1

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

60

book

2009/5/27

page 60

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

a

1

0

1

0

=

0 Ln1 Un1

Pn1 l Ln1

0

Pn1 l In1

a

Un1

ics

In the vector Pn1 l, the permutation Pn1 reorders the multipliers l, but

does not change their values. To combine all permutations into a single

permutation matrix P , we have to pull all permutation matrices in front of

the lower triangular matrix. This, in turn, requires reordering the multipliers

in earlier steps.

at

he

m

at

Fact 3.21 (LU Factorization with Partial Pivoting). Every nonsingular matrix

A has a factorization P A = LU , where P is a permutation matrix, L is unit lower

triangular, and U is nonsingular upper triangular.

nd

pp

lie

choices for the permutation matrix.

With a factorization P A = LU , the rows of the linear system Ax = b are

rearranged, and the system to be solved is P Ax = P b. The process of solving this

linear system is called Gaussian elimination with partial pivoting.

la

tr

ia

Output: Solution of Ax = b

fo

rI

nd

us

2. Solve the system Ly = P b.

3. Solve the system U x = y.

So

cie

ty

The next bound implies that Gaussian elimination with partial pivoting is

stable in exact arithmetic if the elements of U are not much larger in magnitude

than those of A.

Corollary 3.22 (Stability in Exact Arithmetic of Gaussian Elimination with

Partial Pivoting). If A Cnn is nonsingular, Ax = b, and

P (A + E) = LU ,

Ly = P b + rL ,

U z = y + rU ,

E

,

A

rL

L =

,

L y

rU

U =

,

U z

A =

z x

(A) (A + ) ,

z

where = n

U

U + L (1 + U ) .

A

i

i

book

2009/5/27

page 61

i

This electronic version is for personal use and may not be duplicated or distributed.

3.5. LU Factorization

61

Proof. Apply Fact 3.17 to A + E = S1 S2 , where S1 = P T L and S2 = U . Permutation matrices do not change p-norms, see Exercise (iv) in Section 2.6, so that

P T L = L . Because the multipliers are the elements of L, and |lij | 1

with partial pivoting, we get L n.

ics

The ratio U /A represents the element growth during Gaussian elimination. In practice, U /A tends to be small, but there are n n matrices for which U /A = 2n1 /n is possible; see Exercise 2 below. If

U A , then Gaussian elimination is unstable.

he

m

at

Exercises

0

,

0

lie

0

A1

pp

A=

at

Express the elements of L and U in terms of the elements of A.

(ii) Determine a factorization A = LU when A is upper triangular.

(iii) For

A11

A=

A21

A12

,

A22

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

lower triangular and U is upper triangular.

(iv) LDU Factorization.

One can make an LU factorization more symmetric by requiring that both

triangular matrices have ones on the diagonal and factoring A = LD U , where

L is unit lower triangular, D is diagonal, and U is unit upper triangular.

Given an LU factorization A = LU , express the diagonal elements dii of D

and the elements u ij in terms of elements of U .

(v) Block LU Factorization.

Suppose we can partition the invertible matrix A as

where

I

0

A11 A12

L=

,

,

U=

0

S

A21 A1

I

11

11 A12 is a Schur complement. Note that L is unit lower

triangular. However, U is only block upper triangular, because A11 and S

are in general not triangular. Hence a block LU factorization is not the same

as an LU factorization.

Determine a block LDU factorization A = LDU , where L is unit lower

triangular, U is unit upper triangular, and D is block diagonal.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

62

book

2009/5/27

page 62

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

0

1

A=

1

3

1

0

2

4

1

3

1

3

2

4

2

4

ics

factorization A = LU with

0 1

.

A11 =

1 0

at

he

m

at

Determine L and U .

(vii) UL Factorization.

Analogous to Algorithm 3.3, present an algorithm that factors any square

matrix A into P A = U L, where P is a permutation matrix, U is unit upper

triangular, and L is lower triangular.

pp

lie

A11 A12

PA =

A21 A22

us

tr

ia

la

nd

11 are less than one in

magnitude, then

2

A

A22 A21 A1

11 12 n (A).

nd

1

1

...

1

..

.

...

..

.

1

1

1

1

.

..

.

1

So

cie

ty

fo

rI

1

1

1

A=

.

..

Show that pivoting is not necessary. Determine the one norms of A and U .

3. Let A Cnn and A + uv be nonsingular, where u, v Cn . Show how

to solve (A + uv )x = b using two linear system solves with A, two inner

products, one scalar vector multiplication, and one vector addition.

4. This problem shows that if Gaussian elimination with partial pivoting encounters a small pivot, then A must be ill-conditioned.

Let A Cnn be nonsingular and P A = LU , where P is a permutation

matrix, L is unit triangular with elements |lij | 1, and U is upper triangular

with elements uij . Show that (A) A / minj |ujj |.

5. The following matrices G are generalizations of the lower triangular matrices

in the LU factorization. The purpose of G is to transform all elements of a

column vector into zeros, except for the kth element.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 63

i

This electronic version is for personal use and may not be duplicated or distributed.

63

elements of g have to satisfy so that G is invertible? Determine G1 when

it exists.

Given an index k and a vector x Cn , which conditions do the elements of

x have to satisfy so that Gx = ek ? Determine the vector g when it exists.

3.6

Cholesky Factorization

lie

at

he

m

at

ics

It would seem natural that a Hermitian matrix should have a factorization that

reflects the symmetry of the matrix. For an n n Hermitian matrix, we need to

store only n(n + 1)/2 elements, and it would be efficient if the same were true for

the factorization. Unfortunately, this is not possible in general. For instance, the

matrix

0 1

A=

1 0

la

nd

pp

is nonsingular and Hermitian. But it cannot be factored into a lower times upper

triangular matrix, as illustrated in Example 3.19. Fortunately, a certain class of

matrices, so-called Hermitian positive definite matrices, do admit a symmetric

factorization.

rI

nd

us

tr

ia

all x Cn with x = 0.

A Hermitian matrix A Cnn is positive semidefinite if x Ax 0 for all

n

xC .

A symmetric matrix A Rnn is positive definite if x T Ax > 0 for all x Rn

with x = 0, and positive semidefinite if x T Ax 0 for all x Rn .

ty

fo

So

cie

A=

We derive several properties of Hermitian positive definite matrices. We

start by showing that all Hermitian positive definite matrices are nonsingular.

Fact 3.24. If A Cnn is Hermitian positive definite, then A is nonsingular.

Proof. Suppose to the contrary that A were singular. Then Ax = 0 for some x = 0,

implying x Ax = 0 for some x = 0, which contradicts the positive definiteness of

A; i.e., x Ax > 0 for all x = 0.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

64

book

2009/5/27

page 64

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

Fact 3.25. If A Cnn is Hermitian positive definite, then its diagonal elements

are positive.

Proof. Since A is positive definite, we have x Ax > 0 for any x = 0, and in

particular 0 < ej Aej = ajj , 1 j n.

Below is a transformation that preserves Hermitian positive definiteness.

at

ics

then B AB is also Hermitian positive definite.

lie

at

he

m

Proof. The matrix B AB is Hermitian because A is Hermitian. Since B is nonsingular, y = Bx = 0 if and only if x = 0. Hence

pp

Hermitian positive definiteness.

la

nd

Fact 3.27. If A Cnn is Hermitian positive definite, then its leading principal

submatrices and Schur complements are also Hermitian positive definite.

ty

fo

rI

nd

us

tr

ia

submatrix B is Hermitian because it is a principal submatrix of a Hermitian matrix.

To keep the notation simple, we permute the rows and columns of A so that the

submatrix B occupies the leading rows and columns. That is, let P be a permutation

matrix, and partition

B A12

T

A = P AP =

.

A12 A22

So

cie

> 0 for

Fact 3.26 implies that A is also Hermitian

positive definite. Thus x Ax

y

any vector x = 0. In particular, let x =

for y Ck . Then for any y = 0 we

0

have

B A12

y

0

0 < x Ax = y

= y By.

0

A12 A22

This means y By > 0 for y = 0, so that B is positive definite. Since the submatrix B

is a principal submatrix of a Hermitian matrix, B is also Hermitian. Therefore,

any principal submatrix B of A is Hermitian positive definite.

Now we prove Hermitian positive definiteness for Schur complements.

Fact 3.24 implies that B is nonsingular. Hence we can set

Ik

0

L=

,

A12 B 1 Ink

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 65

i

This electronic version is for personal use and may not be duplicated or distributed.

so that

B

LAL =

0

65

0

,

S

where

Since L is unit lower triangular, it is nonsingular. From Fact 3.26 follows then that

is Hermitian positive definite. Earlier in this proof we showed that principal

LAL

submatrices of Hermitian positive definite matrices are Hermitian positive definite,

thus the Schur complement S must be Hermitian positive definite.

at

he

m

at

ics

Now we have all the tools we need to factor Hermitian positive definite

matrices. The following algorithm produces a symmetric factorization A = LL

for a Hermitian positive definite matrix A. The algorithm exploits the fact that

the diagonal elements of A are positive and the Schur complements are Hermitian

positive definite.

lie

A = LL , where L is (lower or upper) triangular with positive diagonal elements,

is called a Cholesky factorization of A.

nd

pp

is a lower triangular matrix.

la

rI

nd

us

tr

ia

Output: Lower triangular matrix L with positive diagonal elements

such that A = LL

1. If n = 1, then L A.

2. If n > 1, partition and factor

1

fo

cie

ty

1

A=

n1

n1

1/2

a

=

An1

a 1/2

1

0

In1

0

0

S

1/2

1/2 a

,

In1

So

where S An1 a 1 a .

3. Compute S = Ln1 Ln1 , where Ln1 is lower triangular with positive diagonal elements.

4. Then

1/2

0

L

.

a 1/2 Ln1

A Cholesky factorization of a positive matrix is unique.

Fact 3.29 (Uniqueness of Cholesky factorization). Let A Cnn be Hermitian

positive definite. If A = LL where L is lower triangular with positive diagonal,

then L is unique. Similarly, if A = LL where L is upper triangular with positive

diagonal elements, then L is unique.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

66

book

2009/5/27

page 66

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

Proof. This can be shown in the same way as the uniqueness of the LU factorization.

The following result shows that one can use a Cholesky factorization to

determine whether a Hermitian matrix is positive definite.

Fact 3.30. Let A Cnn be Hermitian. A is positive definite if and only if A =

LL where L is triangular with positive diagonal elements.

he

m

at

ics

assume that A = LL . Since L is triangular with positive diagonal elements, it is

nonsingular. Therefore, Lx = 0 for x = 0, and x Ax = L x22 > 0.

at

The next bound shows that a Cholesky solver is numerically stable in exact

arithmetic.

A + E = LL ,

pp

us

tr

ia

la

L z = y + r2 ,

E2

,

A2

r1 2

1 =

,

b2

r2 2

2 =

.

y2

A =

nd

Ly = b + r1 ,

lie

Hermitian positive definite matrices, Ax = b, b = 0, and

fo

rI

nd

z x2

22 (A) (A + 1 + 2 (1 + 1 )) .

x2

cie

ty

is L 2 L1 2 /(A + E)1 2 = 1 because Fact 2.19 implies

So

Exercises

(i) The magnitude of an off-diagonal element of a Hermitian positive definite

matrix is bounded by the geometric mean of the corresponding diagonal

elements.

Let A Cnn be Hermitian positive definite. Show: |aij | < aii ajj for

i = j .

Hint: Use the positive definiteness of the Schur complement.

(ii) The magnitude of an off-diagonal element of a Hermitian positive definite

matrix is bounded by the arithmetic mean of the corresponding diagonal

elements.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 67

i

This electronic version is for personal use and may not be duplicated or distributed.

(iv)

nd

pp

lie

(vii)

at

(vi)

he

m

at

(v)

Let A Cnn be Hermitian positive definite. Show: |aij | (aii + ajj )/2

for i = j .

Hint: Use the relation between arithmetic and geometric mean.

The largest element in magnitude of a Hermitian positive definite matrix is

on the diagonal.

Let A Cnn be Hermitian positive definite. Show: max1i,j n |aij | =

max1in aii .

Let A Cnn be Hermitian positive definite. Show: A1 is also positive

definite.

Modify Algorithm 3.5 so that it computes a factorization A = LDL for a

Hermitian positive definite matrix A, where D is diagonal and L is unit lower

triangular.

Upper-Lower Cholesky Factorization.

Modify Algorithm 3.5 so that it computes a factorization A = L L for a Hermitian positive definite matrix A, where L is lower triangular with positive

diagonal elements.

Block Cholesky Factorization.

Partition the Hermitian positive definite matrix A as

A11 A12

.

A=

A21 A22

ics

(iii)

67

nd

us

tr

ia

la

Analogous to the block LU factorization in Exercise (v) of Section 3.5 determine a factorization A = LL , where L is block lower triangular. That is,

L is of the form

0

L11

,

L=

L21 L22

cie

ty

fo

rI

(viii) Let

A11 A12

A=

A21 A22

So

A22 A21 A1

11 A12 2 A2

and

2 (A22 A21 A1

11 A12 ) 2 (A).

(ix) Prove: A = MM for some nonsingular matrix M if and only if A is Hermitian positive definite.

(x) Generalized Cholesky Factorization.

Let M Cnn be Hermitian positive definite. Prove: If M = M1 M1 =

M2 M2 , for square matrices M1 and M2 , then there exists a unitary matrix Q

such that M2 = QM1 .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 68

i

This electronic version is for personal use and may not be duplicated or distributed.

68

3. Linear Systems

are real square matrices. Show that the matrix

A B

C=

B A

is real symmetric positive definite.

QR Factorization

ics

3.7

he

m

at

and the other one is triangular. We derive the existence of a QR factorization from

the Cholesky factorization.

at

QR, where Q is unitary and R is upper triangular with positive diagonal elements.

tr

ia

la

nd

pp

lie

which implies that M = A A is Hermitian positive definite. Let M = LL be a

Cholesky factorization of M, where L is lower triangular with positive diagonal

elements. Then M = A A = LL . Multiplying by A on the left gives A = QR,

where Q = A L, and where R = L is upper triangular with positive diagonal

elements. Exercise (ix) in Section 3.6 shows that Q is unitary.

The uniqueness of the QR factorization follows from the uniqueness of the

Cholesky factorization, as well as from Exercise 6 in Section 1.13.

nd

us

arithmetic.

So

cie

ty

fo

rI

b = 0, and

A + E = QR,

Qy = b + r1 ,

Rz = y + r2 ,

E2

,

A2

r1 2

1 =

,

b2

r2 2

2 =

.

y2

A =

z x2

22 (A) (A + 1 + 2 (1 + 1 )) .

x2

Proof. Apply Fact 3.14 to A + E, where S1 = Q and S2 = R. The stability factor

is R 1 2 Q 2 /(A + E)1 2 = 1, because Exercise (v) in Section 2.6 implies

Q 2 = 1 and (A + E)1 2 = R 1 2 .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 69

i

This electronic version is for personal use and may not be duplicated or distributed.

3.7. QR Factorization

69

algorithm that is based on Givens rotations; see Definition 1.17. Givens rotations

are unitary, see Example 1.16, and they are often used to introduce zeros into

matrices. Lets start by using a Givens rotation to introduce a single zero into a

vector.

Example. Let x, y C.

x

d

c s

=

,

0

s c y

d=

|x|2 + |y|2 .

ics

where

at

he

m

at

both components of the vector are zero, then there is nothing to do and the unitary

matrix is the identity. Note that d 0 and |c|2 + |s|2 = 1.

When introducing zeros into a longer vector, we embed each Givens rotation

in an identity matrix.

pp

lie

a unitary matrix. We can apply three Givens rotations in the following order.

x1

x1

0

0 x2 x 2

=

,

s 4 x3 y 3

x4

0

c4

tr

ia

la

1 0

0

0

0 1

0 0 c

4

0 0 s 4

nd

So

cie

ty

fo

rI

nd

us

where y3 = |x3 |2 + |x4 |2 0. If x4 = x3 = 0, then c4 = 1 and s4 = 0;

otherwise c4 = x 3 /y3 and s4 = x 4 /y3 .

2. Apply a Givens rotation to rows 2 and 3 to zero out element 3,

1

0

0 c3

0 s

3

0

0

0

s3

c3

0

x1

x1

0

0 x2 y2

=

,

0 y3 0

0

1

0

where y2 = |x2 |2 + |y3 |2 0. If y3 = x2 = 0, then c3 = 1 and s3 = 0;

otherwise c3 = x 2 /y2 and s3 = y 3 /y2 .

3. Apply a Givens rotation to rows 1 and 2 to zero out element 2,

c2

s 2

0

0

s2

c2

0

0

0 0

y1

x1

0 0 y2 0

,

=

1 0 0 0

0

0

0 1

where y1 = |x1 |2 + |y2 |2 0. If y2 = x1 = 0, then c2 = 1 and s2 = 0;

otherwise c2 = x 1 /y1 and s2 = y 2 /y1 .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

70

book

2009/5/27

page 70

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

s2

c2

0

0

c2

s 2

Q=

0

0

0 0

1

0 0 0

1 0 0

0

0 1

0

c3

s 3

0

1 0

0

0 0 1

0 0 0

0 0

1

0

s3

c3

0

0

0

c4

s 4

0

0

.

s4

c4

ics

There are many possible orders in which to apply Givens rotations, and

Givens rotations dont have to operate on adjacent rows either. The example

below illustrates this.

he

m

at

We can apply three Givens rotations that all involve the leading row.

x1

y1

s4

0 x2 x 2

=

,

0 x3 x 3

c4

x4

0

0 0

1 0

0 1

0 0

lie

c4

0

0

s 4

pp

at

0 s3

1 0

0 c3

0 0

nd

us

tr

c3

0

s

3

0

ia

la

nd

where y1 = |x1 |2 + |x4 |2 0. If x4 = x1 = 0, then c4 = 1 and s4 = 0;

otherwise c4 = x 1 /y1 and s4 = x 4 /y1 .

2. Apply a Givens rotation to rows 1 and 3 to zero out element 3,

0

z1

y1

0 x2 x2

,

=

0 x3 0

0

0

1

cie

ty

fo

rI

where z1 = |y1 |2 + |x3 |2 0. If x3 = y1 = 0, then c3 = 1 and s3 = 0;

otherwise c3 = y 1 /z1 and s3 = x 3 /z1 .

3. Apply a Givens rotation to rows 1 and 2 to zero out element 2,

So

c2

s 2

0

0

s2

c2

0

0

x1

0 0

u1

0 0 y2 0

,

=

1 0 0 0

0 1

0

0

where u1 = |z1 |2 + |x2 |2 0. If x2 = z1 = 0, then c2 = 1 and s2 = 0;

otherwise c2 = z1 /u1 and s2 = x 2 /u1 .

Therefore Qx = u1 e1 , where u1 = Qx2 and

c2

s 2

Q=

0

0

s2

c2

0

0

0 0

c3

0 0 0

1 0 s 3

0 1

0

0 s3

1 0

0 c3

0 0

0

c4

0 0

0 0

s 4

1

0 0

1 0

0 1

0 0

s4

0

.

0

c4

i

i

book

2009/5/27

page 71

i

This electronic version is for personal use and may not be duplicated or distributed.

3.7. QR Factorization

71

rows i and j , then the c and s elements occupy positions (i, i), (i, j ), (j , i), and

(j , j ).

At last here is a sketch of how one can reduce a square matrix to upper

triangular form by means of Givens rotations.

at

he

m

at

ics

Example. We introduce zeros one column at a time, from left to right, and within

a column from bottom to top. The Givens rotations operate on adjacent rows.

Elements that can be nonzero are represented by . Elements that were affected

by the ith Givens rotation have the label i. We start by introducing zeros into

column 1,

3 3 3 3

1 2 2 2 2 2 3 0 3 3 3

1 1 1 1 0 2 2 2 0 2 2 2 .

0 1 1 1

0 1 1 1

0 1 1 1

3

5

6

0

3

5

.

6

6

la

3

5

0

0

nd

pp

lie

3 3 3 3

3 3 3 3

3

3 3 3 3

0 3 3 3 4 0 3 3 3 5 0 5 5 5 6 0

0 2 2 2 0 4 4 4 0 0 5 5 0

0 0 4 4

0 0 4 4

0

0 1 1 1

ia

nd

us

tr

Output: Unitary matrix Q Cnn and upper triangular matrix R

Cnn with positive diagonal elements such that A = QR

So

cie

ty

fo

rI

2. If n > 1, zero out elements n, n 1, . . . , 2 in column 1 of A as follows.

(i) Set bn1 bn2 . . . bnn = an1 an2 . . . ann .

(ii) For i = n, n 1, . . . , 2

Zero out element (i, 1) by applying a rotation to rows i and i 1,

ai1,1 ai1,2 . . . ai1,n

si

ci

s i ci

bi1

bi2

...

bin

b

bi1,2 . . . bi1,n

= i1,1

,

0

a i2

...

a in

where bi1,1 |bi1 |2 + |ai1,1 |2 . If bi1 = ai1,1 = 0, then ci 1

and si 0; otherwise ci a i1,1 /bi1,1 and si bi1 /bi1,1 .

(iii) Multiply all n 1 rotations,

In2

0

0

0

c 2 s2

c n sn .

0 0

Qn s 2 c2

0

s n cn

0

0 In2

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

72

book

2009/5/27

page 72

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

r11 r

,

where

Qn A =

0 A

r

b12

a 22

..

A .

...

a n2

a 2n

.. ,

.

. . . a nn

...

he

m

at

ics

3. Compute A = Qn1 Rn1 , where Qn1 is unitary and Rn1 is upper triangular with positive diagonal elements.

4. Then

1

0

r

r

Q Qn

,

R 11

.

0 Qn1

0 Rn1

at

Exercises

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

(ii) QR Factorization of Outer Product.

Let x, y Cn , and apply Algorithm 3.6 to xy . How many Givens rotations

do you have to apply at the most? What does the upper triangular matrix R

look like?

(iii) Let A Cnn be a tridiagonal matrix, that is, only elements aii , ai+1,i , and

ai,i+1 can be nonzero; all other elements are zero. We want to compute a

QR factorization A = QR with n 1 Givens rotations. In which order do

the elements have to be zeroed out, on which rows do the rotations act, and

which elements of R can be nonzero?

(iv) QL Factorization.

Show: Every nonsingular matrix A Cnn has a unique factorization

A = QL, where Q is unitary and L is lower triangular with positive diagonal elements.

(v) Computation of QL Factorization.

Suppose we want to compute the QL factorization of a nonsingular matrix

A Cnn with Givens rotations. In which order do the elements have to be

zeroed out, and on which rows do the rotations act?

(vi) The elements in a Givens rotation

c s

G=

s c

are named to invoke an association with sine and cosine, because

|c|2 + |s|2 = 1. One can also express the elements in terms of tangents

or cotangents. Let

x

d

G

=

,

where d = |x|2 + |y|2 .

y

0

Show the following: If |y| > |x|, then

=

x

,

y

s=

y

1

,

|y| 1 + | |2

c = s,

i

i

book

2009/5/27

page 73

i

This electronic version is for personal use and may not be duplicated or distributed.

73

=

y

,

x

c=

x

1

,

|x| 1 + | |2

s = c.

at

3.8

he

m

at

ics

Here is another way to introduce zeros into a vector without changing

its two norm. Let x Cn and x1 = 0. Define Q = I 2vv /v v, where

v = x + x2 e1 and = x1 /|x1 |. Show that Q is unitary and that

Qx = x2 e1 . The matrix Q is called a Householder reflection.

(viii) Householder Reflections for Real Vectors.

Let x, y Rn with x2 = y2 . Show how to choose a vector v in the

Householder reflection so that Qx = y.

b1

b = b2 ,

b3

ia

la

1

A = 1 ,

1

nd

Example. If

pp

lie

i.e., m n. If A is involved in a linear system Ax = b, then we must have b Cm

and x Cn . Such linear systems do not always have a solution; and if they do

happen to have a solution, then the solution may not be unique.

fo

rI

nd

us

tr

then the linear system Ax = b has a solution only for those b all of whose elements

are the same, i.e., = b1 = b2 = b3 . In this case the solution is x = .

Fortunately, there is one right-hand side for which a linear system Ax = b

always has a solution, namely, b = 0. That is, Ax = 0 always has the solution

x = 0. However, x = 0 may not be the only solution.

ty

Example. If

cie

1

1 ,

1

T

then Ax = 0 has infinitely many solutions x = x1 x2 with x1 = x2 .

We distinguish matrices A where x = 0 is the unique solution for Ax = 0.

So

1

A = 1

1

Ax = 0 implies x = 0. If Ax = 0 has infinitely many solutions, then the columns

of A are linearly dependent.

Example.

The columns of a nonsingular matrix

Aare linearly independent.

A

If A is nonsingular, then the matrix

has linearly independent columns.

0

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

i

book

2009/5/27

page 74

i

This electronic version is for personal use and may not be duplicated or distributed.

74

3. Linear Systems

column. If x = 0, then x is linearly dependent.

If A Cmn with A A = In , then A has linearly independent columns. This

is because multiplying Ax = 0 on the left by A implies x = 0.

A b

If the linear system Ax = b has a solution x, then

the matrix

B

=

x

has linearly dependent columns. That is because B

= 0.

1

he

m

at

ics

How can we tell whether a tall and skinny matrix has linearly independent

columns? We can use a QR factorization.

lie

at

Output: Unitary matrix Q Cmm and upper triangular matrixR

R

Cnn with nonnegative diagonal elements such that A = Q

0

tr

ia

la

nd

pp

and R A2 .

2. If n > 1, then, as in Algorithm 3.6, determine a unitary matrix Qm Cmm

to zero out elements 2, . . . , m in column 1 of A, so that

r11 r

,

Qm A =

0 A

cie

ty

fo

rI

nd

us

C(m1)(n1)

.

R

n1

3. Compute A = Qm1

, where Qm1 C(m1)(m1) is unitary, and

0

Rn1 C(n1)(n1) is upper triangular with nonnegative diagonal elements.

4. Then

1

0

r

r

Q Qm

,

R 11

.

0 Qm1

0 Rn1

So

R

where Q Cmm is uni0

tary, and R Cnn is upper triangular. Then A has linearly independent columns

if and only if R has nonzero diagonal elements.

Fact 3.35. Let A Cmn with m n, and A = Q

R

0 x = 0 if and only if

0

Rx = 0 x = 0. This is the case if and only if R is nonsingular and has nonzero

diagonal elements.

Proof.

Since Q is nonsingular, Ax = Q

and omitting part of the unitary matrix.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 75

i

This electronic version is for personal use and may not be duplicated or distributed.

75

a matrix Q1 Cmn with Q1 Q1 = In , and an upper triangular matrix R Cnn

with nonnegative diagonal elements so that A = Q1 R.

R

Proof. Let A = Q

be a QR factorization as in Fact 3.35.

0

Partition

at

ics

he

m

decomposition.

pp

lie

at

orthonormal because A A = In , and so are the rows because AA = In . This

means, a square matrix with orthonormal columns must be a unitary matrix. A real

square matrix with orthonormal columns is an orthogonal matrix.

Exercises

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

A2 = R2 .

(ii) Uniqueness of Thin QR Factorization.

Let A Cmn have linearly independent columns. Show: If A = QR,

where Q Cmn satisfies Q Q = In and R is upper triangular with positive

diagonal elements, then Q and R are unique.

(iii) Generalization of Fact 3.35.

C

mn

Let A C

, m n, and A = B

, where B Cmn has linearly

0

independent columns, and C Cnn . Show: A has linearly independent

columns if and only if C is nonsingular.

(iv) Let A Cmn where m > n. Show: There exists a matrix Z Cm(mn)

such that Z A = 0.

(v) Let A Cmn , m n, have a thin QR factorization A = QR. Express the

kth column of A as a linear combination of columns of Q and elements of R.

How many columns of Q are involved?

(vi) Let A Cmn , m n, have a thin QR factorization A = QR. Determine a

QR factorization of A Qe1 e1 R from the QR factorization of A.

aj , 1

j n.

Let A = QR be a thin QR factorization where Q = q1 . . . qn and R is

upper triangular with positive diagonal elements. Express the elements of R

in terms of the columns aj of A and the columns qj of Q.

(viii) Let A be a matrix with linearly independent columns. Show how to compute the lower-upper Cholesky factorization of A A without forming the

product A A.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

76

book

2009/5/27

page 76

i

This electronic version is for personal use and may not be duplicated or distributed.

3. Linear Systems

Let V Cmn with V = v1

x Cm . Show:

...

n

|vj x|2 x x.

j =1

he

m

at

ics

This problem presents a method to compute QR factorizations of arbitrary

matrices. Let A Cmn with rank(A) = r. Then there exists a permutation

matrix P so that

R R2

AP = Q 1

,

0

0

Show how to modify Algorithm 3.7 so that it computes such a factorization. In the first step, choose a permutation matrix Pn that brings

the column with largest two norm to the front; i.e.,

lie

(a)

at

pp

1j n

nd

i.e., (R1 )11 (R1 )22 (R1 )rr .

So

cie

ty

fo

rI

nd

us

tr

ia

la

(b)

i

i

book

2009/5/27

page 77

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

4. Singular Value

Decomposition

nd

pp

lie

introduce the singular value decomposition. It is one of the most important tools

in numerical linear algebra, because it contains a lot of information about a matrix,

including rank, distance to singularity, column space, row space, and null spaces.

rI

nd

us

tr

ia

la

(SVD) of A is a decomposition

1

..

A=U

V ,

where

=

, 1 n 0,

.

0

n

fo

If m n, then an SVD of A is

A=U

0 V ,

where

=

So

cie

ty

..

1 m 0,

The matrix U is called a left singular vector matrix, V is called a right

singular vector matrix, and the scalars j are called singular values.

Remark 4.2.

An m n matrix has min{m, n} singular values.

The singular values are unique, but the singular vector matrices are not.

Although an SVD is not unique, one often says the SVD instead of an

SVD.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

77

i

i

i

78

book

2009/5/27

page 78

i

This electronic version is for personal use and may not be duplicated or distributed.

Let A Cmn with m n. If A = U

V is an SVD of A, then

0

A = V

0 U is an SVD of A . Therefore, A and A have the same

singular values.

A Cnn is nonsingular if and only if all singular values are nonzero, i.e.,

j > 0, 1 j n.

If A = U

V is an SVD of A, then A1 = V

1 U is an SVD of A1 .

at

he

m

1

A=

0

ics

lie

2 + ||2 + || 4 + ||2

2 =

1/2

at

nd

pp

absolute distance of A to singularity decreases.

la

Exercises

So

cie

ty

fo

rI

nd

us

tr

ia

(i) Let A Cnn . Show: All singular values of A are the same if and only if A

is a multiple of a unitary matrix.

(ii) Show that the singular values of a Hermitian idempotent matrix are 0 and 1.

(iii) Show: A Cnn is Hermitian positive definite if and only if it has an SVD

A = V

V where

is nonsingular.

(iv) Let A, B Cmn . Show: A and B have the same singular values if and only

if there exist unitary matrices Q Cnn and P Cmm such that B = P AQ.

R

(v) Let A Cmn , m n, with QR decomposition A = Q

, where

0

Q Cmm is unitary and R Cnn . Determine an SVD of A from an

SVD of R.

(vi) Determine an SVD of a column vector, and an SVD of a row vector.

(vii) Let A Cmn with m n. Show: The singular values of A A are the

squares of the singular values of A.

1. Show: If A Cnn is Hermitian positive definite and > n , then A + In

is also Hermitian positive definite with singular values j + .

2. Let A Cmn and > 0. Express the singular values of (A A + I )1 A

in terms of and the singular values of A.

I

mn

3. Let

with m n. Show: The singular values of n are equal to

AC

A

1 + j2 , 1 j n.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 79

i

This electronic version is for personal use and may not be duplicated or distributed.

4.1

79

The smallest and largest singular values of a matrix provide information about the

two norm of the matrix, the distance to singularity, and the two norm of the inverse.

Fact 4.4 (Extreme Singular Values). If A Cmn

1 p , where p = min{m, n}, then

A2 = max

Ax2

= 1 ,

x2

min

x=0

singular

values

Ax2

= p .

x2

ics

x=0

has

lie

1/2

pp

V z2 =

y2 =

i2 |yi |2

n y2 = n .

x2 =1

n

at

he

m

at

Proof. The two norm of A does not change when A is multiplied by unitary

matrices; see Exercise (iv) in Section 2.6. Hence A2 =

2 . Since

is a

diagonal matrix, Exercise (i) in Section 2.6 implies

2 = maxj |j | = 1 .

To show the expression

for p , assume that m n, so p = n. Then A

has an SVD A = U

V . Let z be a vector so that z2 = 1 and Az2 =

0

minx2 =1 Ax2 . With y = V z we get

nd

i=1

la

ia

n =

en 2 = U AV en 2 = A(V en )2 min Ax2 .

tr

x2 =1

nd

us

So

cie

ty

fo

rI

The extreme singular values are useful because they provide information

about the two-norm condition number with respect to inversion, and about the

distance to singularity.

The expressions below show that the largest singular value determines how

much a matrix can stretch a unit-norm vector and the smallest singular value

determines how much a matrix can shrink a unit-norm vector.

Fact 4.5. If A Cnn is nonsingular with singular values 1 n > 0, then

A1 2 =

1

,

n

2 (A) =

1

.

n

n = min {E2 : A + E is singular}

and the relative distance is

E2

n

= min

: A + E is singular .

1

A2

i

i

80

book

2009/5/27

page 80

i

This electronic version is for personal use and may not be duplicated or distributed.

Proof. Remark 4.2 implies that 1/j are the singular values of A1 , so that

A1 2 = maxj 1/|j | = 1/n . The expressions for the distance to singularity

follow from Fact 2.29 and Corollary 2.30.

at

ics

Fact 4.5 implies that a nonsingular matrix is almost singular in the absolute

sense if its smallest singular value is close to zero. If the smallest and largest

singular values are far apart, i.e., if 1 n , then the matrix is ill-conditioned with

respect to inversion in the normwise relative sense, and it is almost singular in the

relative sense.

The singular values themselves are well-conditioned in the normwise absolute sense. We show this below for the extreme singular values.

| p p | E2 .

| 1 1 | E2 ,

at

he

m

singular values of A and 1 p the singular values of A + E. Then

pp

lie

Proof. The inequality for 1 follows from 1 = A2 and Fact 2.13, which states

that norms are well-conditioned.

Regarding the bound for p , let y be a vector so that p = Ay2 and

y2 = 1. Then the triangle inequality implies

nd

la

x2 =1

ia

= p + Ey2 p + E2 .

nd

us

tr

p = (A + E)y2 and y2 = 1. Then the triangle inequality yields

p = min Ax2 Ay2 = (A + E)y Ey2 (A + E)y2 + Ey2

rI

x2 =1

So

cie

Exercises

ty

fo

= p + Ey2 p + E2 .

Let A Ckm , B Cmn , q = min{k, n}, and p = min{m, n}. Show:

1 (AB) 1 (A)1 (B),

2. Appending a column to a tall and skinny matrix does not increase the smallest

singular value but can decrease it, because the new column may depend

linearly on the old ones. The largest singular value does not decrease but it

can increase, because more mass is added to the matrix.

Show:

Let A Cmn with m > n, z Cm , and B = A z .

n+1 (B) n (A) and 1 (B) 1 (A).

3. Appending a row to a tall and skinny matrix does not decrease the smallest

singular value but can increase it. Intuitively, this is because the columns

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 81

i

This electronic version is for personal use and may not be duplicated or distributed.

4.2. Rank

81

independent. The largest singular value does not decrease but can increase,

because more mass is added to the matrix.

A

Let A Cmn with m n, z Cn , and B = . Show that

z

1 (A) 1 (B) 1 (A)2 + z22 .

n (B) n (A),

4.2

Rank

he

m

at

ics

For a nonsingular matrix, all singular values are nonzero. For a general matrix, the

number of nonzero singular values measures how much information is contained

in a matrix, while the number of zero singular values indicates the amount of

redundancy.

at

A Cmn is called the rank of A. An m n zero matrix has rank 0.

lie

Example 4.8.

ty

fo

rI

nd

us

tr

ia

la

nd

pp

This follows from Remark 4.2.

If A Cnn is nonsingular, then rank(A) = n = rank(A1 ).

A nonsingular matrix A contains the maximum amount of information, because it can reproduce any vector b Cn by means of b = Ax.

For any m n zero matrix 0, rank(0) = 0.

The zero matrix contains no information. It can only reproduce the zero

vector, because 0x = 0 for any vector x.

mn

If A C

has rank(A) = n, then A has an SVD A = U

V , where

0

0 V , where

cie

= (v x)u, the outer product uv can produce only multiples of the vector u.

So

uv x

rank(uv ) = 1.

To see this, determine an SVD of uv . Let U Cmm be a unitary matrix so

that U u = u2 e1 , and let V Cnn be a unitary matrix so that V v = v2 e1 .

Substituting these expressions into uv shows that uv = U

V is an SVD, where

Rmn and

= u2 v2 e1 e1 . Therefore, the singular values of uv are

u2 v2 , and (min{m, n} 1) zeros. In particular, uv 2 = u2 v2 .

The above example demonstrates that a nonzero outer product has rank one.

Now we show that a matrix of rank r can be represented as a sum of r outer

products. To this end we distinguish the columns of the left and right singular

vector matrices.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

82

book

2009/5/27

page 82

i

This electronic version is for personal use and may not be duplicated or distributed.

V if

Definition 4.10 (Singular Vectors). Let A Cmn , with SVD A = U

0

m n, and SVD A = U

0 V if m n. Set p = min{m, n} and partition

..

V = v1 . . . vn ,

=

U = u1 . . . um ,

,

.

p

ics

where 1 p 0.

We call j the j th singular value, uj the j th left singular vector, and vj the

j th right singular vector.

1 i p.

at

A u i = i vi ,

Avi = i ui ,

he

m

at

Corresponding left and right singular vectors are related to each other.

lie

This follows from the fact that U and V are unitary, and

is Hermitian.

la

nd

pp

the size of the representation is proportional to the rank of the matrix. Fact 4.12

below shows that a matrix of rank r can be expressed in terms of r outer products.

These outer products involve the singular vectors associated with the nonzero

singular values.

j =1

rI

nd

us

tr

ia

Fact 4.12 (Reduced SVD). Let A Cmn have an SVD as in Definition 4.10. If

rank(A) = r, then

r

A=

j uj vj .

So

cie

ty

fo

values to the matrix

r , so that

1

r 0

..

,

and

A

=

U

V

r =

.

0 0

r

is an SVD of A. Partitioning the singular vectors conformally with the nonzero

singular values,

r nr

mr

Umr ,

V = Vr Vnr ,

yields A = Ur

r Vr . Using Ur = u1 . . . ur and Vr = v1 . . . vr , and

viewing matrix multiplication as an outer product, as in View 4 of Section 1.7,

shows

v

r

.1

.

A = Ur

r Vr = 1 u1 . . . r ur . =

j uj vj .

j =1

vr

U=

r

Ur

i

i

book

2009/5/27

page 83

i

This electronic version is for personal use and may not be duplicated or distributed.

4.2. Rank

83

For a nonsingular matrix, the reduced SVD is equal to the ordinary SVD.

Based on the above outer product representation of a matrix, we will now

show that the singular vectors associated with the k largest singular values of A

determine the rank k matrix that is closest to A in the two norm. Moreover, the

(k + 1)st singular value of A is the absolute distance of A, in the two norm, to the

set of rank k matrices.

at

BCmn ,rank(B)=k

j =1 j uj vj .

he

m

where Ak =

ics

Fact 4.13 (Optimality of the SVD). Let A Cmn have an SVD as in Definition 4.9. If k < rank(A), then the absolute distance of A to the set of rank k

matrices is

k+1 =

min

A B2 = A Ak 2 ,

V ,

where

1 =

pp

A=U

..

lie

at

k+1

nd

us

tr

ia

la

nd

1 is nonsingular. The idea is to show that the

distance of

1 to the set of singular matrices, which is k+1 , is a lower bound for

the distance of A to the set of all rank k matrices.

Let C Cmn be a matrix with rank(C) = k, and partition

C11 C12

,

U CV =

C21 C22

So

cie

ty

fo

rI

it is intuitively clear, it is proved rigorously in Fact 6.19), so that C11 is singular.

Since the two norm is invariant under multiplication by unitary matrices, we obtain

1

U CV

A C2 =

2

2

1 C11

C

12

1 C11 2 .

=

C21

2 C22 2

Since

1 is nonsingular and C11 is singular, Facts 2.29 and 4.5 imply that

1 C11 2 is bounded below by the distance of

1 from singularity, and

1 C11 2 min{

1 B11 2 : B11 is singular} = k+1 .

A matrix C for which A C2 = k+1 is C = Ak . This is because

..

.

C11 =

, C12 = 0, C21 = 0, C22 = 0.

k

0

i

i

book

2009/5/27

page 84

i

This electronic version is for personal use and may not be duplicated or distributed.

84

Since the

1 C11 has k diagonal elements equal to zero, and the diagonal elements

of

2 are less than or equal to k+1 , we obtain

1 C11 0

k+1

= k+1 .

A C2 =

=

2 2

0

2 2

The singular values also help us to relate the rank of A to the rank of A A

and AA . This will be important later on for the solution of least squares problems.

Fact 4.14. For any matrix A Cmn ,

he

m

at

ics

rank(A) = rank(A ),

rank(A) = rank(A A) = rank(AA ),

rank(A) = n if and only if A A is nonsingular,

rank(A) = m if and only if AA is nonsingular.

at

1.

2.

3.

4.

Proof.

rI

nd

us

tr

ia

la

nd

pp

lie

1. This follows from Remark 4.2, because A and A have the same singular

values.

V , and A A = V

2 V is an

0

SVD of A A. Since

and

2 have the same number

diagonal

2of nonzero

U is an SVD

0 0

of AA . As before, rank(A) = rank(AA ) because

and

2 have the same

number of nonzero diagonal elements.

A similar argument applies when m < n.

3. Since A A is n n, A A is nonsingular if and only if n = rank(A A) =

rank(A), where the second equality follows from item 2.

4. The proof is similar to that of item 3.

ty

fo

In item 3 above the matrix A has linearly independent columns, and in item

4 it has linearly independent rows. Below we give another name to such matrices.

So

cie

Definition 4.15 (Full Rank). A matrix A Cmn has full column rank if

rank(A) = n, and full row rank if rank(A) = m.

A matrix A Cmn has full rank if A has full column rank or full row rank.

A matrix that does not have full rank is rank deficient.

Example.

A nonsingular matrix has full row rank and full column rank.

A nonzero column vector has full column rank, and a nonzero row vector

has full row rank.

If A Cnn is nonsingular,

then A B has full row rank for any matrix

A

B Cnm , and

has full column rank, for any matrix C Cmn .

C

A singular square matrix is rank deficient.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 85

i

This electronic version is for personal use and may not be duplicated or distributed.

4.2. Rank

85

Below we show that matrices with orthonormal columns also have full column rank. Recall from Definition 3.37 that A has orthonormal columns if A A = I .

Fact 4.16. A matrix A Cmn with orthonormal columns has rank(A) = n, and

all singular values are equal to one.

Fact 4.14 implies rank(A) = rank(A A) = rank(In )=

n. Thus A

V . Then

0

In = A A = V

2 V implies

= In , so that all singular values of A are equal

to one.

at

ics

Proof.

he

m

Exercises

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

(i) Let A Cmn . Show: If Q Cmm and P Cnn are unitary, then

rank(A) = rank(QAP ).

(ii) What can you say about the rank of a nilpotent matrix, and the rank of an

idempotent matrix?

(iii) Let A Cmn . Show: If rank(A) = n, then (A A)1 A 2 = 1/n , and if

rank(A) = m, then (AA )1 A2 = 1/m .

(iv) Let A Cmn with rank(A) = n. Show that A(A A)1 A is idempotent

and Hermitian, and A(A A)1 A 2 = 1.

(v) Let A Cmn with rank(A) = m. Show that A (AA )1 A is idempotent

and Hermitian, and A (AA )1 A2 = 1.

(vi) Nilpotent Matrices.

j 1 = 0 for some j 1.

Let A Cnn be nilpotent so that Aj = 0 and

A

n

j

1

Let b C with A b = 0. Show that K = b Ab . . . Aj 1 b has full

column rank.

(vii) In Fact 4.13 let B be a multiple of Ak , i.e., B = Ak . Determine A B2 .

So

cie

ty

1. Let A Cnn . Show that there exists a unitary matrix Q such that

A = QAQ.

2. Polar Decomposition.

Show: If A Cmn has rank(A) = n, then there is a factorization A = P H ,

where P Cmn has orthonormal columns, and H Cnn is Hermitian

positive definite.

3. The polar factor P is the closest matrix with orthonormal columns in the two

norm.

Let A Cnn have a polar decomposition A = P H .

Show that

A P 2 A Q2 for any unitary matrix Q.

4. The distance of a matrix A from its polar factor P is determined by how

close the columns A are to being orthonormal.

Let A Cmn , with rank(A) = n, have a polar decomposition A = P H .

Show that

A A In 2

A A In 2

A P 2

.

1 + A2

1 + n

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 86

i

This electronic version is for personal use and may not be duplicated or distributed.

86

the matrix

A

I

I

A

Singular Vectors

lie

4.3

at

he

m

at

ics

is singular.

6. Rank Revealing QR Factorization.

With an appropriate permutation of the columns, a QR factorization can

almost reveal the smallest singular value of a full column rank matrix.

Let A Cmn with rank(A) = n and smallest singular value n . Let the corresponding singular vectors be Av = n u, where v2 = u2 = 1. Choose

R

0

la

nd

pp

The singular vectors of a matrix A give information about the column spaces and

null spaces of A and A .

The column space of a matrix A is the set of all right-hand sides b for which

the system Ax = b has a solution, and the null space of A determines whether these

solutions are unique.

tr

ia

Definition 4.17 (Column Space and Null Space). If A Cmn , then the set

nd

us

m

n

R(A) = {b C : b = Ax for some x C }

fo

rI

Ker(A) = {x Cn : Ax = 0}

cie

ty

So

Example.

The column space of an m n zero matrix is the zero vector, and the null

space is Cn , i.e., R(0mn ) = {0m1 } and Ker(0mn ) = Cn .

The column space of an n n nonsingular complex matrix is Cn , and the

null space consists of the single vector 0n1 .

Ker(A) = {0} if and only if the columns of the matrix A are linearly independent.

If A Cmn , then for all k 1

R A

0mk = R(A),

Ker

0kn

= Ker(A).

i

i

book

2009/5/27

page 87

i

This electronic version is for personal use and may not be duplicated or distributed.

87

A

Ker

= {0n1 }.

R A B = R(A),

C

The column and null spaces of A are also important, and we give them

names that relate to the matrix A.

Definition 4.18 (Row Space and Left Null Space). Let A Cmn . The set

ics

m

R(A ) = {d C : d = A y for some y C }

at

he

m

Ker(A ) = {y Cm : A y = 0}

at

pp

lie

la

nd

The singular vectors reproduce the four spaces associated with a matrix. Let

A Cmn with rank(A) = r and SVD

r 0

A=U

V ,

0 0

tr

nd

us

nr

0

,

0

r

0

U=

r

Ur

mr

Umr ,

V =

r

Vr

nr

Vnr .

rI

r

mr

ia

where

r is nonsingular, and

ty

fo

So

cie

If A = 0, then R(Ur ) = R(A); otherwise R(A) = {0m1 }.

2. The trailing n r right singular vectors represent the null space of A:

If rank(A) = r < n, then R(Vnr ) = Ker(A); otherwise Ker(A) = {0n1 }.

3. The leading r right singular vectors represent the row space of A:

If A = 0, then R(A ) = R(Vr ); otherwise R(A ) = {0n1 }.

4. The trailing m r left singular vectors represent the left null space of A:

If r < m, then R(Umr ) = Ker(A ); otherwise Ker(A ) = {0m1 }.

Proof. Although the statements may be intuitively obvious, they are proved rigorously in Section 6.1.

The singular vectors help us to relate the spaces of A A and AA to those

of the matrix A. Since A A and AA are Hermitian, we need to specify only two

spaces; see Example 4.19.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 88

i

This electronic version is for personal use and may not be duplicated or distributed.

88

1. Ker(A A) = Ker(A) and R(A A) = R(A ).

2. R(AA ) = R(A) and Ker(AA ) = Ker(A ).

ics

Proof. Fact 4.14 implies that A A and AA have the same rank as A. Since A A

has the same right singular vectors as A, Fact 4.20 implies Ker(A A) = Ker(A)

and R(A A) = R(A ). Since AA has the same left singular vectors as A, Fact

4.20 implies R(AA ) = R(A) and Ker(AA ) = Ker(A ).

at

he

m

at

In the special case when the rank of a matrix is equal to the number of rows,

then the number of elements in the column space is as large as possible. When the

rank of the matrix is equal to the number of columns, then the number of elements

in the null space is as small as possible.

lie

2. rank(A) = n if and only if Ker(A) = {0}.

pp

0

V be an SVD of A, where

r is nonsingular.

0

nd

r

Proof. Let A = U

0

us

tr

ia

la

because U is nonsingular so that R(U ) = Cm .

2. Fact 4.20 also implies r = n if and only if Vnr is empty, which means that

Ker(A) = {0}.

rI

nd

If the matrix in a linear system has full rank, then existence or uniqueness of

a solution is guaranteed.

ty

fo

So

cie

b Cm .

2. If rank(A) = n and if b R(A), then Ax = b has the unique solution

x = (A A)1 A b.

Proof.

1. Fact 4.22 implies that Ax = b has a solution for every b Cm , and

Fact 4.14 implies that AA is nonsingular. Clearly, x = A (AA )1 b satisfies Ax = b.

2. Since b R(A), Ax = b has a solution. Multiplying on the left by A

gives A Ax = A b. According to Fact 4.14, A A is nonsingular, so that

x = (A A)1 A b.

Suppose Ax = b and Ay = b; then A(x y) = 0. Fact 4.22 implies that

Ker(A) = {0}, so x = y, which proves uniqueness.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 89

i

This electronic version is for personal use and may not be duplicated or distributed.

89

Exercises

(i) Fredholms Alternatives.

(a)

he

m

at

ics

(b)

The first alternative implies that R(A) and Ker(A ) have only the zero

vector in common. Assume b = 0 and show:

If Ax = b has a solution, then b A = 0.

In other words, if b R(A), then b Ker(A ).

The second alternative implies that Ker(A) and R(A ) have only the

zero vector in common. Assume x = 0 and show:

If Ax = 0, then there is no y such that x = A y.

In other words, if x Ker(A), then x R(A ),

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

If A Cn is Hermitian, then R(A ) = R(A) and Ker(A ) = Ker(A). These

equalities remain true for a larger class of matrices, the so-called normal

matrices. A matrix A Cn is normal if A A = AA .

Show: If A Cnn is normal, then R(A ) = R(A) and Ker(A ) = Ker(A).

i

i

book

2009/5/27

page 90

i

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

i

i

book

2009/5/27

page 91

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

la

nd

pp

lie

the column space of A, there is no x such that Ax = b. The best we can do is to

find a vector y that brings left- and right-hand sides of the linear system as close

as possible; in other words y is chosen to make the distance between Ay and b as

small as possible. That is, we want to minimize the distance Ax b2 over all x,

and distance will again be measured in the two norm.

us

tr

ia

Definition 5.1 (Least Squares Problem). Let A Cmn and b Cm . The least

squares problem consists of finding a vector y Cn so that

min Ax b2 = Ay b2 .

nd

fo

rI

So

cie

ty

5.1

x

x

least

|(Ax b)i |2 .

squares

Let A Cmn have rank(A) = r and an SVD

r

A=U

0

0

V ,

0

U=

r

Ur

mr

Umr ,

V =

r

Vr

nr

Vnr ,

r is a diagonal matrix with

diagonal elements 1 r > 0, i.e.,

r is nonsingular.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

91

i

i

i

92

book

2009/5/27

page 92

i

This electronic version is for personal use and may not be duplicated or distributed.

Fact 5.2 (All Least Squares Solutions). Let A Cmn and b Cm . The solutions of minx Ax b2 are of the form y = Vr

r1 Ur b + Vnr z for any z Cnr .

Proof. Let y be a solution of the least squares problem, partition

Vr y

w

V y =

y = z ,

Vnr

at

ics

r w Ur b

r 0

.

V y b = U

Ay b = U

b

0 0

Umr

at

b22 .

Ay b22 =

r w Ur b22 + Umr

he

m

pp

lie

Since the second summand is constant and independent of w and z, the residual

is minimized if the first summand is zero, that is, if w =

r1 Ur b. Therefore, the

solution of the least squares problem equals

w

y=V

= Vr w + Vnr z = Vr

r1 Ur b + Vnr z.

z

la

nd

Fact 4.20 implies that Vnr z Ker(A) for any vector z. Hence Vnr z does not

have any effect on the least squares residual, so that z can assume any value.

rI

nd

us

tr

ia

Fact 5.1 shows that if A has rank r < n, then the least squares problem has

infinitely many solutions. The first term in a least squares solution contains the

matrix

1

r

0

1

U

Vr

r Ur = V

0

0

cie

ty

fo

which is obtained by inverting only the nonsingular parts of an SVD. This matrix

is almost an inverse, but not quite.

So

Definition

Inverse). If A Cmn and rank(A) = r 1, let

5.3 (MoorePenrose

r 0

V be an SVD where

r is nonsingular. The n m matrix

A=U

0 0

1

r

0

U

A = V

0

0

is called MoorePenrose inverse of A. If A = 0mn , then A = 0nm .

The MoorePenrose inverse of a full rank matrix can be expressed in terms

of the matrix itself.

Remark 5.4 (MoorePenrose Inverses of Full Rank Matrices). Let A Cmn .

If A is nonsingular, then A = A1 .

i

i

book

2009/5/27

page 93

i

This electronic version is for personal use and may not be duplicated or distributed.

93

This means A A = In , so that A is a left inverse of A.

If A Cmn and rank(A) = m, then A = A (AA )1 .

This means AA = Im , so that A is a right inverse of A.

Now we can express the least squares solutions in terms of the Moore

Penrose inverse, without reference to the SVD.

he

m

at

ics

Corollary 5.5 (All Least Squares Solutions). Let A Cmn and b Cmn . The

solutions of minx Ax b2 are of the form y = A b + q, where q Ker(A).

lie

at

Although a least squares problem can have infinitely many solutions, all

solutions have the part A b in common, and they differ only in the part that belongs

to Ker(A). As a result, all least squares solutions have not just residuals of the

same norm, but they have the same residual.

nd

pp

Fact 5.6 (Uniqueness of the Least Squares Residual). Let A Cmn and

b Cm . All solutions y of minx Ax b2 have the same residual b Ay =

(I AA )b.

us

tr

ia

la

Proof. Let y1 and y2 be solutions to minx Ax b2 . Corollary 5.5 implies

y1 = A b + q1 and y2 = A b + q2 , where q1 , q2 Ker(A). Hence Ay1 = AA b =

Ay2 , and both solutions have the same residual, b Ay1 = b Ay2 =

(I AA )b.

rI

nd

Besides being unique, the least squares residual has another important property: It is orthogonal to the column space of the matrix.

ty

fo

and y a solution of minx Ax b2 with residual r = b Ay. Then A r = 0.

So

cie

Proof. Fact 5.6 implies that the unique residual is r = (I AA )b. Let A have

an SVD

r 0

A=U

V ,

0 0

where U and V are unitary, and

r is a diagonal matrix with positive diagonal

elements. From Definition 5.3 of the MoorePenrose inverse we obtain

0

0

Ir

0rr

AA = U

U ,

U .

I AA = U

0 0(mr)(mr)

0

Imr

Hence A (I AA ) = 0nm and A r = 0.

The part of the least squares problem solution y = A b + q that is responsible

for lack of uniqueness is the term q Ker(A). We can force the least squares

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 94

i

This electronic version is for personal use and may not be duplicated or distributed.

94

problem to have a unique solution if we add the constraint q = 0. It turns out that

the resulting solution A b has minimal norm among all least squares solutions.

Fact 5.8 (Minimal Norm Least Squares Solution). Let A Cmn and b Cm .

Among all solutions of minx Ax b2 the one with minimal two norm is y = A b.

at

ics

Proof. From the proof of Fact 5.2 follows that any least squares solution has the

form

1

r Ur b

.

y=V

z

he

m

Hence

at

y22 =

r1 Ur b22 + z22

r1 Ur b22 = Vr

r1 Ur b22 = A b22 .

lie

Thus, any least squares solution y satisfies y2 A b2 . This means y = A b

is the least squares solution with minimal two norm.

pp

The most pleasant least squares problems are those where the matrix A has

full column rank because then Ker(A) = {0} and the least squares solution is unique.

la

nd

Fact 5.9 (Full Column Rank Least Squares). Let A Cmn and b Cm . If

rank(A) = n, then minx Ax b2 has the unique solution y = (A A)1 A b.

us

tr

ia

Proof. From Fact 4.22 we know that rank(A) = n implies Ker(A) = {0}. Hence

q = 0 in Corollary 5.5. The expression for A follows from Remark 5.4.

cie

Exercises

ty

fo

rI

nd

In particular, when A is nonsingular, then the MoorePenrose inverse reduces to the ordinary inverse. This means, if we solve a least squares problem

minx Ax b2 with a nonsingular matrix A, we obtain the solution y = A1 b of

the linear system Ax = b.

So

row vector?

(ii) Let u Cmn and v Cn with v = 0. Show that uv 2 = u2 /v2 .

(iii) Let A Cmn . Show that the following matrices are idempotent:

AA ,

A A,

Im AA ,

In A A.

(v) Let A Cmn . Show:

(Im AA )A = 0mn ,

A(In A A) = 0mn .

i

i

book

2009/5/27

page 95

i

This electronic version is for personal use and may not be duplicated or distributed.

95

pp

lie

at

he

m

at

ics

(viii) Let A Cmn have rank(A) = n. Show: (A A)1 2 = A 22 .

(ix) Let A = BC where B Cmn has rank(B) = n and C Cnn is nonsingular.

Show: A = C 1 B .

(x) Let A Cmn with rank(A) = n and thin QR factorization A = QR, where

Q Q = In and R is upper triangular. Show: A = R 1 Q .

(xi) Show: If A has orthonormal columns, then A = A .

(xii) Partial Isometry.

A matrix A Cmn is called a partial isometry if A = A . Show: A is a

partial isometry if and only if all its singular values are 0 or 1.

(xiii) What is the minimal norm solution to minx Ax b2 when A = 0?

(xiv) If y is the minimal norm solution to minx Ax b2 and A b = 0, then what

can you say about y?

(xv) Given an approximate solution z to a linear system Ax = b, this problem

shows how to construct a linear system (A + E)x = b for which z is the exact

solution.

Let A Cmn and b Cm . Let z Cn with z = 0 and residual r = b Az.

Show: If E = rz , then (A + E)z = b.

fo

rI

nd

us

tr

ia

la

nd

1. What is the minimal norm solution to minx Ax b2 when A = uv , where

u and v are column vectors?

I

mn

2. Let A C

. Show: The singular values of n are equal to 1/ 1 + j2 ,

A

1 j n.

3. Let A Cmn have rank(A) = n. Show: I AA 2 = min{1, m n}.

4. Let A Cmn . Show: A is the MoorePenrose inverse of A if and only if

A satisfies

MP1: AA A = A, A AA = A ,

MP2: AA and A A are Hermitian.

cie

ty

Let A Cmn have rank(A) = n and be partitioned as A = A1

So

(a)

A =

B1

B2

A2 . Show:

,

where

B1 = (I A2 A2 )A1 ,

B2 = (I A1 A1 )A2 .

(b) B1 2 = minZ A1 A2 Z2 and B2 2 = minZ A2 A1 Z2 .

(c) Let 1 k n, and let V11 be the leading k k principal submatrix of V .

1

Show: If V11 is nonsingular, then A1 2 V11

2 /k .

5.2

Least squares problems are much more sensitive to perturbations than linear systems. A least squares problem whose matrix is deficient in column rank is so

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

96

book

2009/5/27

page 96

i

This electronic version is for personal use and may not be duplicated or distributed.

sensitive that we cannot even define a condition number. The example below

illustrates this.

Example 5.10 (Rank Deficient Least Squares Problems are Ill-Posed).

sider the least squares problem minx Ax b2 with

1

1

1 0

b=

,

y=A b=

.

A=

=A ,

1

0

0 0

Con-

he

m

at

ics

The matrix A is rank deficient and y is the minimal norm solution. Let us perturb

the matrix so that

1 0

A+E =

,

where

0 < 1.

0

pp

lie

at

The matrix A + E has full column rank and minx (A + E)x b2 has the unique

solution z where

1

1

.

z = (A + E) b = (A + E) b =

1/

nd

us

tr

ia

la

nd

Comparing the two minimal norm solutions shows that the second element of z

grows as the (2, 2) element of A + E decreases, i.e., z2 = 1/ as 0. But

at = 0 we have z2 = 0. Therefore, the least squares solution does not depend

continuously on the (2, 2) element of the matrix. This is an ill-posed problem.

In an ill-posed problem the solution is not a continuous function of the inputs.

The ill-posedness of a rank deficient least squares problem comes about because a

small perturbation can increase the rank of the matrix.

fo

rI

the exact and perturbed matrices have full column rank. Below we determine the

sensitivity of the least squares solution to changes in the right-hand side.

So

cie

ty

let y be the solution to minx Ax b2 , and let z be the solution to

minx Ax (b + f )2 . If y = 0, then

f 2

z y2

2 (A)

,

y2

A2 y2

and if z = 0, then

f 2

z y2

2 (A)

,

z2

A2 z2

where 2 (A) = A2 A 2 .

Proof. Fact 5.9 implies that y = A b and z = A (b + f ) are the unique solutions

to the respective least squares problems. From y = A b = (A A)1 A b, see

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 97

i

This electronic version is for personal use and may not be duplicated or distributed.

97

Remark 5.4, and the assumption A b = 0 follows y = 0. Applying the bound for

matrix multiplication in Fact 2.22 yields

z y2 A 2 b2 f 2

f 2

= A 2

.

y2

y2

A b2 b2

Now multiply and divide by A2 on the right.

ics

In Fact 5.11 we have extended the two-norm condition number with respect

to inversion from nonsingular matrices to matrices with full column rank.

he

m

at

the two-norm condition number of A with regard to left inversion.

pp

lie

at

Fact 5.11 implies that 2 (A) is the normwise relative condition number of

the least squares solution to changes in the right-hand side. If the columns of A

are close to being linearly dependent, then A is close to being rank deficient and

the least squares solution is sensitive to changes in the right-hand side.

With regard to changes in the matrix, though, the situation is much bleaker.

It turns out that least squares problems are much more sensitive to changes in the

matrix than linear systems.

0 < 1,

0 < 1 , 3 .

tr

ia

la

nd

1 0

1

where

A = 0 ,

b = 0 ,

3

0 0

ty

fo

rI

nd

us

The element 3 represents the part of b outside R(A). The matrix A has full column

rank, and the least squares problem minx Ax b2 has the unique solution y

where

1

0

0

,

y = A b = 1 .

A = (A A)1 A =

0 1/ 0

0

So

cie

Let us perturb the matrix and change its column space so that

1 0

A + E = 0 ,

where 0 < 1.

0

Note that R(A + E) = R(A). The matrix A + E has full column rank and Moore

Penrose inverse

1

1

0

0

.

(A + E) = 0

(A + E) = (A + E) (A + E)

2 + 2

2 + 2

The perturbed problem minx (A + E)x b2 has the unique solution z, where

1

.

z = (A + E) b =

3 /( 2 + 2 )

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

98

book

2009/5/27

page 98

i

This electronic version is for personal use and may not be duplicated or distributed.

3

3

z y2

=

2 .

y2

1 ( 2 + 2 )

1

he

m

at

Ay b2

r2

E2

z y2

A 22

E2 = [2 (A)]2

,

y2

y2

A2 y2 A2

ics

inside R(A), then the perturbation is amplified by at least 1/ 2 .

In other words, since E = , A 2 = 1/, and 3 /1 = Ay b2 /y2 ,

we can write

pp

r2

r2

,

A2 y2 Ay2

lie

at

where r = Ay b is the residual. This means, if the right-hand side is far away

from the column space, then the condition number with respect to changes in the

matrix is [2 (A)]2 , rather than just 2 (A).

We can give a geometric interpretation for the relative residual norm. If we

bound

r2

b2

2

Ay2

b2

2

.

us

tr

ia

1=

la

nd

then we can exploit the relation between r2 and Ay2 from Exercise (iii) below.

There, it is shown that b22 = r22 + Ay22 , hence

rI

nd

It follows that r2 /b2 and Ay2 /b2 behave like sine and cosine. Thus there

is so that

ty

fo

1 = sin 2 + cos 2 ,

where

sin =

r2

,

b2

cos =

Ay2

,

b2

So

cie

and can be interpreted as the angle between b and R(A). This allows us to bound

the relative residual norm by

r2

sin

r2

=

= tan .

A2 y2 Ay2

cos

This means if the angle between right-hand side and column space is large enough,

then the least squares solution is sensitive to perturbations in the matrix, and this

sensitivity is represented by [2 (A)]2 .

The matrix in Example 5.13 is representative of the situation in general.

Least squares solutions are more sensitive to changes in the matrix when the righthand side is too far from the column space. Below we present a bound for the

relative error with regard to the perturbed solution z, because it is much easier to

derive than a bound for the relative error with regard to the exact solution y.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 99

i

This electronic version is for personal use and may not be duplicated or distributed.

99

with rank(A) = rank(A + E) = n, let y be the solution to minx Ax b2 , and let

z = 0 be the solution to minx (A + E)x (b + f )2 . Then

z y2

s2

2 (A) A + f + [2 (A)]2

A ,

z2

A2 z2

s = (A + E)z (b + f ),

A =

E2

,

A2

f =

f 2

.

A2 z2

ics

where

at

he

m

at

unique solutions to the respective least squares problems. Applying Fact 5.7 to

the perturbed least squares problem gives (A + E) s = 0, hence A s = E s.

Multiplying by (A A)1 and using A = (A A)1 A from Remark 5.4 gives

pp

lie

and use the fact that (A A)1 2 = A 22 , see Exercise (viii) in Section 5.1, to

obtain

nd

tr

ia

la

At last divide both sides of the inequality by z2 , and multiply and divide the

right side by A22 .

us

Remark 5.15.

So

cie

ty

fo

rI

nd

Therefore, the least squares solution is more sensitive to changes in the

matrix than to changes in the right-hand side.

The first term 2 (A)(A + f ) in the above bound is the same as the perturbation bound for linear systems in Fact 3.8. It is because of the second

term in Fact 5.14 that least squares problems are more sensitive than linear

systems to perturbations in the matrix.

We can interpret s2 /(A2 z2 ) as an approximation to the distance between perturbed right-hand side and perturbed matrix. From Exercise (ii)

and Example 5.13 follows

s2

s2

(1 + A ) tan (1 + A ),

A2 z2 A + E2 z2

where is the angle between b + f and R(A + E).

If most of the right-hand side lies in the column space, then the condition

number of the least squares problem is 2 (A).

2

In particular, if As

A , then the second term in the bound in Fact 5.14

2 z2

2

2

is about [2 (A)] A , and negligible for small enough A .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

100

book

2009/5/27

page 100

i

This electronic version is for personal use and may not be duplicated or distributed.

Exercises

at

he

m

(ii) Under the assumptions of Fact 5.14 show that

at

ics

If the right-hand side is far away from the column space, then the condition

number of the least squares problem is [2 (A)]2 .

Therefore, the solution of the least squares is ill-conditioned in the normwise

relative sense, if A is close to being rank deficient, i.e., 2 (A) 1, or if the

relative residual norm is large, i.e., (A+E)z(b +f )2 /(A2 z2 ) 0.

If the perturbation does not change the column space so that R(A + E) =

R(A), then the least squares problem is no more sensitive than a linear

system; see Exercise 1 below.

s2

s2

s2

(1 A )

(1 + A ).

A + E2 z2

A2 z2 A + E2 z2

pp

lie

(iii) Let A Cmn , and let y be a solution to the least squares problem

minx Ay b2 . Show:

nd

nd

us

tr

ia

la

(iv) Let A Cmn have rank(A) = n. Show that the solution y of the least

squares problem minx Ax b2 and the residual r = b Ay can be viewed

as solutions to the linear system

I A

r

b

=

,

A 0

y

0

rI

and that

I

A

1

A

I AA

=

0

A

(A )

.

(A A)1

cie

ty

fo

So

rank(A + E) = n, and let z be the solution of the least squares problem

minx (A + E)x (b + f )2 with residual s = b + f (A + E)z. Show:

f Ez

(A )

s r

I AA

.

=

E s

zy

A

(A A)1

(vi) Let A, A + E Cmn and rank(A) = n. Show: If E2 A 2 < 1, then

rank(A + E) = n.

1. Matrices with the Same Column Space.

When the perturbed matrix has the same column space as the original matrix,

then the least squares solution is less sensitive, and the error bound is the

same as the one for linear systems in Fact 3.9.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 101

i

This electronic version is for personal use and may not be duplicated or distributed.

101

solution to minx Ax b2 , and let z = 0 be the solution to

minx (A + E)x (b + f )2 . Show: If R(A) = R(A + E), then

z y2

2 (A) A + f ,

z2

A =

where

E2

,

A2

f =

f

.

A2 z2

he

m

at

ics

This bound shows that the least squares residual is insensitive to changes in

the right-hand side.

Let A Cmn have rank(A) = n. Let y be the solution to minx Ax b2

with residual r = Ay b, and let z be the solution to minx Az (b + f )2

with residual s = Az (b + f ). Show:

s r2 f 2 .

where

A =

E2

,

A2

b =

f 2

.

b2

ia

la

r2

s2

+ 2 (A) A + b ,

b2 b2

nd

pp

lie

at

The following bound gives an indication of how sensitive the norm of the

least squares residual may be to changes in the matrix and right-hand side.

Let A, A + E Cmn so that rank(A) = rank(A + E) = n. Let y be the

solution to minx Ax b2 with residual r = Ay b, and let z be the solution

to minx (A + E)x (b + f )2 with residual s = (A+E)z(b +f ). Show:

If b = 0, then

fo

rI

nd

us

tr

4. This bound suggests that the error in the least squares solution depends on

the error in the least squares residual.

Under the conditions of Fact 5.14 show that

r s2

z y2

2 (A)

+ A + f .

z2

A2 z2

So

cie

ty

construct a least squares problem for which z is the exact solution.

Let z = 0 be an approximate solution of the least squares problem

minx Ax b2 . Let rc = b Az be the computable residual, h an arbitrary vector, and F = hh A + (I hh )rc z . Show that z is a least squares

solution of minx (A + F )x b2 .

5.3

Problems

We present two algorithms for computing the solution to a least squares problem

with full column rank.

Let A Cmn have rank(A) = n and an SVD

A=U

V ,

0

U=

n

Un

mn

Umn ,

i

i

book

2009/5/27

page 102

i

This electronic version is for personal use and may not be duplicated or distributed.

102

Cnn is a diagonal matrix

with diagonal elements 1 n > 0.

Fact 5.16 (Least Squares via SVD). Let A Cmn with rank(A) = n, let b

Cm , and let y be the solution to minx Ax b2 . Then

y = V

1 Un b,

b2 .

x

at

he

m

at

ics

Proof. The expression for y follows from Fact 5.9. With regard to the residual,

Ay b = U

V V

1 Un b b

0

Un b

Un b

=U

0

Umn

b

0

=U

.

Umn

b

lie

b2 .

Therefore, minx Ax b2 = Ay b2 = Umn

pp

us

tr

ia

la

nd

Output: Solution y of min

x Ax

b2 , residual norm = Ay b2

1. Compute an SVD A = U

V where U Cmm and V Cnn are

0

unitary, and

is diagonal.

2. Partition U = Un Umn , where Un has n columns.

fo

rI

nd

3. Multiply y V

1 Un b.

4. Set Umn

b2 .

So

cie

ty

which may be cheaper than an SVD. Let A Cmn have rank(A) = n and a QR

factorization

n mn

R

A=Q

,

Q = Qn Qmn ,

0

where Q Cmm is unitary and R Cnn is upper triangular with positive diagonal

elements.

Fact 5.17 (Least Squares Solution via QR). Let A Cmn with rank(A) = n,

let b Cm , and let y be the solution to minx Ax b2 . Then

y = R 1 Qn b,

x

Proof. Fact 5.9 and Remark 5.4 imply for the solution

y = A b = (A A)

A b=

R 1

0 Q b=

R 1

Qn b

= R 1 Qn b.

Qmn b

i

i

book

2009/5/27

page 103

i

This electronic version is for personal use and may not be duplicated or distributed.

103

Qn b

Qn b

0

R

=

Q

.

Ay b = Q

R 1 Qn b b = Q

0

Qmn b

Qmn b

0

Therefore, minx Ax b2 = Ay b2 = Qmn b2 .

ics

at

lie

2.

3.

4.

pp

1.

he

m

at

Output: Solution y of minx Ax b2 , residual norm = Ay b2

R

Factor A = Q

where Q Cmm is unitary and R Cnn is triangular.

0

Partition Q = Qn Qmn , where Qn has n columns.

Solve the triangular system Ry = Qn b.

Set Qmn b2 .

Exercises

z y2

f 2

[2 (A)]2

.

y2

A A2 y2

cie

ty

fo

rI

nd

us

tr

ia

la

nd

1. Normal Equations.

Let A Cmn and b Cm . Show: y is a solution of minx Ax b2 if and

only if y is a solution of A Ax = A b.

2. Numerical Instability of Normal Equations.

Show that the normal equations can be a numerically unstable method for

solving the least squares problem.

Let A Cmn with rank(A) = n, and let A Ay = A b with A b = 0. Let z

be a perturbed solution with A Az = A b + f . Show:

So

That is, the numerical stability of the normal equations is always determined

by [2 (A)]2 , even if the least squares residual is small.

i

i

book

2009/5/27

page 104

i

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

i

i

book

2009/5/27

page 105

i

This electronic version is for personal use and may not be duplicated or distributed.

at

he

m

at

ics

6. Subspaces

pp

lie

We present properties of column, row, and null spaces; define operations on them;

show how they are related to each other; and illustrate how they can be represented

computationally.

nd

us

tr

ia

la

nd

Remark 6.1. Column and null spaces of a matrix A are more than just ordinary

sets.

If x, y Ker(A), then Ax = 0 and Ay = 0. Hence A(x + y) = 0, and

A(x) = 0 for C. Therefore, x + y Ker(A) and x Ker(A).

Also, if b, c R(A), then b = Ax and c = Ay for some x and y. Hence

b + c = A(x + y) and b = A(x) for C. Therefore b + c R(A) and

b R(A).

ty

fo

rI

The above remark illustrates that we cannot fall out of the sets Ker(A) and

(A)

by adding vectors from the set or by multiplying a vector from the set by a

R

scalar. Sets with this property are called subspaces.

So

cie

addition and scalar multiplication. That is, if v, w S, then v + w S and v S

for C.

A set S Rn is a subspace of Rn if v, w S implies v + w S and v S

for R.

A subspace is never empty. At the very least it contains the zero vector.

Example.

Extreme cases: {0n1 } and Cn are subspaces of Cn ; and {0n1 } and Rn are

subspaces of Rn .

If A Cmn , then R(A) is a subspace of Cm , and Ker(A) is a subspace

of Cn .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

105

i

i

i

book

2009/5/27

page 106

i

This electronic version is for personal use and may not be duplicated or distributed.

106

6. Subspaces

of Rn .

For simplicity, we will state subsequent results and definitions only for complex subspaces, but they hold also for real subspaces.

Exercises

la

nd

pp

lie

at

he

m

at

ics

(i) Let S C3 be the set of all vectors with first and third components equal to

zero. Show that S is a subspace of C3 .

(ii) Let S C3 be the set of all vectors with first component equal to 17. Show

that S is not a subspace of C3 .

(iii) Let u Cn . Show that the set {x Cn : x u = 0} is a subspace of Cn .

(iv) Let u Cn and u = 0. Show that the set {x Cn : x u = 1} is not a subspace

of Cn .

(v) Let A Cmn . For which b Cm is the set of all solutions to Ax = b a

subspace of Cn ?

x

mn

mp

(vi) Let A C

and B C

. Show that the set

: Ax = By is a

y

subspace of Cn+p .

(vii) Let A Cmn and B Cmp . Show that the set

!

"

b : b = Ax + By for some x Cn , y Cp

tr

us

nd

6.1

ia

is a subspace of Cm .

fo

rI

We give a rigorous proof of Fact 4.20, which shows that the four subspaces of a

matrix are generated by singular vectors. In order to do so, we first relate column

and null spaces of a product to those of the factors.

cie

ty

Fact 6.3 (Column Space and Null Space of a Product). Let A Cmn and

B Cnp . Then

So

2. Ker(B) Ker(AB).

If A has linearly independent columns, then

Ker(B) = Ker(AB).

Proof.

1. If b R(AB), then b = ABx for some vector x. Setting y = Bx implies

b = Ay, which means that b is a linear combination of columns of A and

b R(A). Thus R(AB) R(A).

If B has linearly independent rows, then B has full row rank, and Fact 4.22

implies R(B) = Cn . Let b R(A) so that b = Ax for some x Cn . Since

B has full row rank, there exists a y Cp so that x = By. Hence b = ABy

and b R(AB). Thus R(A) R(AB).

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 107

i

This electronic version is for personal use and may not be duplicated or distributed.

107

Ker(B) Ker(AB).

If A has linearly independent columns, then A has full column rank, and Fact

4.22 implies that Ker(A) = {0n1 }. Hence ABx = 0 implies that Bx = 0.

Thus Ker(AB) Ker(B).

ics

and that Ker(AB) = Ker(B) if A is nonsingular.

If we partition a nonsingular matrix and its inverse appropriately, then we

can relate null spaces in the inverse to column spaces in the matrix proper.

nk

A2 ,

A1 =

k

nk

he

m

k

A1

B1

,

B2

at

A=

at

nd

us

tr

ia

la

nd

pp

lie

A1 A = In .

If b R(A2 ), then b = A2 x for some x and B1 b = B1 A2 x = 0, so

b Ker(B1 ). Thus R(A2 ) Ker(B1 ).

If b Ker(B1 ), then B1 b = 0. Write b = AA1 b = A1 x1 + A2 x2 , where

x1 = B1 b and x2 = B2 b. But b Ker(B1 ) implies x1 = 0, so b = A2 x2 and

b R(A2 ). Thus Ker(B1 ) R(A2 ).

The equality Ker(B2 ) = R(A1 ) is shown in an analogous fashion.

rI

Example 6.5.

So

cie

ty

fo

0 0

1

0 1 0

Ker 1 0 0 = R 1 0 ,

Ker

= R 0 .

0 0 1

0 1

0

Ker(A1 ) = R(A2 ),

Ker(A2 ) = R(A1 ).

vectors. Let A Cmn have rank(A) = r and an SVD

A=U

r

0

0

V ,

0

U=

r

Ur

mr

Umr ,

V =

r

Vr

nr

Vnr ,

r is a diagonal matrix with

positive diagonal elements 1 r > 0.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

108

book

2009/5/27

page 108

i

This electronic version is for personal use and may not be duplicated or distributed.

6. Subspaces

Proof. In the reduced SVD A = Ur

r Vr , the matrices Vr and

r have linearly

independent rows. Fact 6.3 implies R(A) = R(Ur ).

Fact 6.7 (Null Space). If A Cmn and rank(A) = r < n, then R(Vnr ) = Ker(A).

he

m

at

ics

r Vr , the matrices Ur and

r have linearly

independent columns. Fact 6.3 implies Ker(A) = Ker(Vr ). From Example 6.5

follows Ker(Vr ) = R(Vnr ).

at

The analogous statements for row space and left null space in Fact 4.20 can

be proved by applying Facts 6.6 and 6.7 to A .

(i) Let

lie

Exercises

A12

,

A22

nd

A11

A=

0

pp

A11

= Ker 0

0

A12

= Ker A1

R A

11

22

tr

ia

,

A1

22

1

.

A1

11 A12 A22

nd

us

la

So

cie

ty

fo

rI

(iii) Let A, B Cnn , and B idempotent. Show: AB = A if and only if

Ker(B) Ker(A).

(iv) Let A Cnn be idempotent. Show: R(A AB) and R(AB B) have

only the zero vector in common.

1. QR Factorization.

Let A Cmn with m n have a QR decomposition

R

A=Q

,

0

Q=

n

Qn

mn

Qmn ,

R(A) R(Qn ),

Ker(A) = Ker(R),

R(Qmn ) Ker(A ).

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 109

i

This electronic version is for personal use and may not be duplicated or distributed.

6.2. Dimension

6.2

109

Dimension

All subspaces of Cn , except for {0}, have infinitely many elements. But some

subspaces have more infinitely many elements than others. To quantify the size

of a subspace we introduce the concept of dimension.

Definition 6.8 (Dimension). Let S be a subspace of Cm , and let A Cmn be a

matrix so that S = R(A). The dimension of S is dim(S) = rank(A).

ics

Example.

at

dim(Cn ) = dim(Rn ) = n.

he

m

dim({0n1 }) = 0.

at

defined.

pp

lie

A Cmn and B Cmp be matrices so that S = R(A) = R(B). Then

rank(A) = rank(B).

ty

fo

rI

nd

us

tr

ia

la

nd

rank(B) = 0.

If S = {0}, set = rank(A) and = rank(B). Fact 6.6 implies that

R(A) = R(UA ), where UA is an m matrix of left singular vectors associated

with the nonzero singular values of A. Similarly, R(B) = R(UB ) where UB

is an m matrix of left singular vectors associated with the nonzero singular

values of B.

Now suppose to the contrary that > . Since S = R(UA ) = R(UB ), each

of the columns of UA can be expressed as a linear combination of UB . This

means UA = UB Y , where Y is a matrix. Using the fact that UA and UB have

orthonormal columns gives

I = UA UA = Y UB UB Y = Y Y .

So

cie

= rank(I ) = rank(Y Y ) = rank(Y ) min{, } = .

= , so that the dimension of S is unique.

The so-called dimension formula below is sometimes called the first part of

the fundamental theorem of linear algebra. The formula relates the dimensions

of column and null spaces to the number of rows and columns.

Fact 6.10 (Dimension Formula). If A Cmn , then

rank(A) = dim(R(A)) = dim(R(A ))

and

n = rank(A) + dim(Ker(A)),

i

i

110

book

2009/5/27

page 110

i

This electronic version is for personal use and may not be duplicated or distributed.

6. Subspaces

Proof. The first set of equalities follows from rank(A) = rank(A ); see Fact 4.14.

The remaining equalities follow from Fact 4.20, and from Fact 4.16 which implies

that a matrix with k orthonormal columns has rank equal to k.

Fact 6.10 implies that the column space and the row space of a matrix have the

same dimension. Furthermore, for an m n matrix, the null space has dimension

n rank(A), and the left null space has dimension m rank(A).

Example 6.11.

dim(Ker(vu )) = m1.

at

dim(Ker(uv )) = n1,

rank(uv ) = 1,

he

m

at

ics

rank(0mn ) = 0 and dim(Ker(0mn )) = n.

If u Cm , v Cn , u = 0 and v = 0, then

pp

lie

The following bound confirms that the dimension gives information about

the size of a subspace: If a subspace V is contained in a subspace W, then the

dimension of V cannot exceed the dimension of W but it can be smaller.

us

tr

ia

la

nd

Proof. Let A and B be matrices so that V = R(A) and W = R(B). Since each

element of V is also an element of W, then in particular each column of A must

be in W. Thus there is a matrix X so that A = BX. Fact 6.13 implies that

rank(A) rank(B). But from Fact 6.9 we know that rank(A) = dim(V) and

rank(B) = dim(W).

rI

nd

cie

ty

fo

So

Proof. The inequality rank(AB) rank(A) follows from R(AB) R(A) in Fact

6.3, Fact 6.12, and rank(A) = dim(R(A)). To derive rank(AB) rank(B), we

use the fact that a matrix and its transpose have the same rank, see Fact 4.14, so

that rank(AB) = rank(B A ). Now apply the first inequality.

Exercises

(i) Let A be a 17 4 matrix with linearly independent columns. Determine the

dimensions of the four spaces of A.

(ii) What can you say about the dimension of the left null space of a 25 7

matrix?

(iii) Let A Cmn . Show: If P Cmm and Q Cnn are nonsingular, then

rank(P AQ) = rank(A).

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 111

i

This electronic version is for personal use and may not be duplicated or distributed.

111

at

6.3

he

m

at

ics

(v) Let A and B be matrices with n columns, and let C and D be matrices with

n rows. Show:

AC AD

A

rank

min rank

, C D .

BC BD

B

Hint: Write A + B as product.

(viii) Let A Cmn and B Cnp . Show: If AB = 0, then rank(A) +

rank(B) n.

lie

We define operations on subspaces, so that we can relate column, row, and null

spaces to those of submatrices.

pp

of Cn . The intersection of two subspaces is defined as

la

nd

V W = {x : x V and x W},

ia

nd

us

tr

V + W = {z : z = v + w, v V and w W}.

rI

Example.

cie

ty

fo

So

and

V {0n1 } = {0n1 },

V Cn = V

V + {0n1 } = V,

V + C n = Cn .

R 0

0

0

0

0 = R 0 .

1

1

0

0

0 R 1

1

0

0

0 0

1 + R 1 0 = C3 .

0

0 1

is nonsingular, and A = A1 A2 , then

1

R 0

0

If A Cnn

n

R(A1 ) + R(A2 ) = C .

i

i

112

book

2009/5/27

page 112

i

This electronic version is for personal use and may not be duplicated or distributed.

6. Subspaces

Fact 6.15 (Intersection and Sum of Subspaces). If V and W are subspaces of

Cn , then V W and V + W are also subspaces of Cn .

at

ics

subspaces, this implies x + y V and x + y W. Hence x + y V W.

Let x, y V + W. Then x = v1 + w1 and y = v2 + w2 for some v1 , v2 V

and w1 , w2 W. Since V and W are subspaces, v1 + v2 V and w1 + w2 W.

Hence x + y = (v1 + v2 ) + (w1 + w2 ) where v1 + v2 V and w1 + w2 W.

The proofs for x where C are analogous.

he

m

With the sum of subspaces, we can express the column space of a matrix in

terms of column spaces of subsets of columns.

at

R A B = R(A) + R(B).

la

nd

pp

lie

Proof. From Definition 4.17 of a column space, and second view of matrix times

column vector in Section 1.5 we obtain the following equivalences:

x

b R A B b = A B x = Ax1 + Bx2 for some x = 1 Cn+p

x2

b = v + w where v = Ax1 R(A) and w = Bx2 R(B).

nd

us

tr

ia

Example.

Let A Cmn and B Cmp . If R(B) R(A), then

fo

rI

If A Cmn , then

m

m

R A Im = R(A) + R(Im ) = R(A) + C = C .

So

cie

ty

With the help of sums of subspaces, we can now show that the row space

and null space of an m n matrix together make up all of Cn , while the column

space and left null space make up Cm .

Fact 6.17 (Subspaces of a Matrix are Sums). If A Cmn , then

Cn = R(A ) + Ker(A),

Cm = R(A) + Ker(A ).

and

Vnr = R(V ) = Cn

Umr = R(U ) = Cm .

Fact6.18

(Intersection of Null Spaces). If A C

A

Ker

= Ker(A) Ker(B).

B

i

i

book

2009/5/27

page 113

i

This electronic version is for personal use and may not be duplicated or distributed.

113

Proof. From Definition 4.17 of a null space, and first view of matrix times column

vector in Section 1.5 we obtain the following equivalences:

A

A

Ax

x Ker

0 =

x=

Ax = 0 and Bx = 0

B

B

Bx

x Ker(A) and x Ker(B) x Ker(A) Ker(B).

he

m

at

If A Cmn , then

A

= Ker(A) Ker(0kn ) = Ker(A) Cn = Ker(A).

Ker

0kn

ics

Example.

at

A

Ker

= Ker(A) Ker(B) = Ker(A) {0n1 } = {0n1 }.

B

pp

lie

The rank of a submatrix cannot exceed the rank of a matrix. We already used

this for proving the optimality of the SVD in Fact 4.13.

nd

rank(B) rank(A).

us

tr

ia

la

Proof. Let P Cmm and Q Cnn be permutation matrices that move the

elements of B into the top left corner of the matrix,

B A12

.

P AQ =

A21 A22

cie

ty

fo

rI

nd

Since the permutation matrices P and Q can only affect singular vectors but not

singular values, rank(A) = rank(P AQ); see also Exercise (i) in Section 4.2.

We relate rank(B) to rank(P AQ) by gradually isolating B with the help of

Fact 6.18. Partition

C

P AQ =

,

where C = B A12 , D = A21 A22 .

D

So

C

Ker(P AQ) = Ker

= Ker(C) Ker(D) Ker(C).

D

Hence Ker(P AQ) Ker(C). From Fact 6.12 follows dim(Ker(P AQ))

dim(Ker(C)). We use the dimension formula in Fact 6.10 to relate the dimension of Ker(C) to rank(C),

rank(A) = rank(P AQ) = n dim(Ker(P AQ)) n dim(Ker(C)) = rank(C).

Thus rank(C) rank(A).

In order to show that rank(B) rank(C), we repeat the above argument

for C and use the fact that a matrix has the same rank as its transpose; see

Fact 4.14.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

114

book

2009/5/27

page 114

i

This electronic version is for personal use and may not be duplicated or distributed.

6. Subspaces

Exercises

(i) Solution of Linear Systems.

mn and b Cm . Show: Ax = b has a solution if and only if

Let A C

R A b = R(A).

(ii) Let A Cmn and B Cmp . Show:

Cm = R(A) + R(B) + Ker(A ) Ker(B ).

at

he

m

at

ics

(iii) Let A Cmn and B Cmp . Show that R(A) R(B) and Ker A

have the same number of elements.

(iv) Rank of Block Diagonal Matrix.

Let A Cmn and B Cpq . Show:

A 0

rank

= rank(A) + rank(B).

0 B

us

tr

ia

la

nd

pp

lie

Let A Cmn and

A11 A12

A=

,

0

A22

where A11 is nonsingular. Show: rank(A) rank(A11 ) + rank(A22 ).

Give an example to illustrate that this inequality may not hold anymore when

A11 is singular or not square.

(vi) Rank of Schur Complement.

Let A Cmn be partitioned so that

A11 A12

A=

,

A21 A22

nd

11 A12 show that

fo

Show:

rank(AB) rank(A) + rank(B) n.

cie

ty

(vii) Let A, B

rI

Cnn .

So

Intersections of subspaces can produce smaller subspaces, while sums can

produce larger subspaces.

Let V and W be subspaces of Cn . Show:

(a) V W V, and V W W.

(b) V W = V if and only if V W.

(c) V V + W, and W V + W.

(d) V + W = V if and only if W V.

1. Let A, B Cnn be idempotent and AB = BA. Show:

(a) R(AB) = R(A) R(B).

(b) Ker(AB) = Ker(A) + Ker(B).

(c) If also AB = 0, then A + B is idempotent.

i

i

book

2009/5/27

page 115

i

This electronic version is for personal use and may not be duplicated or distributed.

6.4

115

It turns out that the column and null space pairs in Fact 6.17 have only the minimal

number of elements in common. Sums of subspaces that have minimal overlap

are called direct sums.

ics

If V W = {0}, then S is a direct sum of V and W, and we write S = V W.

Subspaces V and W are also called complementary subspaces.

2

1

R

2

3

2

6

4

= C2 .

12

1

R 1

lie

pp

1

0

0

3

R 0 1 0 = C .

0

0

1

at

he

m

at

Example.

nd

us

tr

ia

la

nd

The example above illustrates that the columns of the identity matrix In form

a direct sum of Cn . In general, linearly independent columns form direct sums.

That is, in a full column rank matrix, the columns form a direct sum of the column

space.

Fact 6.21 (Full Column Rank Matrices). Let A Cmn with A = A1 A2 .

If rank(A) = n, then R(A) = R(A1 ) R(A2 ).

cie

ty

fo

rI

Proof. Fact 6.16 implies R(A) = R(A1 ) + R(A2 ). To show that R(A1 )

R(A2 ) = {0}, suppose that b R(A1 ) R(A2 ). Then b = A1 x1 = A2 x2 , and

x1

0 = A1 x1 A2 x2 = A1 A2

.

x2

So

Since A has full column rank, Fact 4.22 implies x1 = 0 and x2 = 0, hence b = 0.

We are ready to show that the row space and null space of a matrix have

only minimal overlap, and so do the column space and left null space. In other

words, for an m n matrix, row space and null space form a direct sum of Cn ,

while column space and left null space form a direct sum of Cm .

Fact 6.22 (Subspaces of a Matrix are Direct Sums). If A Cmn , then

Cn = R(A ) Ker(A),

Cm = R(A) Ker(A ).

i

i

116

book

2009/5/27

page 116

i

This electronic version is for personal use and may not be duplicated or distributed.

6. Subspaces

Since the unitary matrices V = Vr Vnr and U = Ur Umr have full column rank, Fact 6.21 implies R(A ) Ker(A) = {0n1 } and R(A) Ker(A ) =

{0m1 }.

at

he

m

1

V =R

,

W =R

.

0

ics

is only one way to complement V and fill up all of Cn . However, that is not

truethere are infinitely many complementary subspaces. Below is a very simple

example.

lie

at

This is because for = 0 the matrix

1

A=

0

nd

pp

6.21 implies V W = {0}.

ia

la

There is a particular type of direct sum, where the two subspaces are as far

apart as possible.

rI

nd

us

tr

V + W = Cn . If v w = 0 for all v V and w W, then the spaces V and W are

orthogonal subspaces. We write V = W , or equivalently, W = V .

In particular, (Cn ) = {0n1 } and {0n1 } = Cn .

So

cie

ty

fo

generalization of a unitary matrix.

R(A2 ) = R(A1 ).

Proof. Since A has full column rank, Fact 6.16 implies R(A1 ) + R(A2 ) = R(A) =

Cn . From A1 A2 = 0 follows 0 = x A1 A2 y = (A1 x) (A2 y). With v = A1 x and

w = A2 y we conclude that v w = 0 for all v R(A1 ) and w R(A2 ).

Now we come to what is sometimes referred to as the second part of the

fundamental theorem of linear algebra. It says that any matrix has two pairs of

orthogonal subspaces: column space and left null space are orthogonal subspaces,

and row space and null space are orthogonal subspaces.

Fact 6.26 (Orthogonal Subspaces of a Matrix). If A Cmn , then

Ker(A) = R(A ) ,

Ker(A ) = R(A) .

i

i

book

2009/5/27

page 117

i

This electronic version is for personal use and may not be duplicated or distributed.

117

Cn = R(A ) + Ker(A),

R(A ) = R(Vr ),

Ker(A) = R(Vnr )

and

Cm = R(A) + Ker(A ),

Ker(A ) = R(Umr ).

Vr Vnr and

Now apply Fact

U = Ur Umr .

Exercises

he

m

(ii) Show: If A Cnn is idempotent, then

at

ics

R(A) = R(Ur ),

R(A ) = R(In A).

at

R(A) = R(In A ),

ia

la

nd

pp

lie

R

, where

(iv) Let A Cmn have rank(A) = n and a QR factorization A = Q

0

Q = Qn Qmn is unitary. Show: R(A) = R(Qmn ).

(v) Let A Cmn have rank(A) = n, and let y be the solution of the least squares

problem minx Ax b2 . Show:

So

cie

ty

fo

rI

nd

us

tr

(vi) Let A Cnn be a matrix all of whose rows sum to zero. Show:

R(e) R(A ) , where e is the n 1 vector of all ones.

(vii) Orthogonal Subspaces Form Direct Sums.

Let V and W be subspaces of Cn so that W = V . Show: V W = Cn .

(viii) Direct sums of subspaces produce unique representations in the following

sense.

Let S be a subspace of Cm and S = V + W. Show: S = V W if and only

if for every b S there exist unique vectors v V and w W such that

b = v + w.

(ix) Normal Matrices.

Show: If A Cn is normal, i.e., A A = AA , then Ker(A) = R(A) .

1. Let A Cnn with

X1 AX =

A1

0

0

,

A2

then for k large enough we have Cn = R(Ak ) Ker(Ak ).

2. Properties of Orthogonal Subspaces.

Let V and W be subspaces of Cn . Show:

(a) (V ) = V.

(b) If V W, then W V .

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 118

i

This electronic version is for personal use and may not be duplicated or distributed.

118

6. Subspaces

(c) (V + W) = V W .

(d) (V W) = V + W .

6.5

Bases

ics

just a finite number. The elements of a basis are much like members of parliament,

with a few representatives standing for large constituency. A basis contains just

enough vectors to capture the whole space, but sufficiently few to avoid redundancy.

he

m

at

Definition 6.27 (Basis). The columns of a matrix W Cmn represent a basis for

a subspace S of Cm if

at

B2: R(W ) = S.

pp

lie

orthonormal basis for S.

Example.

nk

A2 ,

tr

k

A1

us

A1 =

nd

A=

ia

la

nd

A is unitary, then the columns of A represent an orthonormal basis for Cn .

Let A Cnn be nonsingular, and

k

nk

B1

.

B2

So

cie

ty

fo

rI

Then the columns of A1 represent a basis for Ker(B2 ), and the columns of

A2 represent a basis for Ker(B1 ).

This follows from Fact 6.4.

represent an orthonormal basis for Ker(U2 ), and the columns of U2 represent

an orthonormal basis for Ker(U1 ).

many different bases for V. But all bases have the same number of vectors; this

follows from Fact 6.9.

The singular vectors furnish orthonormal bases for all four subspaces of a

matrix. Let A Cmn have rank(A) = r and an SVD

r

A=U

0

0

V ,

0

U=

r

Ur

mr

Umr ,

V =

r

Vr

nr

Vnr ,

r is a diagonal matrix with

positive diagonal elements 1 r > 0.

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 119

i

This electronic version is for personal use and may not be duplicated or distributed.

6.5. Bases

119

If A = 0, then the columns of Ur represent an orthonormal basis for R(A),

and the columns of Vr represent an orthonormal basis for R(A ).

If r < n, then the columns of Vnr represent an orthonormal basis for Ker(A).

If r < m, then the columns of Umr represent an orthonormal basis for

Ker(A ).

at

ics

Proof. This follows from applying Facts 6.6 and 6.7 to A and to A .

at

he

m

Why Orthonormal Bases? Orthonormal bases are attractive because they are

easy to work with, and they do not amplify errors. For instance, if x is the solution

of the linear system Ax = b where A has orthonormal columns, then x = A b can

be determined with only a matrix vector multiplication. The bound below justifies

that orthonormal bases do not amplify errors.

pp

lie

Let z be an approximate solution with residual r = Az b. Then

nd

z x2

r2

2 (A)

,

x2

A2 x2

tr

ia

la

If A has orthonormal columns, then 2 (A) = 1.

fo

rI

nd

us

Proof. This follows from Fact 5.11. If A has orthonormal columns, then all

singular values are equal to one, see Fact 4.16, so that 2 (A) = 1 /n = 1.

ty

Exercises

So

cie

basis for R(uv ).

(ii)

Let A Cmn be nonsingular and B Cmp . Prove: The columns of

A1 B

represent a basis for Ker A B .

Ip

(iii) Let A Cmn with rank(A) = n have a QR decomposition

R

A=Q

,

0

Q=

n

Qn

mn

Qmn ,

columns of Qn represent an orthonormal basis for R(A), and the columns

of Qmn represent an orthonormal basis for Ker(A ).

Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.html

i

i

book

2009/5/27

page 120

i

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

at

ics

This electronic version is for personal use and may not be duplicated or distributed.

i

i

book

2009/5/27

page 121

i

This electronic version is for personal use and may not be duplicated or distributed.

at

lower-upper . . . . . . . . . . . . . . . 65

uniqueness . . . . . . . . . . . . . . . . 65

upper-lower . . . . . . . . . . . . . . . 67

Cholesky solver . . . . . . . . . . . . . . . 66

circular shift matrix . . . . . . . . . . . . 20

closest matrix in the two norm . . 36

column pivoting . . . . . . . . . . . . . . . 76

column space . . . . . . . . . . . . . . 86, 87

and residual . . . . . . . . . . . . . . . 93

and singular vectors . . . 87, 107

as a subspace . . . . . . . . . . . . 105

in partitioned matrix . . . . . . 107

of full rank matrix . . . . . . . . . 88

of matrix product . . . . . . . . . 106

of outer product . . . . . . . . . . 110

of transpose . . . . . . . . . . . . . . . 87

sum of . . . . . . . . . . . . . . . . . . 112

column vector . . . . . . . . . . . . . . . . . . 1

common digits . . . . . . . . . . . . . . . . . 26

commutativity . . . . . . . . . . . . . . . 4, 10

diagonal matrices . . . . . . . . . . 22

complementary subspace . see direct

sum

complete pivoting . . . . . . . . . . . . . . 58

complex. . . . . . . . . . . . . . . . . . . . . . . .1

conjugate . . . . . . . . . . . . . . . . . 12

multiplication . . . . . . . . . . . . . 11

number . . . . . . . . . . . . . . . 12, 14

condition number . . . . . . . . . . . . . . 28

bidiagonal matrix . . . . . . . . . . 50

componentwise . . . . . . . . . . . 51

least squares . . . . . . 96, 99, 100

left inverse . . . . . . . . . . . . . . . 97

linear system . . . . . . 45, 47, 48,

50, 51

So

cie

ty

fo

rI

nd

us

tr

ia

la

nd

pp

lie

at

he

m

A

absolute error . . . . . . . . . . . . . . . . . . 26

componentwise . . . . . . . . . . . 31

in subtraction . . . . . . . . . . . . . 27

normwise . . . . . . . . . . . . . . . . . 31

angle in least squares

problem . . . . . . . . . . 98, 99

approximation to 0 . . . . . . . . . . . . . 26

associativity . . . . . . . . . . . . . . . . . . 4, 9

misuse . . . . . . . . . . . . . . . . . . . 10

B

backsubstitution . . . . . . . . . . . . . . . 51

basis . . . . . . . . . . . . . . . . . . . . . . . . . 118

orthonormal . . . . . . . . . . . . . 118

battleshipcaptain analogy . . . . . . 27

Bessels inequality . . . . . . . . . . . . . 75

bidiagonal matrix . . . . . . . . . . . . . . 50

Bill Gatez . . . . . . . . . . . . . . . . . . . . . 25

block Cholesky factorization . . . . 67

block diagonal matrix . . . . . . . . . 114

block LU factorization. . . . . . . . . .61

block triangular matrix . . 52, 61, 67

rank of . . . . . . . . . . . . . . . . . . 114

ics

Index

C

canonical vector . . . . . . . . . . . 3, 7, 13

in linear combination . . . . . . . 5

in outer product . . . . . . . . . . . 14

captainbattleship analogy . . . . . . 27

catastrophic cancellation . . . . 27, 28

example of . . . . . . . . . . . . . . . 27

occurrence of . . . . . . . . . . . . . 28

CauchySchwarz inequality . . . . . 30

Cholesky factorization . . . . . . 52, 65

algorithm . . . . . . . . . . . . . . . . . 65

generalized . . . . . . . . . . . . . . . 67

121

i

i

i

122

book

2009/5/27

page 122

i

This electronic version is for personal use and may not be duplicated or distributed.

Index

direct sum . . . . . . . . . . . . . . . 115

in linear system . . . . . . . . . . . 88

least squares . . . . . . . . . . . . . . 94

MoorePenrose inverse . . . . 92

null space. . . . . . . . . . . . . . . . .88

full rank . . . . . . . . . . . . . . . . . . . . . . 84

full row rank . . . . . . . . . . . . . . . . . . 84

column space . . . . . . . . . . . . . 88

in linear system . . . . . . . . . . . 88

MoorePenrose inverse . . . . 92

fundamental theorem of linear

algebra

first part . . . . . . . . . . . . . . . . . 109

second part . . . . . . . . . . . . . . 116

So

cie

ty

fo

rI

nd

us

F

finite precision arithmetic . . . . xi, 23

fixed point arithmetic . . . . . . . . . . . xi

floating point arithmetic . . . . . xi, 26

IEEE . . . . . . . . . . . . . . . . . . . . . 26

unit roundoff . . . . . . . . . . . . . . 27

forward elimination . . . . . . . . . . . . 51

Fredholms alternatives . . . . . . . . . 89

at

he

m

at

pp

lie

G

Gaussian elimination . . . . . . . . . . . 60

growth in . . . . . . . . . . . . . . . . . 61

stability . . . . . . . . . . . . . . . . . . 60

generalized Cholesky

factorization . . . . . . . . . . 67

Givens rotation . . . . . . . . . . . . . . . . 19

in QL factorization . . . . . . . . 72

in QR factorization . . . . . 71, 74

not a . . . . . . . . . . . . . . . . . . . . . 19

order of . . . . . . . . . . . . . . . . . . 70

position of elements . . . . . . . 71

grade point average . . . . . . . . . . . . 29

nd

la

tr

ia

D

diagonal matrix . . . . . . . . . . . . . . . . 22

commutativity . . . . . . . . . . . . 22

in SVD . . . . . . . . . . . . . . . . . . . 77

diagonally dominant matrix . . . . . 42

dimension . . . . . . . . . . . . . . . . . . . . 109

of left null space . . . . . . . . . 109

of null space . . . . . . . . . . . . . 109

of row space . . . . . . . . . . . . . 109

uniqueness of . . . . . . . . . . . . 109

dimension formula . . . . . . . . . . . . 109

direct method . . . . . . . . . . . . . . . . . . 52

solution of linear systems

by . . . . . . . . . . . . . . . . . . . 52

stability. . . . . . . . . . . .54, 56, 57

direct sum . . . . . . . . . . . . . . . . . . . . 115

and least squares . . . . . . . . . 117

not unique . . . . . . . . . . . . . . . 116

of matrix columns . . . . . . . . 115

of subspaces of a matrix. . .115

unique representation . . . . . 117

distance to lower rank

matrices . . . . . . . . . . . . . 83

distance to singularity . . . . . . . . . . 41

distributivity . . . . . . . . . . . . . . . . . 4, 9

division . . . . . . . . . . . . . . . . . . . . . . . 29

dot product . . . . . . see inner product

ics

matrix addition . . . . . . . . . . . . 37

matrix inversion . . . . . . . 4042

matrix multiplication . . . . . . 37,

38, 55

normal equations . . . . . . . . . 103

scalar division . . . . . . . . . . . . 29

scalar multiplication . . . . . . . 28

scalar subtraction . . . . . . 27, 28

triangular matrix . . . . . . . . . . 52

conjugate transpose . . . . . . . . . . . . 12

inverse of . . . . . . . . . . . . . . . . . 17

H

Hankel matrix . . . . . . . . . . . . . . . . . . 3

Hermitian matrix . . . . . . . . 15, 16, 20

and direct sum . . . . . . . . . . . 117

column space . . . . . . . . . . . . . 87

inverse of . . . . . . . . . . . . . . . . . 18

null space. . . . . . . . . . . . . . . . .87

positive definite . . . . . . . . . . . 63

test for positive

definiteness . . . . . . . . . . 66

unitary . . . . . . . . . . . . . . . . . . . 20

Hermitian positive definite

matrix . . . . . . . see positive

definite matrix

Hilbert matrix . . . . . . . . . . . . . . . . . . 3

Hlder inequality . . . . . . . . . . . . . . 30

Householder reflection . . . . . . . . . 73

i

i

i

Copyright 2009 by the Society for Industrial and Applied Mathematics

This electronic version is for personal use and may not be duplicated or distributed.

123

he

m

at

ics

perturbation . . . . . . . . . . . . . . . 39

perturbed . . . . . . . . . . . . . . . . . 42

perturbed identity . . . . . . 38, 42

residuals . . . . . . . . . . . . . . . . . . 41

right . . . . . . . . . . . . . . . . . . . . . 93

singular values . . . . . . . . . . . . 78

SVD . . . . . . . . . . . . . . . . . . . . . 78

two norm . . . . . . . . . . . . . . . . . 79

invertible matrix . . . see nonsingular

matrix

involutory matrix . . . . 11, 11, 18, 20

inverse of . . . . . . . . . . . . . . . . . 16

at

K

kernel . . . . . . . . . . . . . . see null space

So

cie

ty

fo

rI

nd

us

pp

lie

L

LDU factorization . . . . . . . . . . . . . 61

least squares . . . . . . . . . . . . . . . . . . . 91

and direct sum . . . . . . . . . . . 117

angle . . . . . . . . . . . . . . . . . 98, 99

condition number . . 96, 99, 100

effect of right-hand

side . . . . . . . . . . . . . . 96, 99

full column rank. . . . . . . . . . .94

ill-conditioned . . . . . . . . . . . 100

ill-posed . . . . . . . . . . . . . . . . . . 96

rank deficient . . . . . . . . . . . . . 96

relative error . . . . . . . . . . 96, 98,

100, 101

residual . . . . . . . . . . . . . . . . . . 93

least squares residual . . . . . . . . . . . 91

conditioning . . . . . . . . . . . . . 101

least squares solutions . . . . . . . . . . 91

by QR factorization . . . . . . 102

full column rank. . . . . . . . . . .94

in terms of SVD . . . . . . 91, 102

infinitely many . . . . . . . . . . . . 92

minimal norm . . . . . . . . . . . . . 94

MoorePenrose inverse . . . . 93

left inverse . . . . . . . . . . . . . . . . . . . . 93

condition number . . . . . . . . . . 97

left null space . . . . . . . . . . . . . . . . . 87

and singular vectors . . . . . . . 87

dimension of . . . . . . . . . . . . . 109

left singular vector . . . . see singular

vector

nd

tr

ia

I

idempotent matrix . . . . . . . . . . . . . . 11

and direct sum . . . . . . . . . . . 117

and MoorePenrose

inverse . . . . . . . . . . . . . . . 94

and orthogonal

subspaces . . . . . . . . . . . 117

column space . . . . . . . . . . . . 108

norm . . . . . . . . . . . . . . . . . 36, 85

null space . . . . . . . . . . . . . . . 108

product . . . . . . . . . . . . . . . . . . . 11

singular . . . . . . . . . . . . . . . . . . 17

singular values . . . . . . . . . . . . 78

subspaces. . . . . . . . . . . . . . . .114

identity matrix . . . . 3, 13, 14, 38, 42

IEEE double precision

arithmetic . . . . . . . . . . . . 26

unit roundoff . . . . . . . . . . . . . . 27

ill-conditioned linear

system . . . . . . . . . . . 24, 25

ill-posed least squares

problem . . . . . . . . . . . . . . 96

imaginary unit . . . . . . . . . . . . . . . . . 12

infinity norm . . . . . . . . . . . . . . . 30, 34

and inner product . . . . . . . . . . 30

in Gaussian elimination . . . . 60

in LU factorization . . . . . . . . 59

one norm of transpose . . . . . 36

outer product . . . . . . . . . . . . . 36

relations with other

norms . . . . . . . . . 32, 33, 36

inner product . . . . . . . . . . . . . . . . . . . 5

for polynomial . . . . . . . . . . . . . 5

for sum of scalars . . . . . . . . . . . 5

in matrix vector

multiplication . . . . . . . 6, 7

properties . . . . . . . . . . . . . . . . . 14

intersection of null spaces . . . . . 112

intersection of subspaces . . 111, 112

in direct sum . . . . . . . . . . . . . 115

properties. . . . . . . . . . . . . . . .114

inverse

partitioned . . . . . . . . . . . . . . . 107

inverse of a matrix . . . . . . . . . . 16, 17

condition number . . . . . . 40, 46

distance to singularity . . . . . . 41

left . . . . . . . . . . . . . . . . . . . . . . . 93

la

Index

book

2009/5/27

page 123

i

i

i

i

Copyright 2009 by the Society for Industrial and Applied Mathematics

This electronic version is for personal use and may not be duplicated or distributed.

Index

So

cie

ty

fo

rI

nd

us

M

matrix . . . . . . . . . . . . . . . . . . . . . . . . . 1

distance to singularity . . . . . . 41

equality . . . . . . . . . . . . . . . . . . . 8

pp

lie

at

he

m

at

ics

full rank . . . . . . . . . . . . . . . . . . 84

idempotent . . . . . . . . . . . . . . 114

multiplication . . . . . . . . . . . . . 38

nilpotent . . . . . . . . . . . . . 85, 117

normal . . . . . . . . . . . . . . . 89, 117

notation . . . . . . . . . . . . . . . . . . . 2

powers . . . . . . . . . . . . . . . . 11, 18

rank . . . . . . . . . . . . . . . . see rank

rank deficient . . . . . . . . . . . . . 84

skew-symmetric . . . . . . . . . . . 18

square . . . . . . . . . . . . . . . . . . . . . 1

vector multiplication . . . . . .6, 9

with linearly dependent

columns . . . . . . . . . . . . . 73

with linearly independent

columns . . . . . . . . . . . . . 73

with orthonormal

columns . . . . . . . . . . 75, 85

matrix addition . . . . . . . . . . . . . . . . . 4

condition number . . . . . . . . . . 37

matrix inversion . . . see inverse of a

matrix

matrix multiplication . . . . . . . . . . . . 9

condition number . . . . . . 37, 38,

46, 55

matrix norm . . . . . . . . . . . . . . . . . . . 33

well-conditioned . . . . . . . . . . 33

matrix product . . . . . . . . . . . . . . . . . . 9

column space . . . . . . . . . . . . 106

condition number . . . . . . . . . . 37

null space . . . . . . . . . . . . . . . 106

rank of . . . . . . . . . . . . . . . 85, 110

maximum norm . . see infinity norm

minimal norm least squares solution

94

MoorePenrose inverse . . . . . . . . . 92

column space . . . . . . . . . . . . . 94

defining properties . . . . . . . . . 95

in idempotent matrix . . . . . . . 94

in least squares . . . . . . . . . . . . 93

norm . . . . . . . . . . . . . . . . . 94, 95

null space. . . . . . . . . . . . . . . . .94

of full rank matrix . . . . . . . . . 92

of nonsingular matrix . . . . . . 94

of product . . . . . . . . . . . . . . . . 95

orthonormal columns . . . . . . 95

outer product . . . . . . . . . . . . . 94

nd

tr

ia

linear combination . . . . . . . . . . . . 5, 8

in linear system . . . . . . . . . . . 43

in matrix vector

multiplication . . . . . . . 6, 7

linear system . . . . . . . . . . . . . . . . . . 43

condition number . . . . . . 4548,

50, 51

effect of right-hand

side . . . . . . . . . . . . . . 49, 50

full rank matrix . . . . . . . . . . . 88

ill-conditioned . . . . . . . . . . . . 46

nonsingular . . . . . . . . . . . . . . . 43

perturbed . . . . . . . . . . . . . . . . . 44

relative error . . . . . . 45, 47, 48,

50, 51

residual . . . . . . . . . . . . . . . . . . 44

residual bound . . . . . 45, 47, 50

triangular . . . . . . . . . . . . . . . . . 45

linear system solution . . . . . . 43, 114

by direct method . . . . . . . . . . 52

by Cholesky factorization . . 66

by LU factorization . . . . . . . . 60

by QR factorization . . . . . . . . 68

full rank matrix . . . . . . . . . . . 88

nonsingular matrix . . . . . . . . 43

what not to do . . . . . . . . . . . . . 57

linearly dependent columns . . . . . 73

linearly independent columns . . . 73

test for . . . . . . . . . . . . . . . . . . . 74

lower triangular matrix . . . . . . . . . 20

forward elimination . . . . . . . . 51

in Cholesky factorization . . . 65

in QL factorization . . . . . . . . 72

lower-upper Cholesky

factorization . . . . . . . . . . 65

LU factorization . . . . . . . . . . . . 52, 58

algorithm . . . . . . . . . . . . . . . . . 59

in linear system solution . . . 60

permutation matrix . . . . . . . . 59

stability . . . . . . . . . . . . . . . . . . 60

uniqueness . . . . . . . . . . . . . . . . 21

with partial pivoting . . . . 59, 60

la

124

book

2009/5/27

page 124

i

i

i

Index

book

2009/5/27

page 125

i

This electronic version is for personal use and may not be duplicated or distributed.

125

partial isometry . . . . . . . . . . . 95

partitioned . . . . . . . . . . . . . . . . 95

QR factorization . . . . . . . . . . 95

multiplication . . . . . . . . . . . . . . . . . 28

multiplier in LU factorization . . . 59

So

cie

ty

fo

rI

nd

us

pp

lie

at

he

m

at

ics

O

one norm . . . . . . . . . . . . . . . . . . 30, 34

and inner product . . . . . . . . . . 30

relations with other

norms . . . . . . . . . 32, 33, 36

orthogonal matrix . . . . . . . . . . . . . . 19

in QR factorization . . . . . . . . 21

orthogonal subspace . . . . . . . . . . . 116

and QR factorization . . . . . . 117

direct sum . . . . . . . . . . . . . . . 117

properties. . . . . . . . . . . . . . . .117

subspaces of a matrix . . . . . 116

orthonormal basis . . . . . . . . . . . . . 118

orthonormal columns . . . . . . . . . . . 75

condition number . . . . . . . . 100

in polar decomposition . . . . . 85

MoorePenrose inverse . . . . 95

singular values . . . . . . . . . . . . 85

outer product . . . . . . . . . . . . . 8, 9, 14

column space . . . . . . . . . . . . 110

in Schur complement . . . . . . 59

in singular matrix . . . . . . . . . 40

infinity norm . . . . . . . . . . . . . . 36

MoorePenrose inverse . . . . 94

null space . . . . . . . . . . . . . . . 110

of singular vectors . . . . . . . . . 82

rank . . . . . . . . . . . . . . . . . 81, 110

SVD . . . . . . . . . . . . . . . . . . . . . 81

two norm . . . . . . . . . . . . . . . . . 36

nd

la

tr

ia

N

nilpotent matrix . . . . 11, 18, 85, 117

singular . . . . . . . . . . . . . . . . . . 18

strictly triangular . . . . . . . . . . 21

nonsingular matrix . . . . . . 16, 17, 18,

42, 43

condition number w.r.t.

inversion . . . . . . . . . . . . . 40

distance to singularity . . . . . . 41

LU factorization of . . . . . . . . 60

positive definite . . . . . . . . . . . 63

product of . . . . . . . . . . . . . . . . 18

QL factorization of . . . . . . . . 72

QR factorization of . . . . . . . . 68

triangular . . . . . . . . . . . . . . . . . 21

unit triangular . . . . . . . . . . . . . 21

norm . . . . . . . . . . . . . . . . . . . . . . 29, 33

defined by matrix . . . . . . . . . . 33

of a matrix . . . . . . . . . . . . . . . . 33

of a product . . . . . . . . . . . . . . . 35

of a submatrix . . . . . . . . . . . . . 35

of a vector . . . . . . . . . . . . . . . . 29

of diagonal matrix . . . . . . . . . 36

of idempotent matrix . . . . . . 36

of permutation matrix . . . . . . 36

reverse triangle

inequality . . . . . . . . . . . . 32

submultiplicative . . . . . . 34, 35

unit norm . . . . . . . . . . . . . . . . . 30

normal equations . . . . . . . . . . . . . 103

instability . . . . . . . . . . . . . . . 103

normal matrix . . . . . . . . . . . . . 89, 117

notation . . . . . . . . . . . . . . . . . . . . . . . . 2

null space . . . . . . . . . . . . . . . . . . 86, 87

and singular vectors . . . 87, 108

as a subspace . . . . . . . . . . . . 105

dimension of . . . . . . . . . . . . . 109

in partitioned matrix . . . . . . 107

intersection of . . . . . . . . . . . . 112

of full rank matrix . . . . . . . . . 88

of outer product . . . . . . . . . . 110

of product . . . . . . . . . . . . . . . 106

of transpose . . . . . . . . . . . . . . . 87

numerical stability . . . . . see stability

P

p-norm . . . . . . . . . . . . . . . . . . . . 30, 33

with permutation matrix . . . . 32

parallelogram equality . . . . . . . . . . 32

partial isometry . . . . . . . . . . . . . . . . 95

partial pivoting . . . . . . . . . . . . . 58, 59

partitioned inverse . . . . . 17, 18, 107

partitioned matrix . . . . . . . . . . . . . . 20

permutation matrix . . . . . . . . . . . . . 19

in LU factorization . . . . . . . . 59

partitioned . . . . . . . . . . . . . . . . 20

product of . . . . . . . . . . . . . . . . 20

transpose of . . . . . . . . . . . . . . . 20

perturbation . . . . . . . . . . . . . . . . . . . 23

i

i

i

Copyright 2009 by the Society for Industrial and Applied Mathematics

This electronic version is for personal use and may not be duplicated or distributed.

Index

us

So

cie

ty

fo

rI

nd

Q

QL factorization . . . . . . . . . . . . . . . 72

QR factorization . . . . . . . . . . . . 52, 68

algorithm . . . . . . . . . . . . . 71, 74

and orthogonal subspace . . 117

column space . . . . . . . . . . . . 108

MoorePenrose inverse . . . . 95

null space . . . . . . . . . . . . . . . 108

orthonormal basis . . . . . . . . 119

rank revealing . . . . . . . . . . . . . 86

thin . . . . . . . . . . . . . . . . . . . 74, 75

uniqueness . . . . . . . . . . . . 21, 68

with column pivoting . . . . . . 76

QR solver . . . . . . . . . . . . . . . . 68, 102

R

range . . . . . . . . . . . see column space

rank . . . . . . . . . . . . . . . . . . . . . . . . . . 81

and reduced SVD. . . . . . . . . .82

deficient . . . . . . . . . . . . . . . . . . 84

full. . . . . . . . . . . . . . . . . . . . . . .84

pp

lie

at

he

m

at

ics

of a submatrix . . . . . . . . . . . 113

of block diagonal matrix . . 114

of block triangular

matrix . . . . . . . . . . . . . . 114

of matrix product . . . . . 85, 110

of outer product . . . . . . . 81, 110

of Schur complement . . . . . 114

of transpose . . . . . . . . . . . . . . . 84

of zero matrix . . . . . . . . . . . . . 81

real . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

reduced SVD . . . . . . . . . . . . . . . . . . 82

reflection . . . . . . . . . . . . . . . . . . . . . . 19

Householder . . . . . . . . . . . . . . 73

relative error . . . . . . . . . . . . . . . . . . 26

componentwise . . . . . . . . . . . 31

in subtraction . . . . . . . . . . . . . 28

normwise . . . . . . . . . . . . . . . . . 31

relative perturbation . . . . . . . . . . . . 27

residual . . . . . . . . . . . . . . . . . . . . 44, 91

and column space . . . . . . . . . 93

computation of . . . . . . . . . . . 102

large norm . . . . . . . . . . . . . . . . 97

matrix inversion . . . . . . . . . . . 41

norm . . . . . . . . . . . . . . . . . 4547

of a linear system . . . . . . . . . . 44

of least squares problem . . . 91,

101

relation to perturbations . . . . 44

small norm . . . . . . . . . . . . . . . 45

uniqueness . . . . . . . . . . . . . . . . 93

residual bound . . . . . 45, 47, 50, 101

right inverse . . . . . . . . . . . . . . . . . . . 93

right singular vector . . . see singular

vector

row space . . . . . . . . . . . . . . . . . . . . . 87

and singular vectors . . . . . . . 87

dimension of . . . . . . . . . . . . . 109

row vector . . . . . . . . . . . . . . . . . . . . . . 1

nd

tr

ia

pivot . . . . . . . . . . . . . . . . . . . . . . . . . . 59

polar decomposition . . . . . . . . . . . . 85

closest unitary matrix . . . . . . 85

polar factor . . . . . . . . . . . . . . . . . . . . 85

polarization identity . . . . . . . . . . . . 33

positive definite matrix . . . . . . . . . 63

Cholesky factorization . . 65, 66

diagonal elements of . . . 63, 67

generalized Cholesky

factorization . . . . . . . . . . 67

in polar decomposition . . . . . 85

lower-upper Cholesky

factorization . . . . . . . . . . 65

nonsingular . . . . . . . . . . . . . . . 63

off-diagonal elements of . . . 66

principal submatrix of . . . . . 64

Schur complement of . . . . . . 64

SVD . . . . . . . . . . . . . . . . . . . . . 78

test for . . . . . . . . . . . . . . . . . . . 66

upper-lower Cholesky

factorization . . . . . . . . . . 67

positive semidefinite matrix . . . . . 63

principal submatrix . . . . . . . . . . . . . . 2

positive definite . . . . . . . . . . . 64

product of singular values . . . . . . . 80

Pythagoras theorem . . . . . . . . . . . . 32

la

126

book

2009/5/27

page 126

i

S

scalar . . . . . . . . . . . . . . . . . . . . . . . . . . 1

scalar matrix multiplication . . . . . . 4

Schur complement . . . . . . 18, 59, 61

positive definite . . . . . . . . . . . 64

rank . . . . . . . . . . . . . . . . . . . . . 114

sensitive . . . . . . . . . . . . . . . . . . . xi, 23

i

i

Copyright 2009 by the Society for Industrial and Applied Mathematics

This electronic version is for personal use and may not be duplicated or distributed.

127

So

cie

ty

fo

rI

nd

us

pp

lie

at

he

m

at

ics

submatrix . . . . . . . . . . . . . . . . . . . . . . 2

principal . . . . . . . . . . . . . . . . . . . 2

rank of . . . . . . . . . . . . . . . . . . 113

submultiplicative

inequality . . . . . . . . . 34, 35

subspace . . . . . . . . . . . . . . . . . . . . . 105

complementary. . . . . . . . . . .115

dimension of . . . . . . . . . . . . . 109

direct sum . . . . . . . . . . . . . . . 115

intersection . . . . . . . . . . . . . . 111

orthogonal . . . . . . . . . . . . . . . 116

real . . . . . . . . . . . . . . . . . . . . . 106

sum . . . . . . . . . . . . . . . . . . . . . 111

subtraction . . . . . . . . . . . . . . . . . . . . 27

absolute condition

number . . . . . . . . . . . . . . 27

absolute error . . . . . . . . . . . . . 27

relative condition number . . 28

relative error . . . . . . . . . . . . . . 28

sum of subspaces . . . . . . . . . 111, 112

column space . . . . . . . . . . . . 112

of a matrix . . . . . . . . . . . . . . . 112

properties. . . . . . . . . . . . . . . .114

SVD . . . . . . . . . . . . . . . . . . . . . . . . . . 77

of inverse . . . . . . . . . . . . . . . . . 78

of positive definite

matrix . . . . . . . . . . . . . . . 78

of transpose . . . . . . . . . . . . . . . 77

optimality . . . . . . . . . . . . . . . . 83

reduced . . . . . . . . . . . . . . . . . . 82

singular vectors . . . . . . . . . . . 81

symmetric matrix . . . . . . . . . . . 15, 16

diagonal . . . . . . . . . . . . . . . . . . 22

inverse of . . . . . . . . . . . . . . . . . 18

nd

tr

ia

ShermanMorrison

formula. . . . . . . . . . .17, 18

shift matrix . . . . . . . . . . . . . . . . . . . . 13

circular . . . . . . . . . . . . . . . . . . . 20

singular matrix . . . . . . . . . . . . . 16, 17

distance to . . . . . . . . . . . . . . . . 41

singular value decomposition . . . see

SVD

singular values . . . . . . . . . . . . . . . . 77

relation to condition

number . . . . . . . . . . . . . . 79

conditioning of . . . . . . . . . . . . 80

extreme . . . . . . . . . . . . . . . . . . 79

matrix with orthonormal

columns . . . . . . . . . . . . . 85

of 2 2 matrix . . . . . . . . . . . . 78

of idempotent matrix . . . . . . 78

of inverse . . . . . . . . . . . . . . . . . 79

of product . . . . . . . . . . . . . 78, 80

of unitary matrix . . . . . . . . . . 78

relation to rank . . . . . . . . . . . . 81

relation to two norm . . . . . . . 79

uniqueness . . . . . . . . . . . . . . . . 77

singular vector matrix . . . . . . . . . . 77

singular vectors . . . . . . . . . . . . . . . . 81

column space . . . . . . . . . . . . 107

in outer product . . . . . . . . . . . 82

in reduced SVD . . . . . . . . . . . 82

left . . . . . . . . . . . . . . . . . . . . . . . 81

null space . . . . . . . . . . . . . . . 108

orthonormal basis . . . . . . . . 118

relations between . . . . . . . . . . 82

right . . . . . . . . . . . . . . . . . . . . . 81

skew-Hermitian matrix . . . . . . . . 15,

15, 16

skew-symmetric matrix . . . . . 15, 15,

16, 18

stability . . . . . . . . . . . . . . . . . . . . . . . 54

of Cholesky solver . . . . . . . . 66

of direct methods . . . 54, 56, 57

of Gaussian elimination . . . . 60

of QR solver . . . . . . . . . . . . . . 68

steep function . . . . . . . . . . . . . . . . . 24

strictly column diagonally dominant

matrix . . . . . . . . . . . . . . . 42

strictly triangular matrix . . . . . . . . 21

nilpotent . . . . . . . . . . . . . . . . . . 21

la

Index

book

2009/5/27

page 127

i

T

thin QR factorization . . . . . . . . . . . 74

uniqueness . . . . . . . . . . . . . . . . 75

Toeplitz matrix . . . . . . . . . . . . . . . 3, 7

transpose . . . . . . . . . . . . . . . . . . 12, 85

inverse of . . . . . . . . . . . . . . . . . 17

of a product . . . . . . . . . . . . . . . 13

partial isometry . . . . . . . . . . . 95

rank . . . . . . . . . . . . . . . . . . . . . . 84

SVD . . . . . . . . . . . . . . . . . . . . . 77

triangle inequality . . . . . . . . . . . . . . 29

reverse . . . . . . . . . . . . . . . . . . . 32

i

i

Copyright 2009 by the Society for Industrial and Applied Mathematics

This electronic version is for personal use and may not be duplicated or distributed.

Index

So

cie

ty

fo

rI

nd

us

U

UL factorization . . . . . . . . . . . . . . . 62

uncertainty . . . . . . . . . . . . . 23, 27, 44

unit roundoff . . . . . . . . . . . . . . . . . . 27

pp

lie

at

he

m

at

ics

in LU factorization . . . . . 21, 58

inverse of . . . . . . . . . . . . . . . . . 21

unit-norm vector . . . . . . . . . . . . . . . 30

unitary matrix . . . . . . . . . . 19, 41, 85

2 2 . . . . . . . . . . . . . . . . . . . . . 19

closest . . . . . . . . . . . . . . . . . . . 85

Givens rotation . . . . . . . . . . . . 19

Hermitian . . . . . . . . . . . . . . . . 20

Householder reflection . . . . . 73

in QL factorization . . . . . . . . 72

in QR factorization . . . . . 21, 68

in SVD . . . . . . . . . . . . . . . . . . . 77

partitioned . . . . . . . . . . . . . . . . 20

product of . . . . . . . . . . . . . . . . 20

reflection . . . . . . . . . . . . . . . . . 19

singular values . . . . . . . . . . . . 78

transpose of . . . . . . . . . . . . . . . 20

triangular . . . . . . . . . . . . . . . . . 22

upper triangular matrix . . . . . . . . . 20

backsubstitution . . . . . . . . . . . 51

bound on condition

number . . . . . . . . . . . . . . 52

in Cholesky

factorization . . . . . . 65, 67

in LU factorization . . . . . 21, 58

in QR factorization . . . . . 21, 68

linear system solution . . . . . . 51

nonsingular . . . . . . . . . . . . . . . 21

upper-lower Cholesky

factorization . . . . . . . . . . 67

nd

tr

ia

triangular matrix . . . . . . . . . . . . . . . 20

bound on condition

number . . . . . . . . . . . . . . 52

diagonal . . . . . . . . . . . . . . . . . . 22

ill-conditioned . . . . . . . . . . . . 47

in Cholesky

factorization . . . . . . 65, 67

in QL factorization . . . . . . . . 72

in QR factorization . . . . . . . . 68

linear system solution . . . . . . 51

nonsingular . . . . . . . . . . . . . . . 21

two norm . . . . . . . . . . . . . . . . . . 30, 79

and inner product . . . . . . . . . . 30

CauchySchwarz

inequality . . . . . . . . . . . . 30

closest matrix . . . . . . . . . . . . . 36

closest unitary matrix . . . . . . 85

condition number for

inversion . . . . . . . . . . . . . 79

in Cholesky solver . . . . . . . . . 66

in least squares problem . . . . 91

in QR solver . . . . . . . . . . . . . . 68

of a unitary matrix . . . . . . . . . 36

of inverse . . . . . . . . . . . . . . . . . 79

of transpose . . . . . . . . . . . . . . . 35

outer product . . . . . . . . . . . . . 36

parallelogram inequality . . . 32

polarization identity . . . . . . . 33

relation to singular

values . . . . . . . . . . . . . . . 79

relations with other

norms . . . . . . . . . . . . 33, 36

theorem of Pythagoras . . . . . 32

with unitary matrix . . . . . . . . 32

la

128

book

2009/5/27

page 128

i

V

Vandermonde matrix . . . . . . . . . . 3, 8

vector norm . . . . . . . . . . . . . . . . . . . 29

Z

zero matrix . . . . . . . . . . . . . . . . . . . . . 2

(a)

i

i

- Numerical Methods for Large Eigenvalue ProblemsEnviado porQuinton Boltin
- PythonEnviado porPraveen Pvsm
- 11 Planetary Science Data DictionaryEnviado porMarcus Skookumchuck Vannini
- E&G – Quaternary Science Journal Vol. 59 No 1- 2Enviado porGeozon Science Media
- Introduction to Python BasicsEnviado porluis900000
- E&G - Quaternary Science Journal Vol. 60 No 1: Loess in EuropeEnviado porGeozon Science Media
- Numerical Model TurnerEnviado porChandan Kumar Ray
- Numerical DiffusionEnviado porKenny Mustapha
- Fire History and Climate Change:A Synthesis of KnowledgeEnviado porJoint Fire Science Program
- A Critique of Piketty's "Capital in the Twenty-first Century"Enviado porzadanliran
- Longitudinal and Panel Data Analysis and Applications in the Social SciencesEnviado porMin Huang
- Introduction to Scientific Computing Using PythonEnviado porGonzalo Andrés Peña
- Python ManEnviado poriordache
- Kirghiz and Wakhi of Afghanistan(2002) by ShahraniEnviado porBilal Afridi
- Extreme Fire Behavior: A Synthesis of Knowledge for Fire Managers,Vol.1Enviado porJoint Fire Science Program
- The Bible, The Qur'an and Science - Maurice Bucaille - XKPEnviado porIslamicMobility
- Iop - Ng G W - Intelligent Systems; Fusion, Tracking and Control (2003)Enviado porMohanad Nasreldean
- NumericalIsingModelEnviado porOliver Bardin
- Theory and Applications of Numerical AnalysisEnviado porbstockus
- Numerical Solution ADR Equations_HundsdorferEnviado porKarthik Vazhuthi
- Numerical Differentiation & IntegrationEnviado porUmair Ahmed
- Numerical ControllerEnviado porDhaval Patel
- Europe in figures – Eurostat yearbook 2012Enviado porcavris
- Anatoly Fomenko History Fiction or Science 3Enviado porcesar7771
- Robert Peal - Progressively WorseEnviado porstuffer27
- Open Notebook Science Melting Point DataEnviado poraclinio
- Python for AndroidEnviado porSimpson Macie
- An Intuitive Guide to Numerical Methods HeinoldEnviado porSharad Kumar Kaushal
- 2015-FIeld-Guide-To-Data-Science.pdfEnviado porLordger Liu
- Structural Design of Reinforced Concrete Pile CapsEnviado porapirakq

- Calc ReportEnviado porbitconcepts9781
- Design of Optimum HeatsinkEnviado porbitconcepts9781
- Mil Hdbk 660bEnviado porbitconcepts9781
- Slides 1Enviado porbitconcepts9781
- Mathematical Methods in CombinatoricsEnviado porAaron Merrill
- Electromagnetics and OpticsEnviado porChacko Mathew
- Stops & Pupil EntranceEnviado porbitconcepts9781
- soil properties.pdfEnviado porbitconcepts9781
- A-Complex Numbers Phasors and Circuts.pdfEnviado porSajjad Fattah
- General Design Principles for BearingsEnviado pormehdi7891
- 10175_ Industrial SealsTechManualEnviado porAlvaro
- Astronomy Without a TelescopeEnviado porbitconcepts9781
- Thesis Capstan DesignEnviado porbitconcepts9781
- Design Calculation and Performance Testing of Heating Coil in Induction Surface Hardening MachineEnviado porAdityaRaje
- Induction Heating Coil DesignEnviado porbitconcepts9781
- Telescope PlansEnviado poripavlov
- 350504main_Optics_Building_a_Telescope.pdfEnviado porbitconcepts9781
- Matlab ProgEnviado porbitconcepts9781
- An 1067 Phase Noise CaLCSEnviado porbitconcepts9781
- AntennasEnviado porRaveendra Moodithaya
- Physics 104 Optics NewEnviado porbitconcepts9781
- Boddeker's PHY132 Lecture14Enviado porbitconcepts9781
- Numerical Solution of Laplaces EquationEnviado porbitconcepts9781
- PPLATO _ Basic Mathematics _ Gradients and Directional DerivativesEnviado porbitconcepts9781
- Complex Numbers SolEnviado porbitconcepts9781
- (eBook - Science) Principles of Modern Physics (1)Enviado porkeither713
- 5-Modeling and CFD-Simulation of Woven TextilesEnviado porbitconcepts9781
- Electro MagEnviado porbitconcepts9781
- Met a MaterialsEnviado porbitconcepts9781
- Gillespie Aachen CP vs P OpticsEnviado porbitconcepts9781

- Islam and Contemporary ScienceEnviado porDynamic Sunni Orthodoxy
- Stanet English PrintEnviado porcaligulaminus
- Customized Image Reference GuideEnviado porJesus Lopez
- 3018_0Enviado porMitha Kim Nam Gil
- John Passmore - Reading and Remembering (3ed, 1944) @@Enviado porVitali
- CYCLE2Enviado porvinoth
- CLOUD VISION TECHNOLOGY (Myanmar ).pdfEnviado porHan Myo
- affective capitalism.pdfEnviado porAadad
- BCM_BuildandConst_3_2009_V6Enviado porDevid Nongthombam
- A Pipelined FFT Architecture for Real-Valued SignalsEnviado porBharath S L Bharath
- 9084-ErEEFBh.pptxEnviado porNabeel Khan
- Gen Control Paralleling SystemEnviado porSunny Kumar
- Journal of Structured Finance AcuLeadEnviado porAtanu Mukherjee
- SCJP Parcial 2013Enviado porvimale8
- Workshop-Paper-7-ISCC-and-MSPO-Compliance-for-Certification.pdfEnviado porFazrizza Jalil
- Huawei D105 MiFi Modem-En-UserGuideEnviado porCarlos Rodrigues
- Organic farming and the sustainability of agricultural systemsEnviado porRo Xa
- Pryce on DiracEnviado porRay Mondo
- Electrolytic ConductanceEnviado porBapu Thorat
- Shell Scripts ExamplesEnviado porcezaurus
- Two Pass AssemblerEnviado porWasim Mirza
- Clo2 and Corrosion ScalesEnviado porpi
- Lecture 19 Rapid prototyping concept, advantages.pdfEnviado porRishabh Mendiratta
- 10.2 GravityEnviado porConor Murray
- Oracle 11gr2 Single Node Rac InstallationEnviado porVijay Shankar
- C++ Summarized NotesEnviado porchandni1972
- EDC-S User ManualEnviado porsharfiz
- technology workshop lesson planEnviado porapi-290468840
- SRDF-Interfamily-Connectivity-Information.pdfEnviado porFarha Azad
- 7th House RulershipEnviado porALCHEMISTT

## Muito mais do que documentos

Descubra tudo o que o Scribd tem a oferecer, incluindo livros e audiolivros de grandes editoras.

Cancele quando quiser.