7.

3 Positive Definite Matrices

Definition
Very frequently in engineering and statistics applications,
we end up dealing with symmetric (& square) matrices
which have positive eigenvalues. These matrices are said to
be positive definite.

Theorem 1
If A is positive definite, then it is invertible and det A > 0.
Proof: A (n x n) with eigenvalues 1, 2,, n
det A = 12n > 0 (since all eigenvalues are pos)
Since det A 0, A is invertible.

Theorem 2
A symmetric matrix A is positive definite iff XTAX > 0 for every
column X 0 in n.
Proof: let PTAP = D = diag(1, 2,, n) where P-1=PT and the i are the
eigenvalues of A (since princ axes thm says that symmetric A is
orthogonally diagonalizable).
Let Y = PTX = [y1 y2yn]T and we know A=PDP-1
so XTAX = XT PDP-1 X=YTDY=1y12++ nyn2 (*)
i>0 since A is pos def. Also, if Y=0, we have the homogeneous
system PTX=0 which only has the solution X=0 since P is invertible.
But X0 so Y0. This implies that some y i0, so XTAX > 0 (from *
above)

Theorem 2 (cont)
Given XTAX >0 for every X0, we will select an X0 for
which each i0: let X = PEj 0 (where Ej is col j of In-- so
we have a col of P). Then Y = PTX = PTPEj=P-1PEj=Ej
So recall (*) XTAX = XT PDP-1 X=YTDY=1y12++ nyn2 >0
Becomes: j >0 which tells us that A is positive definite.

Example
IfA=CTCwhereCisinvertible,showthatAispositivedefinite.
Solution:TrytoshowthatXTAX>0forallX0.
XTAX=XTCTCX=(CX)T(CX)=CXCX=||CX||2>0:reason:
withCinvertible,theonlysolutiontothehomogeneoussystem
CX=0isthetrivialsolutionX=0,butX0,soCX0,which
means||CX||2>0.

Definition
Every positive definite matrix A can be factored as A=UTU,
where U is upper triangular with positive entries on the main
diagonal--called the Cholesky Decomposition (which is a
unique factorization).
We find the Cholesky Decomposition using elementary row
operations (as in the next example).

Cholesky Decomposition Algorithm

First be sure that your matrix A is positive definite. Then:
1. Add multiples of a row to lower rows in order to carry A to an
upper triangular matrix U1. (Only multiply the higher row times
a value.)
2. Divide the entries in each row by the square root of the
diagonal entry in the row.
You will now have U such that A=UTU. It gets messy to verify
this in the general case, but we can show it works for various pos.
def. matrices.

Example
Find the Cholesky Decomposition of
Weonlyaddmultiplesofarowto
rowsbelowit.
2

4 3
0 11
4

CanverifythatUTU=A

4 3
A

3
5

3
2
11

Definition
If A is (n x n), and 1r n, let rA be the (r x r) matrix obtained from A by
deleting the last (n - r) rows and columns. The matrices 1A, 2A,,nA = A
are called the principle submatrices of A.

a11
A a11 , A
a21
2

a11 a12
a12 3

,
A

a21 a 22

a22
a31 a32

a13

a23
a33

Theorem 3
If A is positive definite, so is rA for each r = 1,2,,n
Proof:

r A P
Y
r
n
A
,letY

;
X

0
Q
R

0<XTAX(sinceAisposdef)
X T AX Y T

r A P Y
T r
0

Y
AY

Q R 0

SoYTrAY>0sorAispositivedefinite.

Lemma
LetAbean(nxn)matrixforwhichdetrA>0foreachr=1,2,
,n.ThenLA=UwhereLis(nxn)lowertriangularwith1s
onthediagonal,andUis(nxn)uppertriangularwithpositive
entriesonthediagonal.
Proof:(byinduction).
n=1:A=[a],a>0,soletL=[1]andU=A.
n>1:letA=[aij]andnotethata11=det1A>0.SoAcanbetaken
byrowoperationsthataddmultiplesofrow1tolowerrowsto:
a11
0

X
B

Therowoperationstoaccomplishthisareleftmult
byelementarymatrixthatislowertriangularwith
1sonthediagonal.(verify)

Proof (cont)
LetL1betheproductoftheseelementarymatrices,whichwill
alsobelowertriangularwith1sonthemaindiagonal.(verify)
a11 X
L1 A

0 B
WecouldalsowriteL1andAinblockform:
r L1
L1 A
M

0 r A C r L1 r A P

N D E Q
R

LookingatthetwoversionsofL1Aifr2,wecanseethat
r
a11 X a11 X
r
r
r
L1 A (L1 A)

r1
B
0 B 0

Proof (cont)
r

a11
L1 A (L1 A)
0
r

X a11

B 0

X
r1
B

det(rL1rA)= a11det(r-1B)
Since det rL1=1 (since L1 is triangular w/ 1s on diag),
det(rL1rA)= det(rL1) det( rA)= detrA
So a11det(r-1B)= detrA
Since detrA>0 (given) and a11>0 (given), det(r-1B)>0.

Proof (cont)
By induction, let L0B=U0 be as in the lemma.
If L2=

1 0
1 0 a11
0 L ,then : L2 L1 A 0 L 0

X a11 X

B 0 U0

L2L1AisuppertriangularwithpositivediagonalentriessinceU0
isuppertriangularwithposentriesondiag(given)anda11>0.
SinceL2L1islowertriangularwith1sonthediagonal,
LetL=L2L1andU=L2L1Awhichsatisfiesthecondition.

Theorem 4
The following conditions are equivalent for an (n x n) symmetric
matrix A:
1. A is positive definite.
2. det rA>0 for each r=1,2,,n
3. A = UTU where U is upper triangular with positive diagonal
entries.
Proof: (1) (2): thm 3 says that if A is pos. def, then so is rA for
each r=1,2,,n and thm 1 says that if A is pos def, then det A>0.
So done.

Proof (cont)
(2)(3): Let LA = U0 as in the previous lemma (i.e U0 is upper triangular w/ pos entries on the diagonal, L is lower tri w/ 1s on diag.)
Let D = diag(d1,d2,,dn) be the diagonal mtx w/ the same diagonal as U 0.
Then U1=D-1U0 is upper triangular w/ 1s on the diagonal (we can show this). Then also U 0 = DU1
LA = U0 A =L-1U0 = L-1DU1
Also, A = AT = (L-1DU1)T = U1TDT(L-1)T = U1TD(LT)-1

Proof (cont)
So L-1DU1 = A = U1TD(LT)-1
So DU1=LU1TD(LT)-1
So DU1LT=LU1TD
So DU1LTD-1=LU1T
DU1LTD-1 is upper triangular since D is diagonal, U1 is upper triangular, and LT is upper triangular.
LU1T is lower triangular w/ 1s on main diag since both L and U 1T are lower tri with 1s on the main diagonal (verify).
Since DU1LTD-1=LU1T they are upper and lower triang (so diag) w/ 1s on diag (thus we have I n)

Proof (cont)
SoLU1T=InL1=U1T
LetD0=

diag( d1 ,..., dn )

SoD02=D.
ThenU=D01U0isuppertriangularwithpositivediagonal
entriessinceD01isdiagonalw/posentriesandU0isupper
triangularw/posentriesondiagonal.
AndUTU=(D01U0)T(D01U0)=U0T(D01)(D01)U0
(D01)2isdiagonalw/1/diondiagwhichisD1
T

Proof (cont)
(3)(1) basically the same as in our first example.
Also, proofs of the lemma and theorems in this section prove the legitimacy of the Algorithm for the Cholesky Decomposition.