Escolar Documentos
Profissional Documentos
Cultura Documentos
Application in Statistics
Part II
Chaiyaporn Khemapatapan, Ph.D
DPU
Overview
Introduction
LU decomposition
QR decomposition
Cholesky decomposition
Jordan Decomposition
Spectral decomposition
Singular value decomposition
Applications
2
Multiplicity
Algebraic Multiplicity: The number of repetitions of a
certain eigenvalue. If, for a certain matrix, ={3,3,4}, then
the algebraic multiplicity of 3 would be 2 (as it appears twice)
and the algebraic multiplicity of 4 would be 1 (as it appears
once). This type of multiplicity is normally represented by the
Greek letter , where (i) represents the algebraic
multiplicity of i.
Geometric Multiplicity: the geometric multiplicity of an
eigenvalue is the number of linearly independent eigenvectors
associated with it.
4
Jordan Decomposition
Camille Jordan (1870)
0
J J k ( )
0
1
0
0
Camille Jordan
(1838-1921)
such that
J P 1 AP
A PJP 1
where P is an invertable matrix, k1+k2+ + kr =n, i , i=1,2,. . ., r are
the characteristic roots and ki are the algebraic multiplicity of i .
Jordan Decomposition is used in Differential equation and time series analysis.
5
Spectral Decomposition
(Eigen Decomposition)
A. L. Cauchy established the Spectral Decomposition in 1829.
A. L. CAUCHY,
(1789-1857)
V (Y )
i 1
Eugenio Beltrami
(1835-1899)
Camille Jordan
(1838-1921)
James Joseph
Sylvester
(1814-1897)
Erhard Schmidt
(1876-1959)
Hermann Weyl
(1885-1955)
The Singular Value Decomposition was originally developed by two mathematician in the
mid to late 1800s
1. Eugenio Beltrami , 2.Camille Jordan
Several other mathematicians took part in the final developments of the SVD including James
Joseph Sylvester, Erhard Schmidt and Hermann Weyl who studied the SVD into the mid-1900s.
C.Eckart and G. Young prove low rank approximation of SVD (1936).
10
C.Eckart
What is SVD?
Any real (mn) matrix X, where (n m), can be
decomposed,
X = USVT
U is a (mm) or (mn) column orthonormal matrix
(UTU = I), containing the eigenvectors of the symmetric
matrix XXT.
S is a (mn) or (nn) diagonal matrix, containing the
singular values of matrix X. The number of non zero
diagonal elements of S corresponds to the rank of X.
VT is a (nn ) row orthonormal matrix (VTV = I),
containing the eigenvectors of the symmetric matrix XTX.
11
Dimension in SVD
economy-size
normal-size
12
13
Dr
D
0
0
0
Mr
M
0
0
0
14
U Qr
V Rr
1/ 2
r
S diag ( d
1/ 2
1
,d
1/ 2
2
,, d r
1/ 2
1/ 2
r
G T G (Qr Dr
1/ 2
Rr )T Qr Dr
1/ 2
Rr
Rr M r Rr
RMR T
XTX
Similarly,
GG T XX T
From the first relation above we conclude that for an arbitrary orthogonal matrix,
say P1 ,
G PX
1
While from the second we conclude that for an arbitrary orthogonal matrix, say
16
P2
G XP
2
XX T P1 XX T P1 ,
X T X P2T X T XP2
P1 I m ,
P2 I n
17
Decomposition in Diagram
Matrix A
LU decomposition
Not always unique
Rectangular
Full column rank
Square
Symmetric
Cholesky
Decomposition
QR Decomposition
SVD
Asymmetric
Spectral
Decomposition
Jordan
Decomposition
Similar
Diagonalization
P-1AP=
19
Properties Of SVD
Rewriting the SVD
A UV
where
ui i viT
i 1
r = rank of A
i = the i-th diagonal element of .
ui and vi are the i-th columns of U and V
respectively.
20
Proprieties of SVD
Low rank Approximation
Theorem: If A=UVT is the SVD of A and the
singular values are sorted as 1 2 , n
then for any l <r, the best rank-l approximation
l
to A is ~
r
T
~2
A
ui i vi ; A A i2
i 1
i l 1
Low-rank Approximation
SVD can be used to compute optimal low-rank
approximations.
Approximation of A is of rank k such that
~
A
Min
X :rank ( X ) k
A X
Frobenius norm
a
i 1 j 1
di
i 1
22
ij
Low-rank Approximation
Solution via SVD
~
A U diag (1 ,..., k ,0,...,0)V T
set smallest r-k
singular values to zero
* * *
* * *
* * *
*
*
*
* * *
X
K=2
* * *
* * *
* * *
*
*
*
* * *
* * *
* * *
* * *
k
~
A i 1 i u i viT
VT
Approximation error
How good (bad) is this approximation?
Its the best possible, measured by the Frobenius norm of the
error:
r
~
min
X :rank ( X ) k
A X
A A
i k 1
~
A A
2
F
24
A UV
u
k 1
k v
k 1
Ri
R
l
i
We can approximate Ri by
where i = 1,,m.
l
~
T
A U l lVl uk k vkT
T
k
u
k 1
ik
u
k 1
ik
k vk
k vk
;
C j v jk k u k
k 1
C v jk k u k
l
j
k 1
25
; l<r
l<r
Ax b
The SVD of Ag is
Where
27
28
Application of SVD
30
Origin of biplot
Gabriel (1971)
One of the most
important advances in
data analysis in recent
decades
Currently
> 50,000 web pages
Numerous academic
publications
Included in most
statistical analysis
packages
What is a biplot?
Biplot = bi + plot
plot
scatter plot of two rows OR of two columns, or
scatter plot summarizing the rows OR the columns
bi
BOTH rows AND columns
32
Matrix decomposition
P(4, 3)
g1
g2
g3
g 4
G(3, 2)
e1
e2
20
e3
6
g1
x
4
E1
E(2, 3)
y
3
G2
G1
E2
6
12 15 g 2 3 3 x
10 6
9
g 3 1 3 y
g 4 4
8
12 12
0
e1 e2
2 3
4
G4
e3
3
E3
G3
G-by-E table
33
X ij
SVD:
SVP:
SVP
Matrix
characterisi Singular values
ng the rows
SVD
Matrix
characterisi
ng the
columns
uik k vkj
k 1
(uik )(
f
k
k 1
1 f
k
f=1
vkj )
f=0
f=1/2
Rows
scores
Column
scores
Plot
34
Biplot Plot
Biplot
The simplest biplot is to show the first two PCs together with the
projections of the axes of the original variables
35
0.0
10
10
33
333 333
Sepal L.
33 33
22 2
3
22223 3 3
333
3 3 3
2 3
2 233 3 3
Petal W.
3
Petal L.
2
2
3
3
3
222 23
222 22 233 33
2 2 22 333
22 323
222
3 3 3
2
22 22
3
22
2
2 22 3
2
22 3
2
2
2
-0.2
-0.1
0.0
Comp. 1
0.1
0.2
36
-5
111
1
111
1
1
11 1
1 11 111
11
1
1
1
1
11 1
1
11
1
1
1 11
111
1
1
1
1
0.1
-0.1
0.2
1
Sepal W.
1
1 1
-0.2
Comp. 2
1= Setosa
2= Versicolor
3= Virginica
-5
-10
-10
37
38
Rank-1 approximation
Rank- 5
39
approximation
Rank-20 approximation
Rank-30 approximation
40
Rank-50 approximation
Rank-80 approximation
41
Rank-100 approximation
Rank-120 approximation
42
Rank-150 approximation
True Image
43
Scatter plot of Hawkins, Bradu and kass data (a) scatter plot of first two PCs and
(b) scatter plot of first and third PC.
45
48
49
Climatic Variables
The climatic variables are,
1.
Rainfall (RF) mm
2.
Daily mean temperature (T-MEAN)0C
3.
Maximum temperature (T-MAX)0C
4.
Minimum temperature (T-MIN)0C
5.
Day-time temperature (T-DAY)0C
6.
Night-time temperature (T-NIGHT)0C
7.
Daily mean water vapor pressure (VP) MBAR
8.
Daily mean wind speed (WS) m/sec
9.
Hours of bright sunshine as percentage of maximum possible sunshine
hours (MPS)%
10. Solar radiation (SR) cal/cm2/day
50
Consequences of SVD
Generally many missing values may present in the data. It may also contain
unusual observations. Both types of problem can not handle Classical singular
value decomposition.
Robust singular value decomposition can solve both types of problems.
Robust singular value decomposition can be obtained by alternating L1
regression approach (Douglas M. Hawkins, Li Liu, and S. Stanley Young,
(2001)).
51
u1
i 1
,p
x
j 1
ij
d i v j1
; i=1,2,.,n
For the second and subsequent of the SVD, we replaced X by a52deflated matrix
obtained by subtracting the most recently found them in the SVD X
X-kukvkT
53
References
Matrix Decomposition.
http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/csa/node36.html
54
55