Escolar Documentos
Profissional Documentos
Cultura Documentos
Analysis
Contents
Introduction
PCA vs. LDA
Background Mathematics
Statistics
Matrix Algebra
Method
Example
Introduction
Face Recognition
Image compression
Background Mathematics
Statistics
Given a sample X
The mean
Background Mathematics
Statistics
Standard Deviation and variance are 1-dimensional.
More than 1-dimension. Is there a relationship between the
dimensions?
Standard Deviation and variance are computed for each
dimension of the data set independently from each other.
Covariance: measure to find out how much the dimensions
vary from the mean with respect to each other.
If you calculate the covariance between one dimension and
itself, you get the variance.
Background Mathematics
Covariance
The exact value is not as important as its sign (i.e. Positive or
negative).
If the value is positive, then that indicates that both dimensions
increase together
If the value is negative, then as one dimension increases, the other
decreases.
if the covariance is zero, it indicates that the two dimensions are
independent of each other.
This technique is often used to find relationships between
dimensions in high-dimensional data sets
Cov(X,Y) = Cov(Y,X)
Background Mathematics
Covariance
For an n-dimensional data set you can compute n!/2*(n-2)! =
n*(n-1)/2
Covariance Matrix for a set of data with n dimensions is:
Background Mathematics
Matrix Algebra
Eigenvectors and Eigenvalues:
Definition: Let A be an n X n matrix.
Scalar is called an eigenvalue of A if there is an non-zero
vector X such that
AX = X
The vector X is called an eigenvector of A corresponding to .
Example:
A=[3,2 ; 3,-2] X=[2;1] (a column) is an eigenvector of A
corresponding to =4
Background Mathematics
Matrix Algebra
If is an eigenvalue of A, and X is an eigenvector belonging to
, then any non-zero multiple of X will be an eigenvector.
In the previous example: Since X=[2;1] is an eigenvector then:
X=[4;2] or X=[20,10] are also eigenvectors.
A(c.X) = c.A.X = c..X= .(c.X)
Finding Eigenvalues and Eigenvectors: Given A, a 2*2
matrix
1. Multiply an n*n identity matrix by the scalar
2. Subtract the result of the multiplication in step 1 from the
matrix A
Background Mathematics
Finding Eigenvalues and Eigenvectors
3. find the determinant of the matrix resulted in step 2 and the
difference
4. Solve for the values of that satisfy the equation: det(AI)= 0-vector; I is the identity matrix
Solve for the corresponding vector for each
Example A= [7,3 ; 3,-1]
Step 1: . I = . [1,0 ; 0,1] = [,0 ; 0, ]
Setp 2: A .I = [7- , 3 ; 3, -1- ]
Step 3: det([7- , 3 ; 3, -1- ]) = (7- ) * (-1- ) 3*3 = 2 -6
-16
Step 4: Solve the equation: 2 -6 -16 = 0 for
Background Mathematics
Method
Method
Follow the steps:
Standardize the data (in case the data was measured on different
scales for instance).
Obtain the Eigenvectors and Eigenvalues from the covariance
matrix or correlation matrix, or perform Singular Vector
Decomposition.
Sort eigenvalues in descending order and choose the k eigenvectors
that correspond to the k largest eigenvalues where k is the number
of dimensions of the new feature subspace.
Construct the projection matrixW from the selected K eigenvectors.
Transform the original dataset X via W to obtain a k-dimensional
feature subspace Y.
Example
Data =
2.5
2.4
0.69
0.49
0.5
0.7
-1.31
-1.21
2.2
2.9
0.39
0.99
1.9
2.2
0.09
0.29
3.1
3.0
1.29
1.09
2.3
2.7
0.49
0.79
1.6
0.19
-0.31
1.1
-0.81
-0.81
1.5
1.6
-0.31
-0.31
1.1
0.9
-0.71
-1.01
Data-Adjusted=
Example
Calculate the covariance matrix
Cov= [0.616555556 , 0.615444444 ; 0.615444444 ,
0.716555556]
Eigenvalues = [0.0490833989 ; 1.28402771]
Eigenvectors = [-0.735178656 , -0.677873399 ; 0.677873399 ,
-0.735178656]
Choosing components and forming a feature vector
The eigenvector with the highest eigenvalue is the principle
component of the data set.
We can choose to leave out the smaller, less significant
component and only have a single column:
Projection matrix W=[-0.677873399 ; -0.735178656]T
Example
Deriving the new data set
Final Data = W . Data-Adjusted
The transformed data set:
x
-0.827970186
T
1.77758033
-0.992197494
-0.274210416
-1.67580142
-0.912949103
1.14457216
0.438046137
1.22382056