Escolar Documentos
Profissional Documentos
Cultura Documentos
Avinash Kak
Purdue University
1
Expectation Maximization Tutorial by Avi Kak
CONTENTS
3 An Example of EM Estimation in 17
Which the Unobserved Data is
Just the Missing Data
5 Algorithm::ExpectationMaximization 69
a Perl Module
6 Convenience Scripts in 81
Algorithm::ExpectationMaximization
8 Acknowledgments 100
2
Expectation Maximization Tutorial by Avi Kak
3
Expectation Maximization Tutorial by Avi Kak
4
Expectation Maximization Tutorial by Avi Kak
5
Expectation Maximization Tutorial by Avi Kak
6
Expectation Maximization Tutorial by Avi Kak
7
Expectation Maximization Tutorial by Avi Kak
8
Expectation Maximization Tutorial by Avi Kak
9
Expectation Maximization Tutorial by Avi Kak
1 1
2 (~x~T
) 1 (~x~
)
p(~x) = 1/2
e (1)
(2)d/2 ||
10
Expectation Maximization Tutorial by Avi Kak
11
Expectation Maximization Tutorial by Avi Kak
1 1 T 1 (~
p(~x) = 1/2
e 2 (~x~i ) ~i )
x
(2)
(2)d/2 |i|
12
Expectation Maximization Tutorial by Avi Kak
13
Expectation Maximization Tutorial by Avi Kak
14
Expectation Maximization Tutorial by Avi Kak
15
Expectation Maximization Tutorial by Avi Kak
16
Expectation Maximization Tutorial by Avi Kak
3. An Example of EM Estimation in
Which the Unobserved Data is Just the
Missing Data
17
Expectation Maximization Tutorial by Avi Kak
We will
denote the last observation ~x4 =
x4,1
4 , where the variable x4,1 stands for
the missing information in the data.
18
Expectation Maximization Tutorial by Avi Kak
1 1
2 (~x~
) T
1 (~x~
)
p(~x) = 1/2
e (3)
(2)d/2 ||
12
h i
0
= (4)
0 22
19
Expectation Maximization Tutorial by Avi Kak
20
Expectation Maximization Tutorial by Avi Kak
21
Expectation Maximization Tutorial by Avi Kak
4
X
LL = ~
ln p(~xi|) (6)
i=1
22
Expectation Maximization Tutorial by Avi Kak
3
X
LL = ~ + ln p(~x4 |)
ln p(~xi|) ~ (8)
i=1
23
Expectation Maximization Tutorial by Avi Kak
24
Expectation Maximization Tutorial by Avi Kak
25
Expectation Maximization Tutorial by Avi Kak
26
Expectation Maximization Tutorial by Avi Kak
3 Z
X
LL = ~ +
ln p(~xi |) ~
ln p(~x4 |) p(x4,1 |~g , x4,2 = 4) dx4,1
i=1
(10)
27
Expectation Maximization Tutorial by Avi Kak
Z
~
ln p(~x4 | ) p(x4,1 | ~
g , x4,2 = 4) dx4,1
p(x4,1, ~g , x4,2 = 4)
Z
= ~
ln p(~x4 | ) dx4,1
p( ~g , x4,2 = 4)
x4,1
p ~g p(~g )
Z
4
= ~
ln p(~x4 | ) dx4,1
p(x4,2 = 4 | ~g ) p(~g )
x4,1
p ~g
Z
4
= ~
ln p(~x4 | ) dx4,1
p(x4,2 = 4|~g )
x4,1 ~g
Z p 4
= ~
ln p(~x4 | ) dx4,1
x4,1
R
p
~g dx
4 4,1
(11)
28
Expectation Maximization Tutorial by Avi Kak
3 x4,1 ~g
X Z p 4
LL = ~ +
ln p(~xi |) ~
ln p(~x4 | ) dx4,1
x4,1
R
~g
i=1
p
4 dx4,1
(12)
29
Expectation Maximization Tutorial by Avi Kak
0
0
~g = 1 (13)
1
30
Expectation Maximization Tutorial by Avi Kak
3 Z
~ + 1
x4,1
X
LL =
ln p(~xi |) ~ p
ln p(~x4 | )
~g dx4,1
D 4
i=1
(14)
0
!!
Z
x4,1
0
D = p dx4,1
4 1
1
Z
1 1 2 2
= e 2 (x4,1 +4 ) dx4,1
2
e8
= (15)
2
31
Expectation Maximization Tutorial by Avi Kak
1 e8
= ln
21 2 2
1 e8
(1 + 21 )2 12 + (16 82 + 22 )22
2 2
(16)
32
Expectation Maximization Tutorial by Avi Kak
Z
2
z 2 ez dz = 2
(17)
where the second property is a simplifi-
cation of the following result concerning
Gaussian integrals (see Wikipedia page on
Gaussian
Z integrals): 2n+1
z2
2n a2 (2n)! a
z e dz =
0
n! 2
33
Expectation Maximization Tutorial by Avi Kak
0.75
2.0
~new = 0.918 (19)
2.0
34
Expectation Maximization Tutorial by Avi Kak
1.0
2.0
~new = 0.667 (20)
2.0
35
Expectation Maximization Tutorial by Avi Kak
36
Expectation Maximization Tutorial by Avi Kak
37
Expectation Maximization Tutorial by Avi Kak
X = {~x1 , ~x2 , . . . ~
xN } (21)
38
Expectation Maximization Tutorial by Avi Kak
39
Expectation Maximization Tutorial by Avi Kak
N
Y
p(X |) = p(~xi | )
i=1
N K
!
Y X
= a p (~xi | )
i=1 =1
(25)
N K
!
Y X 1 1 T 1 (~
p(X |) = a 1/2
e 2 (~xi ~ )
xi ~ )
(2)d/2 | |
i=1 =1
(26)
40
Expectation Maximization Tutorial by Avi Kak
N K
!
Y X 1 1 T 1 (~
L( | X ) = a 1/2
e 2 (~xi ~ )
xi ~ )
(2)d/2 | |
i=1 =1
(27)
41
Expectation Maximization Tutorial by Avi Kak
N
" K
!#
X X 1 1 T
1
= argmax ln a 1/2
e 2 (~xi ~ )
(~xi ~ )
(2)d/2 ||
i=1 =1
(29)
42
Expectation Maximization Tutorial by Avi Kak
43
Expectation Maximization Tutorial by Avi Kak
L( | X , Y) = p(X , Y | )
N
Y
= p(~xi , yi | )
i=1
N
Y
= p(~xi | yi , ) p(yi ) (30)
i=1
44
Expectation Maximization Tutorial by Avi Kak
N
Y
L( | X , Y) = pyi (~xi | yi ) p(yi )
i=1
YN
= ayi pyi (~xi | yi )
i=1
(31)
45
Expectation Maximization Tutorial by Avi Kak
N
X
LL = ln ayi pyi (~xi | yi ) (32)
i=1
46
Expectation Maximization Tutorial by Avi Kak
K K N
X X X
LL = ... ln ayi pyi (~xi | yi ) p(y1 . . . yN | g , X )
y1 =1 yN =1 i=1
(33)
47
Expectation Maximization Tutorial by Avi Kak
48
Expectation Maximization Tutorial by Avi Kak
49
Expectation Maximization Tutorial by Avi Kak
50
Expectation Maximization Tutorial by Avi Kak
51
Expectation Maximization Tutorial by Avi Kak
52
Expectation Maximization Tutorial by Avi Kak
K K N
The portion xj , g ) of
P P Q
y =1
... j=1 p(yj | ~
y =1 ,y i
1 N
K K N
X X Y
g
... ,yi p(yj | ~xj , )
y1 =1 yN =1 j=1
K
X K K K N
X X X Y
= ... ... p(yj |~xj , g ) p(|~xi , g )
y1 =1 yi1 =1 yi+1 =1 yN =1 j=1,j6=i
(39)
53
Expectation Maximization Tutorial by Avi Kak
K K N
X X Y
g
... ,yi p(yj | ~xj , )
y1 =1 yN =1 j=1
N K
X
Y
= p(yj | ~xj , g ) p(|~xi , g ) (40)
j=1,j6=i yj =1
K K N
X X Y
g
... ,yi p(yj | ~xj , ) = p(|~xi , g ) (41)
y1 =1 yN =1 j=1
54
Expectation Maximization Tutorial by Avi Kak
K N
X X
LL = ln a p (xi | ) p(|~xi , g ) (42)
=1 i=1
K N K N
X X X X
g
LL = ln a p(|~xi , ) + ln p (xi | ) p(|~xi , g )
=1 i=1 =1 i=1
(43)
55
Expectation Maximization Tutorial by Avi Kak
56
Expectation Maximization Tutorial by Avi Kak
K
h X i
LL + a = 1 = 0 (44)
a
=1
K
X N K
X X
ln(a) p(|~xi , g ) + a = 1 = 0
a
=1 i=1 =1
(45)
57
Expectation Maximization Tutorial by Avi Kak
N
X
ln(a ) p( |~xi , g ) + = 0 (46)
a
i=1
Since x
ln x
= 1x , the K equations shown above
reduce to
N
X
p( |~xi , g ) + a = 0 (47)
i=1
= N (48)
58
Expectation Maximization Tutorial by Avi Kak
for = 1 . . . K.
59
Expectation Maximization Tutorial by Avi Kak
60
Expectation Maximization Tutorial by Avi Kak
PN
~x p | ~
i=1 i
xi , g
~ = PN (53)
i=1
p | ~xi , g
for = 1 . . . K .
61
Expectation Maximization Tutorial by Avi Kak
PN
i=1
~
xi p | ~xi , g
~ new
= PN (54)
i=1
xi , g
p |~
62
Expectation Maximization Tutorial by Avi Kak
63
Expectation Maximization Tutorial by Avi Kak
first of these can be solved using the iden-
tity lnA|A| = 2A1 diag(A1) and the second by
tr(AB)
A
= B+B T diag(B). Substituting these deriva-
tives in Eq. (56) and setting the result to
zero gives us the following equation:
N
1X
p |~xi , g 2 diag( )
2
N
i=1 X
1
g
p |~xi , 2N,i diag(N,i ) = 0 (57)
2
i=1
where M ,i = N ,i for = 1 . . . N .
64
Expectation Maximization Tutorial by Avi Kak
2S diag(S) = 0 (59)
with
N
1X g
S = p |~xi, M ,i (60)
2 i=1
N
1X g
p |~xi, M ,i = 0 (61)
2 i=1
for = 1 . . . K
65
Expectation Maximization Tutorial by Avi Kak
PN
i=1
p |~xi, g (~xi ~ )T
~ )(~xi
new
= PN (63)
i=1
p |~xi , g
for = 1 . . . K.
PN
i=1
p |~xi, g (~xi
~ new
~ new
)(~xi )T
new
= PN (64)
i=1
p |~xi , g
for = 1 . . . K.
66
Expectation Maximization Tutorial by Avi Kak
67
Expectation Maximization Tutorial by Avi Kak
g p(~xi|, g ) p(|g )
p(|~xi, ) =
xi, g )
p(, ~
p(~xi|, ) a
=
xi , g )
p(, ~
p(~xi|, ) a
=
numerator normalizer
(65)
68
Expectation Maximization Tutorial by Avi Kak
5. Algorithm::ExpectationMaximization
a Perl Module
69
Expectation Maximization Tutorial by Avi Kak
70
Expectation Maximization Tutorial by Avi Kak
my $mask = "N0111";
71
Expectation Maximization Tutorial by Avi Kak
72
Expectation Maximization Tutorial by Avi Kak
$clusterer->read_data_from_file();
srand(time);
$clusterer->seed_the_clusters();
$clusterer->EM();
$clusterer->run_bayes_classifier();
my $clusters = $clusterer->return_disjoint_clusters();
73
Expectation Maximization Tutorial by Avi Kak
74
Expectation Maximization Tutorial by Avi Kak
75
Expectation Maximization Tutorial by Avi Kak
76
Expectation Maximization Tutorial by Avi Kak
posterior_prob_cluster1.dat
posterior_prob_cluster2.dat
...
77
Expectation Maximization Tutorial by Avi Kak
my $visualization_mask = 111;
$clusterer->visualize_clusters($visualization_mask);
$clusterer->visualize_distributions($visualization_mask);
$clusterer->plot_hardcopy_clusters($visualization_mask);
$clusterer->plot_hardcopy_distributions($visualization_mask);
78
Expectation Maximization Tutorial by Avi Kak
79
Expectation Maximization Tutorial by Avi Kak
80
Expectation Maximization Tutorial by Avi Kak
81
Expectation Maximization Tutorial by Avi Kak
canned example1.pl:
The goal of this script is to show EM-
based clustering of overlapping clusters
starting with randomly selected seeds.
As programmed, this script clusters the
data in the datafile mydatafile.dat. The
mixture data in the file corresponds to
three overlapping Gaussian components
in a star-shaped pattern.
canned example2.pl:
This goal of this script is to use the out-
put of the K-Means clusterer to serve as
seeds for the EM-based clusterer. The
data fed to this script consists of two
well separated blobs.
82
Expectation Maximization Tutorial by Avi Kak
canned example3.pl:
This script gives a demonstration of how
you would structure a call to the module
constructor when you want to specify
the cluster seeds manually.
canned example4.pl:
Whereas the three previous scripts demon-
strate EM based clustering of 2D data,
this script uses the module to cluster 3D
data. This script is meant to demon-
strate how the EM algorithm works on
well-separated but highly anisotropic clus-
ters in 3D.
canned example5.pl:
This script also demonstrates clustering
in 3D but now we have one Gaussian
cluster that cuts through the other
two Gaussian clusters.
83
Expectation Maximization Tutorial by Avi Kak
84
Expectation Maximization Tutorial by Avi Kak
85
Expectation Maximization Tutorial by Avi Kak
86
Expectation Maximization Tutorial by Avi Kak
87
Expectation Maximization Tutorial by Avi Kak
88
Expectation Maximization Tutorial by Avi Kak
89
Expectation Maximization Tutorial by Avi Kak
90
Expectation Maximization Tutorial by Avi Kak
91
Expectation Maximization Tutorial by Avi Kak
92
Expectation Maximization Tutorial by Avi Kak
93
Expectation Maximization Tutorial by Avi Kak
94
Expectation Maximization Tutorial by Avi Kak
95
Expectation Maximization Tutorial by Avi Kak
96
Expectation Maximization Tutorial by Avi Kak
97
Expectation Maximization Tutorial by Avi Kak
98
Expectation Maximization Tutorial by Avi Kak
99
Expectation Maximization Tutorial by Avi Kak
8. Acknowledgments
100
Expectation Maximization Tutorial by Avi Kak
101