Multilayer Perceptron

1
 Output units for mixture of Gaussians?

After minimization the error:

𝑁𝑁
1 1
𝑁𝑁 ln 𝜎𝜎 + ln 2𝜋𝜋 � 𝑇𝑇 𝒙𝒙𝒊𝒊
+ 2 � 𝑑𝑑𝑖𝑖 − 𝒘𝒘 2
2 2𝜎𝜎
𝑖𝑖=1
Residual value
Min w.r.to 𝜎𝜎
𝑁𝑁 𝑁𝑁
𝑁𝑁 1 1
� 𝑇𝑇 𝒙𝒙𝒊𝒊
− 3 � 𝑑𝑑𝑖𝑖 − 𝒘𝒘 2
=0 2 � 𝑇𝑇 𝒙𝒙𝒊𝒊
𝜎𝜎 = � 𝑑𝑑𝑖𝑖 − 𝒘𝒘 2
𝜎𝜎 𝜎𝜎 𝑁𝑁
𝑖𝑖=1 𝑖𝑖=1
Double derivative at 2𝑁𝑁
stationary point: 𝜎𝜎 2
2
 For a single neuron (individually Gaussian)

𝑝𝑝 𝑑𝑑/𝒙𝒙 =

Taking multiple neurons

𝑑𝑑 → 𝒕𝒕 (dim – c) uncorrelated

Gaussian: i.i.d.
This is now individually Gaussian Class k
3
 What if they are individually Gaussian mixture:

Informed
assumptions
k inputs

Bayesian i outputs
uninformative
prior assuming
uniform dist.
outputs

4
 Error (cross entropy) for qth input

Some hidden units… later.