Neural Networks

Multilayer Perceptron (MLP)

Oscar Herrera Alcntara
heoscar@yahoo.com
Introduction to Artificial Intelligence - APSU
Outline
Neuron

Artificial neural networks
Activation functions

Perceptrons

Multilayer perceptrons

Backpropagation
Generalization

Introduction to Artificial Intelligence - APSU
Neuron
A neuron is a cell in the brain
collection, processing, and dissemination of electrical signals
neurons of > 20 types, synapses, 1ms-10ms cycle time
brains information processing relies on networks of such neurons
11
10
14
10
Introduction to Artificial Intelligence - APSU
Biological Motivation
dendrites: nerve fibres carrying electrical signals to the
cell

cell body: computes a non-linear function of its inputs

axon: single long fiber that carries the electrical signal
from the cell body to other neurons

synapse: the point of contact between the axon of one
cell and the dendrite of another, regulating a chemical
connection whose strength affects the input to the cell.
Introduction to Artificial Intelligence - APSU
Artificial neural networks
A mathematical model of the neuron is McCulloch-Pitts unit
Neural networks consists of nodes (units) connected by directed links
A bias weight Wi,0 connected to a fixed input xi,0 = +1
E
Neuron i
1

x
1


x
2


x
3


x
m


w
i1

W
im
y
Inputs

function
b

:Bias

Synaptic
Weights Output
Activation
v
Induced local field
Activation potential

=
+ = =
m
j
j i
b w v y
1
,
) ( ) (
Introduction to Artificial Intelligence - APSU
Activation functions
a) Step function or Threshold function
b) Sigmoid function
c) Hyperbolic tangent function

av
e
v

+
=
1
1
) (
0 if
0 if
<
>
=
v
u

0
1
) (v
0 ,
)) ( tanh( ) (
>
=
b a
n b a v
j
v
Introduction to Artificial Intelligence - APSU
Perceptron learning
Learn by adjusting weights to reduce error on training set
Error correction learning rule
Perform optimization search by gradient descent
) ( )] ( ) ( [ ) ( ) 1 (
2
1
) ( ) ( ) (
2
n x n y n d n w n w
e E
n y n d n e
i ji ji
+ = +
=
=
q
parameter rate learning =
-x(n)e(n) g(n) Gradient = =
Introduction to Artificial Intelligence - APSU
Implementing logic functions
McCulloch-Pitts unit can implement any Boolean function
2 2 1 1 0
1 X w X w w v + + =

>
=
otherwise
v if
y
0
0 1
y
1
X
2
X
B
y y
1
X
1
X
Introduction to Artificial Intelligence - APSU
Expressiveness of perceptrons
A perceptron
can represent AND, OR, NOT
can represent a linear separator (function) in input space:
Introduction to Artificial Intelligence - APSU
Multilayer Perceptron (MLP): Architecture
1
1
1








x
1

x
2

x
3

y
1

y
2



w
ij

w
jk

w
kl

Input
Layer
Hidden Layers
Output
Layer
Inputs
Outputs
Bias
Introduction to Artificial Intelligence - APSU
Solve XOR problem using MLPs
A two-layer network with two nodes in the hidden layer
The hidden layer maps the points from non linear separable space
to linear separable space.
The output layer finds a decision line

y1
y2
A
B
w01
w02
w03
y
w11
w12
w21
w22
w13
w23
-1
-1
-1
g1
g2
g3
(v)
Introduction to Artificial Intelligence - APSU
Back-propagation Algorithm
1. Initialization. Weights are initialized with random values whose mean
is zero
2. Presentations of training examples
3. Forward computation
4.-Backward computation
for the neuron j of the hidden layer l

for the neuron j of the output layer L


5.- Iteration. Repeat step 2 to 4 until E< desired error
o the momentum parameter is ajusted
q the learning-rate parameter is ajusted

+ +
=
k
l
kj
l
k
l
j
l
j
n w n n v n ) ( ) ( )) ( ( ' ) (
1 1
o o
L
j
l
j
l
j
e n v n )) ( ( ' ) ( o =
) ( ) ( )] 1 ( [ ) 1 (
1
n y n n w w n w
l
i
l
j
l
ji
l
ji
l
ji

+ + = + qo o
Introduction to Artificial Intelligence - APSU
MLP Training
Forward Pass
Fix w
ji
(n)
Compute y
j
(n)

Backward Pass
Calculate o
j
(n)
Update weights w
ji
(n+1)
i j k
Left Right
i j k
Left Right
x
y
Introduction to Artificial Intelligence - APSU
Generalization
Total Data are divided in two parts:
Data Training (80%)
MLP is trained with Data Training

Data Test (20%)
MLP is tested with Data Test
Generalization
MLP is used with inputs which have never been
presented in order to predict the outputs