Unit - i

Artificial neural networks

5. Ex OR function using MP neuron model :-
The truth table for XOR functions is

x1 x2 y x
1

1 1 0
y
1 0 1
0 1 1 x
2
0 0 0
The threshold of unit y is 1.
MP Model for XOR
With one layer done, it is not function
possible to predict the value
of the threshold for the neuron to fire, hence another
layer is introduced
x1 XOR x2 = (x1 ANDNOT x2) OR (x2 ANDNOT x1)
x1 XOR x2 = H1 OR H2 ; where H1 = x1 ANDNOT x2
H2 = x2 ANDNOT x1
The activation of H1 and H2 are
= 1 ; H in-1 1
H= 0 ; H in-1 <1
1 = H
in-1

= 1 ; H in-2 1
H= =
i2
0 H ; H in-2 <1
in-2

The net i/p and activation of H1 and H2 are

H1 = x1 ANDNOT xi2 ; Hin-1 = x1w1+x2w2
= x1-x2 (w1=1,w2=-1)
 X1 X2 Hin-1 H1
1 1 0 0
1 0 1 1
0 1 -1 0
0 0 0 0

H2 = x2 ANDNOT x1 ; Hin-2 = x1w1+x2w2

= -x1+x2 (w1 = -1,w2 = 1)

X1 X2 Hin-2 H2
1 1 0 0
1 0 -1 0
0 1 1 1
0 0 0 0
The activation for the o/p unit y = 1
y=f(yin) = 1 ; if yin 1
0 ; if yin < 1
Presenting i/p patterns H1 & H2 and calculating net i/p and
activations gives o/p of XOR.
yin = H1w1+H2w2
= H1+H2 (w1 = w2 = 1)
H1 H2 yin y=H1
(or) H2
0 0 0 0
1 0 1 1
0 1 1 1
0 0 0 0

Thus the XOR function is realized.

Linearly separable means, A functions with n i/p bits
and single o/p bit, a hyper lane exists there by dividing
the i/p points into groups in n- dimensional space out of
which one group contains 0 output and other 1
output.
Discus various learning techniques ? (Rules (or) Laws)
Learning (or) training is the term used to describe process of
finding values weights.
The classification of learning algorithms are given below
Artificial
network

Supervised Unsupervised Rein forced

learning learning learning

Error correction gradient Competiti

Stochastic Hebbian
descent ve

Least mean Back

square propagation
1. Hebbian Learning Rule
.It is also call as correlational learning.
.It can be split into two part rule.
i. If two neurons on either side of a synapse are
activated simultaneously, than the strength of that
synapse is selectively increased.
ii. If two neurons on either side of a synapse are
activated asynchronously, than the strength of that
synapse is selectively weakened or eliminated.
.It is purely feedforward ,unsupervised learning.
.In this rule, the I/p o/p pattern pairs (x y i) are
I,

associated
n by the weight matrix w known as
correlation matrix computed as.
.W= x y T
i=0 y i T = Transpose of the associated o/p
i i
vector yi.
2. Perceptron leaning rule
.It is also known as Discrete Perceptron leaning law.
.For the perceptron learning rule, the leaning signal is
the difference between the desired and actual neurons
response.
.It is supervised leaning.
.It is applicable only for bipolar o/p functions f(.).
.The preceptor leaning rule states that for a finite n no.
of I/p training vector x(n), each with an associate target
value t(n) which is +1 (or) -1, and an activation function.
1 ; if y in >
y= 0 ; if y in
-1 ; if y in <
Then the weight updated is given by
wnew = w old+t x ; if y t
wnew = w old ; if y = t
Perceptron Training Algorithm :
i. Start with random value of w.
ii. Test for w.x i >0, if test succeed for i=1,2,..n, then return
w.
iii. Modify w, as wnew = w prev+xfail.
Limitations of Perceptron :
1. Non-linear reparability is not passive i.e it can only model linearly
single perceptron.
2. Single perceptron does not have enough computing
power
SOL : 1. Use larger network.
2 . Tolerate error.
Perceptron Leaning Algorithm :
x(n) = i/p vector
w(n) = weight vector
b(n) = bias
y(x) = actual response
d(n) = desired response
= learning rate parameter
i. Initialization :- Set w(0) = 0
ii. Activation :- Activate perceptron by applying i/p
iii. Complete actual response of perceptron
y(x) = sgn [wT(n).x(n)]
iv. Adapt weight vector i.e. if y(n) & d(n) are different, then
w(n+1) = w(n)+ [d(n)-y(n)].x(n)
+1 ; x(n) c1 c1 = class 1
where-1d(n)
; x(n)
= c2 c2 = class 2
v. Continuation :- Increment step n by 1 and go to
activation step.
3. Delta Learning law :-
.It is valid only for continuous activation functions and
differentiable o/p function.
.It is supervised learning .
.It is also known as continuous perceptron leaning
 It states that
The adjustment made to a synaptic weight of neuron is
proportional to the product of the error signal and the i/p
signal of the synapse.
Delta rule for signal o/p unit is that
it changes the weight of the connection to minimize
the difference the net i/p to the o/p unit y inand the target
value t.
i.e. w i = (t-y in)x i
Where x = the vector of activation of i/p units.
y in = net i/p to o/p unit i.e. xw i
t = target vector.
= learning rate.
Delta rule for several o/p units is that,
wjk = (tj-yinj)xi
4. Competitive Learning Rule :-
5. Outstar Leaning Rule :-
.It is also known as gross berg leaning.
.It is supervised leaning .
.It is used to provide learning of repetitive and
characteristic properties of i/p o/p relationship.
.The weight matrix

w =(yk-wjk) ; if neuron j wins the competition.

jk
0 ; if neuron j losses the competition.
6. Boltzmann leaning :-
.It is also known as stochastic leaning.
.Here the weights are adjusted in a probabilistic fashion.
.Used in symmetric recurrent network.(i.e. symmetric :
wij= wji )
.Consist of binary units (+1 for on, -1 for off )
.Neurons are divided into two groups i.e. hidden &
visible.
Compare supervised and unsupervised learning ?
Learning (or) Training is term used to describe process
of finding values of weights.

S.n Supervised Unsupervised

o
1 It depends on local and It depends on only local
external information. information.
2 External supervisor is External supervisor is not
required. required.
3 Leaning phase & operation Leaning and operates at
phase are distinct i.e. same time i.e. preferred on-
preferred off line leaning. line leaning.
4 Comparison is made between It does not require target
actual o/p & desired o/p and o/p is no comparison of o/p
desired o/p is compared with is made with desired o/p i.e.
target o/p i.e. it has o/p & i/p. it has only i/ps.
S.n Supervised Unsupervised
o

input Adaptive output

network input Adaptive output
5 network
Network Leanin
leaning g
algorithm signal
Compare single layer and multilayer ANNs ?
S. Single layer ANN
No
1 Not suitable for learning
large & move complex
model.
2 No hidden layers.
3 It is not an auto
associative network.
4 No feedback.
5 Cannot solve linear
separability problems.
6 Processing stage is not
made of neurons.
Multi layer ANN
Suitable for learn large &
more complex model.
Hidden layers present.
It is an auto associative
network.
Feedback is present.
Solve linear separability
problems.
Processing stage is made of
neurons.
S. Single layer ANN
No
8 Connections with in a
network are direct.
9 Hard limiting function is
used as activation function activation function.
10 It makes use of Delta Rule
as learning rule.
Multi layer ANN
Connections within a
network are not direct, but
with help of hidden layers.
Sigmoid function is used as
It makes use of Back
Propagation in Algorithm as
learning Algorithm.
Basic Propagation Algorithm :-
It is also known as generalized delta rule.
The total squared error of the output computed by net is
minimized by a gradient descent method known as back
propagation or generalized delta rule.
It is used for training multi-layer forward network.
Network is trained by supervised learning method.
The architecture of back propagation network resembles
a multilayered feed forward network.
A multilayer feed forward back propagation network with
one layer and two hidden units and the output units and
hidden units have bias. The bias acts like a weights on
connections from units whose output is always 1. The
input layer connected to hidden layer and hidden layer
connected to output layer by means of inter connections
weights. These can be used any no of hidden layers .
The training algorithm back propagation involves four
stages i.e .
1. Initialization of weights
2. Feed forward
3. back propagation of error
4. Updation of weights of and biases
.The various parameters used in training algorithm are
.x = input training vector = (x1,x2x1,..xn)
.T = output target vector = (t1,tk,tm)
. k = error at output unit yk
. j = error at output unit yk
. = learning rate
Voj = bias on hidden unit j.
Zj = hidden unit j.
wok= bias on output unit k.
yk = output unit k.