BY:

Eng.Ismail El-Gayar

Under Supervision Of
Prof.Dr. Sheren Youssef
 • Training Procedures
Introduction
Understanding the Improving Convergence
Brain Momentum
Neural Networks as a Adaptive Learning Rate
Paradigm for Parallel • Learning Time
Processing Time Delay Neural Networks
• The Perceptron Recurrent Networks
Network
• Training a Perceptron
• Multilayer
Perceptrons
• Backpropagation
Algorithm
Two-Class
 :- Processing Human brain
contains a
massively
interconnecte
How our brain
d net of 1010 -
manipulates
1011 (10
with patterns ?
billion)
neurons
A process of
pattern recognition
and pattern
manipulation is
based on:
Associative distributed
Massive parallelism Connectionism memory
Brain computer as an Brain computer is a highly Storage of information in
information or signal interconnected neurons a brain is supposed to be
processing system, is system in such a way that concentrated in synaptic
composed of a large the state of one neuron connections of brain
number of a simple affects the potential of the neural network, or more
processing elements, called large number of other precisely, in the pattern of
neurons. These neurons neurons which are these connections and
are interconnected by connected according to strengths (weights) of the
numerous direct links, weights or strength. The key synaptic connections.
which are called idea of such principle is the
connection, and cooperate functional capacity of
which other to perform a biological neural nets 3
 The Biological Neuron
:-
Synaps
es
Axon
from
other
Soma
neuron
Dendri
Axon te
Dendrite from
s other
The
schematic
model of a
1. Soma or body cell - is a large, round central biological
body in which almost all the logical
functions of the neuron are realized. neuron

2. The axon (output), is a nerve fibre attached to the soma which can serve as a final
output channel of the neuron. An axon is usually highly branched.
3. The dendrites (inputs)- represent a highly branching tree of fibres. These long
irregularly shaped nerve fibres (processes) are attached to the soma.
4. Synapses are specialized contacts on a neuron which are the termination points for
the axons from other neurons.
Brain-like Computer
Artificial Neural Network – Mathematical
Paradigms of Brain-Like Computer
The new paradigm of
computing mathematics
consists of the combination
of such artificial neurons into
Neurons and some artificial neuron net.
Neural Net

Brain-Like Computer

Brain-like computer –

is a mathematical model of humane-

brain principles of computations. This
computer consists of those
elements which can be called the ?
biological neuron prototypes, which
are interconnected by direct links
called connections and which
cooperate to perform parallel
distributed processing (PDP) in order
ANN as a Brain-Like
Computer
NN as an model of  An artificial neural network (ANN) is
brain-like
a massively parallel distributed processor
Computer
that has a natural propensity for storing
experimental knowledge and making it
available for use. It means that:
 
Knowledge is acquired by the network
through a learning (training) process;
 The strength of the interconnections
between neurons is implemented by
Brain
means of the synaptic weights used to
store the knowledge.
The human brain is still not
well understood and indeed its
The learning process is a procedure of the
behavior is very complex!
adapting the weights with a learning
There are about 10 billion
algorithm in order to capture the
neurons in the human cortex
knowledge. On more mathematically, the
and 60 trillion synapses of
aim of the learning process is to map a
connections
given relation between inputs and output
The brain is a highly complex,
(outputs) of the network.
nonlinear and parallel computer
(information-processing
system) 7
Applications of Artificial Neural
Networks
Intelligen
Intelligen
Advanc
Advanc ttControl
Control

ee Technical
Technical
Robotic
Robotic Diagnistic
Diagnistic
ss ss
Machine Intelligent
Intelligent
Machine
Vision Data
Data
Vision
Artificial Analysis
Analysis
Intellect andSignal
and Signal
with Neural Processing
Processing
Networks

Image&&
Image
Pattern
Pattern
Recognitio
Recognitio Intelligent
Intelligent
nn Expert
Expert
Systems
Systems
Intelligen
Intelligen Intelligent
Intelligent
tltl Security
Security
Medicine
Medicine Systems
Systems
Devices
Devices
8
8
Artificial Neural
Networks
Perceptrons
Multiple input nodes
Single output node
Takes a weighted sum of the inputs, call this S
Unit function calculates the output for the network
Useful to study because
We can use perceptrons to build larger networks
Perceptrons have limited representational
abilities
We will look at concepts they can’t learn later
?Why neural network
f ( x1 ,..., xn ) - unknown multi-factor decision
rule

Learning process using a representative learning

set

- a set of weighting vectors is

( w0 , w1 ,..., wn ) the result of the learning
process

fˆ ( x1 ,..., xn ) = - a partially defined function,

which is an approximation of
= P ( w0 + w1 x1 + ... + wn xn ) the decision rule function 11
Artificial Neuron
f ( x1 ,..., xn ) = F ( w0 + w1 x1 + ... + wn xn )
Z is the weighted sum
f is a function to be earned
x1 ,..., xn are the inputs
x1 φ is the activation function

. f ( x1 , ...,xn )
. φ(z)
.

xn z = w0 +w1 x1 ...+ +w
n nx
:-Perceptrons
Output:- using Hardlims function
Simple Example:
Categorising Vehicles
Input to function: pixel data from vehicle images 
Output: numbers: 1 for a car; 2 for a bus; 3 for a tank 

INPUT INPUT INPUT INPUT

OUTPUT = 3 OUTPUT = 2 OUTPUT = 1 OUTPUT=1

 Value calculated using
all the input unit values Choose Cat A

General Idea
(largest output value)

INPUT LAYER HIDDEN LAYERS OUTPUT LAYER CATEGORY

1.1

NUMBERS OUTPUT
NUMBERS INPUT

1.1 Cat A
2.7 7.1
4.2
3.0 0.2 Cat B
2.1
-0.8
-1.3 0.3 Cat C
-1.2

2.7

VALUES PROPAGATE THROUGH THE NETWORK

:-Calculation Example
Categorisation of 2x2 pixel black & white
images
Into “bright” and “dark”
Representation of this rule:
If it contains 2, 3 or 4 white pixels, it is “bright”
If it contains 0 or 1 white pixels, it is “dark”
Perceptron architecture:
Four input units, one for each pixel
One output unit: +1 for white, -1 for dark
:-Calculation Example

Example calculation: x1=-1, x2=1, x3=1, x4=-1

 S = 0.25*(-1) + 0.25*(1) + 0.25*(1) + 0.25*(-1) = 0
0 > -0.1, so the output from the ANN is +1
 So the image is categorised as “bright”
 Step Sigma
Function

Unit Functions
Function

Linear Functions
 Simply output the weighted sum
Threshold Functions
 Output low values
 Until the weighted sum gets over a threshold
 Then output high values
 Equivalent of “firing” of neurons
Step function:
 Output +1 if S > Threshold T
 Output –1 otherwise
Sigma function:
 Similar to step function but differentiable
Learning In
Perceptron
Learning Process of ANN
Learn from
experience Compute
output
Learning algorithms
Recognize pattern of
activities
Involves 3 tasks Adjust No Is
Desired
Weight Output
Compute outputs achieved
Compare outputs
with desired targets yes
Adjust the weights
Stop
and repeat the
process
Training a
:-Perceptron

η -> Learning Rate

T -> target output
O -> output
X -> input
Worked Example
Return to the “bright” and “dark” example
Use a learning rate of η = 0.1
Suppose we have set random weights:
Worked Example
Use this training example, E, to update weights:

Here, x1 = -1, x2 = 1, x3 = 1, x4 = -1 as before

Propagate this information through the network:
 S = (-0.5 * 1) + (0.7 * -1) + (-0.2 * +1) + (0.1 * +1) + (0.9 * -1) = -2.2

Hence the network outputs o(E) = -1

But this should have been “bright”=+1
So t(E) = +1
Calculating the Error
Values
Δ0 = η(t(E)-o(E))x0
= 0.1 * (1 - (-1)) * (1) = 0.1 * (2) = 0.2
Δ1 = η(t(E)-o(E))x1
= 0.1 * (1 - (-1)) * (-1) = 0.1 * (-2) = -0.2
Δ2 = η(t(E)-o(E))x2
= 0.1 * (1 - (-1)) * (1) = 0.1 * (2) = 0.2
Δ3 = η(t(E)-o(E))x3
= 0.1 * (1 - (-1)) * (1) = 0.1 * (2) = 0.2
Δ4 = η(t(E)-o(E))x4
= 0.1 * (1 - (-1)) * (-1) = 0.1 * (-2) = -0.2
Calculating the New
Weights
w’0 = -0.5 + Δ0 = -0.5 + 0.2 = -0.3

w’1 = 0.7 + Δ1 = 0.7 + -0.2 = 0.5

w’2 = -0.2 + Δ2 = -0.2 + 0.2 = 0

w’3= 0.1 + Δ3 = 0.1 + 0.2 = 0.3

w’4 = 0.9 + Δ4 = 0.9 - 0.2 = 0.7

New Look Perceptron

Calculate for the example, E, again:

 S = (-0.3 * 1) + (0.5 * -1) + (0 * +1) + (0.3 * +1) + (0.7 * -1) = -1.2

Still gets the wrong categorisation

 But the value is closer to zero (from -2.2 to -1.2)
 In a few epochs time, this example will be correctly categorised
Time delay neural network (TDNN):-

is an alternative neural network architecture whose primary purpose

is to work on continuous data.

The advantage of this architecture is to adapt the network online

and hence helpful in many real time applications, like time series
prediction, online spell check, continuous speech recognition,etc.

The architecture has a continuous input that is delayed and sent as

an input to the neural network.

As an example, consider training a feed forward neural network

being trained for a time series prediction. The desired output of
the network is the present state of the time series and inputs to
the neural network are the delayed time series (past values).
Hence, the output of the neural network is the predicted next value
in the time series which is computed as the function of the past
values of the time series.
Recurrent Neural Network
 :-TYPES OF ANN
feed-forward
4.1 Feed-forward networks
Feed-forward ANNs allow signals to
travel one way only; from input to
output. There is no feedback (loops)
i.e. the output of any layer does not
affect that same layer. Feed-forward
ANNs tend to be straight forward
networks that associate inputs with
outputs. They are extensively used in
pattern recognition.
feedback
4.2 Feedback networks
Feedback networks can have signals
travelling in both directions by
introducing loops in the network.
Feedback networks are very powerful
and can get extremely complicated.
Feedback networks are dynamic;
 :-Some Topologies of ANN

Fully-connected feed-forward Partially recurrent network

Fully recurrent network

Recurrent Neural
:-Networks

recurrent neural network:-

is a class of
neural network where
connections between units
form a directed cycle. Partially recurrent network
This creates an internal
state of the network which
allows it to exhibit
dynamic temporal behavior

Fully recurrent network

References:-

[1] Simon Colton - www.doc.ic.ac.uk/~sgc/teaching/v231/

[2]
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report
.html
[3] http://www.willamette.edu/~gorr/classes/cs449/intro.html
[4] http://www.scribd.com/doc/12774663/Neural-Network-
Presentation
[5]
http://www.speech.sri.com/people/anand/771/html/node32.html
[6] http://en.wikipedia.org/wiki/Recurrent_neural_network