Escolar Documentos
Profissional Documentos
Cultura Documentos
Made up of Nerve Cells called Neurons Neurons Receive & Transmit information between various parts/organs of the body Sensory (Receptor) Neuron, Motor Neuron, Inter-Neuron Transmission of signal is a complex electro-chemical process
Biological Neuron
Cell Body Soma
Has a Nucleus
Dendrites
Fiber-like; large number; branched structure Receive signals from other neurons
Axon
One per neuron; longer & thicker; branched at its end Transmits signals to other neurons Contains Vesicles, which hold chemical substance called neural transmitters
Pre-synaptic neuron
Transmitting neuron
Post-synaptic neuron
Receiving neuron
The Synapse
Neuron Signals
Complex electro-chemical process Incoming signals raise or lower the electrical potential inside the neuron If potential crosses a threshold, a short electrical pulse is produced
We say the neuron fires [is triggered or activated]
The pulse is sent down the axon electrical activity inside the neurons Chemical activity occurs at the synapses Vesicles in the axon release chemical substance, called neural transmitters These are collected by dendrites of receiving neuron This raises/lowers electric potential in the receiving neuron
Neuron Signals
Each neuron receives lot of input signals thru its dendrites
from many other neurons
Output depends on all inputs Cell body acts like a summing & processing device Process depends on type of neuron
Characteristics of Biological NN
Robustness & Fault Tolerance
Decay of nerve cells does not seem to affect performance significantly
Flexibility
Automatically adjusts to new environment
Collective Computation
Massively Parallel Distributed
Aspect
Computer
Biological NN
Numeric: Faster Patterns: Slower Sequential Less complex in memory locations; addressable; fixed capacity; new info overwrites old info no centralized
Slower Faster Massively Parallel Very Complex In the strengths of the interconnections; Adaptable size, to add new info yes distributed
Output signal
value
a1
a2 ai an
aiwi + b
f(x)
output function
s
[s= f(x)]
Summing part
Output part
Output Function
Binary : if x>=t, s=1 else, s=0 (t = threshold)
1 0 x t s
Linear :
s x s=kt
0
Ramp :
Sigmoid : s = 1/(1+e-x)
a1 a2 a1 a2 -1 s 0 0 0 1
x 1 0
s 1 0
-1
1
1
0
1
0
-1
0
0
Perceptron
Inputs are first processed by Association Units Weights are adjustable, to enable Learning Actual output is compared with desired output; the difference is Error Error is used to adjust the weights, to obtain desired output
Perceptron (contd)
a1 w1
w2 a2
aiwi + b
x f(x) s
Output function
Perceptron (contd)
Expected output = s Actual output = s Error = = s s Weight change = wi = ai is the Learning Rate parameter
Perceptron Learning
Perceptron Learning Rule
Procedure for adjusting the weights
If weight adjustments lead to zero-error, we say it converges Whether error reduces to 0, depends on nature of desired input-output pairs of data Perceptron Convergence Theorem
To determine whether desired input-output pairs are representable [achievable]
Adaline
Adaline = Adaptive Linear Element Similar to Perceptron; difference is :
Employs Linear Output Function (s=x)
Weight update rule minimises the mean squared error, averaged over all inputs
Hence known as LMS (Least Mean Squared) Error Learning Rule Also known as : Gradient Descent Algorithm
Terminology
Processing Unit
Summing part, output part Inputs, weights, bias, activation value Output function, output signal
Update
Synchronous, Asynchronous
Models may be electronic / software Such networks are called Artificial Neural Networks [ANN]
ANN
ANNs exhibit abilities surprisingly similar to Biological NNs They can Learn, Recognize, Remember, Match & Retrieve Patterns of Information Hardware implementations of ANN are also available nowadays
Costly but faster than software implementation
Topology
Topology is the physical organisation of the ANN Arrangement of the processing units, interconnections & pattern input & output ANN is made up of Layers of Neurons All Neurons within one layer have same activation dynamics & output function In addition to interlayer connections, intralayer connections may also be made Connections across the layers may be in feedforward or feed-back manner
Topology (contd)
One Input layer, one output layer zero or more intermediate layers (usually referred as hidden layers) No limit on no. of layers There can be any no. of neurons in any layer; all layers need not have same no. of neurons If there is no hidden layer, the ANN is called single-layer network If one or more hidden layers are present, it is called multi-layer network
Topology (contd)
Feedforward Networks
the units are connected such that data flows only in forward direction, ie. from input layer to output layer, via successive hidden layers if any
Feedback Networks
data flows in forward direction, as above in addition, the connections allow data flow from output layer towards input layer also the reverse flow (feedback) is for errorcorrection, for adjusting weights suitably to get desired output, which is an essential feature of the mechanism for NN Learning
Feedback Network
Neuronal Dynamics
Operation of NN governed by Neuronal Dynamics
Dynamics of activation state Dynamics of synaptic weights
Short term Memory (STM) modelled by activation state of the NN Long Term Memory (LTM) corresponds to encoded pattern of info in synaptic weights
Machine Vision
Image & Pattern Recognition Intelligent l Medicine Devices Intelligent Security Systems
42
Patterns
Computers deal with Data Humans deal with Patterns Objects/Images, voices/sounds, even actions [walking etc] have patterns Different images, sounds & actions have different patterns Patterns enable us to recognise, classify & identify objects & to take decisions based on such identification
Pattern Mapping
Pattern Clustering (aka Pattern Grouping) Feature Mapping
Pattern Association
Every input pattern is associated with an output pattern, to form a pair of input-output patterns There will be many such pairs of input-output patterns A well-designed ANNs can be trained to learn (remember) many such pairs of patterns Whenever a pattern is input, the ANN should retrieve (output) the corresponding output pattern Supervised Learning has to be employed [being taught] This is purely a memory function & is called auto-association task
Pattern Classification
Objects belonging to the same class have many common features/patterns This fact enables us to classify objects into classes & to identify new classes Supervised Learning the patterns for each class has to be taught to the system Pattern classification tasks must exhibit accretive behaviour ie. an incomplete or noisy input should poduce an output corresponding to its closest known input pattern Example of Pattern Classification task: Voice Recognition, Handwriting Recognition
Pattern Mapping
Capturing the relation between the input pattern & its corresponding output pattern This is a generalisation task, not mere memorising This is called interpolative behaviour Example of Pattern Mapping task: Speech Recognition
Pattern Clustering
Identifying subsets of patterns having similar distinctive features & grouping them together Sounds similar to Pattern Classification, but is not the same Has to employ Unsupervised Learning
Classification
Patterns for each class are input separately That is, system is trained to learn patterns of one class first Then it is taught the patterns of another class
Clustering
Patterns belonging to several groups are mixed in the set of inputs System has to resolve them into different groups
Feature Mapping
In several patterns, the features may not be unambiguous May vary over a time-period Therefore, difficult to cluster In this case, system learns a feature map rather than clustering or classifying Has to employ unsupervised learning Example: you see a new object - for the first time never seen it before - & it has some distinct features, as well some features common to many known classes or groups
Simplest Combination of FF & FB NW [aka Competitive Learning NW] is a singlelayer NW in which the units in output layer have feedback connections among themselves
FB NN
Auto-Association, Pattern Storage (LTM), Pattern Environment Storage (LTM)
FF & FB (CL) NN
Pattern Storage (STM), Clustering & Feature Mapping
FF NN Pattern Association
a1 a2 a5 a3 a6 a4 b4 b1
b2
b3
For input pattern ai, the corresponding output pattern is bi. a5 & a6 are noisy versions of a3. In a5 the noise is less, it is nearest to a3 - NW outputs b3 [desired], it is accretive. In a6 noise is more, it is nearer to a4 than a3 NW may output b4.
Real-Life Example
Inputs are 8x8 grids of pixels of binary values. Input pattern space is a binary 256-dimensional space.
A
B
Outputs are 5-bit binary numbers (7-bit ascii characters). Output pattern space is binary 5-dimensional space.
Noisy versions of input patterns can occur, when some values of some pixels get changed, due to noise in transmission channel or dust/stain spots on the document being scanned.
FF NN Pattern Classification
Some of the output patterns may be identical So, a set of input patterns may correspond to the same output pattern No. of distinct output patterns = a class label Input patterns corresponding to each class = samples of that class In such cases, the NN has to classify the input patterns That is: for each input pattern, the NN should identify the class [output pattern] to which it belongs
Real-Life Example
A 1000001
B 1000010
CL NN Pattern Classification
Accretive behaviour
FF NN Pattern Mapping
NN is trained with some pairs of input-output patterns, not all possible pairs When a new input pattern is given, the NN is made to find the coresponding output pattern [though the NN was not trained with this pair] Suppose the NN has been trained with i/o pairs a n & bn If the new input pattern am is closer to some known input pattern am, the NN tries to find an output pattern b which is closer to bn Interpolative behaviour
a3
a4 a5
NN trained with (a1,b1) to (a5,b5) only; not trained with (a6,b6) pair. a6 closer to a3; so, NN maps it on to b6, which is closer to b3.
FB NN Pattern Association
If input patterns are identical to output patterns, input & output spaces are identical Problem reduces to auto-association
trivial; the NW merely stores the input patterns
If a noisy pattern arrives at input, NW outputs the same noisy pattern as output
Absence of accretive behaviour
a1 a2 a5
a1 a2 a5
a3
a4
a3
a4
CL NN Pattern Clustering
Patterns are grouped, based on similarities Input is an individual pattern; ouput is the pattern of group to which the input belongs That is : a group of approximately similar patterns are identified with one & the same cluster label & will produce the same output pattern Two types possible : new input pattern, not belonging to any group, is
forced to one of the groups (Accretive behaviour) shown as belonging to a new group
Input is close to some known input pattern x New group is close to xs group (Interpolative behaviour)
Interpolative behaviour
CL NN Feature Mapping
Similar to clustering; difference is: Similar inputs produce similar output [not the same output] similarities of inputs is retained at the output No accretive behaviour; only interpolative Output patterns are much larger [than for clustering]
Types of Learning
Supervised Learning
desired response [output] of the system is already known/decided network is made [tuned/trained] to give the desired output it is as if the network is "being taught or trained by a teacher"
Unsupervised Learning
output is not known network is allowed to settle into stable state by itself It is as if the network is discovering special features and patterns from available data without using an external help
Learning Equation
Implementation of Synaptic Dynamics Expression for updating of weights Express the weight vector of ith processing unit at time instant t+1, in terms of that weight vector at time instant t wi(t+1) = wi(t) + wi(t) wi(t) is the change in the weight vector Different researchers have proposed different expressions for calculating wi(t); these are called Learning Laws
Learning Laws
Hebbs Law [Hebbian Learning Law] Perceptron Learning Law Delta Learning Law LMS Learning Law Correlation Learning Law Instar [winner-take-all] Learning Law Outstar Learning Law
Boltzmann Learning
Stochastic Learning Algorithm A Network designed to apply Boltzmann Learning Rule is called Boltzmann Machine The neurons constitute a recurrent structure & give binary output [+1 or -1] corresponding to whether the neuron is on or off
Memory-based Learning
Past experiences = patterns which the NN has been trained to recognise/classify Each experience is a pair of input & output patterns All or most of the past experiences are stored in a large memory Any new input pattern can be compared with patterns stored in memory & the corresponding output pattern can be output
Learning Law
Initial Weights
Type of Learning
Remarks
Hebbian
Near zero
Unsupervised
Perceptron
Random
Supervised
Delta
Random
Supervised
Widrow-Hoff (LMS)
[bi - wiTa] aj
Random
Supervised
Correlation
bi aj (aj wkj)
Near zero
Supervised
Winner-Take-All (Instar)
Unsupervised
Outstar
(bi wjk)
Zero
Supervised
Grossberg Learning