Introduction to Neural Networks

for Senior Design

August 9 - 12, 2004

Intro-1

Neural Networks: The Big Picture

Artificial
Intelligence
Neural
Networks

not ruleoriented

August 9 - 12, 2004

Expert
Systems

Machine
Learning

ruleoriented

Intro-2

Types of Neural Networks

Architecture
Recurrent

Feedforward
Supervised
Learning
No Feedback,
Training Data
Available

Learning
Rule

Unsupervised
Learning
August 9 - 12, 2004

Intro-3

What Is a Neural Network?

(Artificial) neural network, or (A)NN:

Information processing system loosely based on the model of

biological neural networks

Implemented in software or electronic circuits

Defining properties
Consists of simple building blocks (neurons)
Connectivity determines functionality
Must be able to learn
Must be able to generalize

August 9 - 12, 2004

Intro-4

Biological Inspiration for Neural Networks

Human Brain: 1011 neurons (or nerve cells)

Dendrites: incoming extensions, carry signals in
Axons: outgoing extensions, carry signals out
Synapse: connection between 2 neurons
Learning:
Forming new connections between the neurons, or
Modifying existing connections
axon

dendrite
synapse

Neuron 1
August 9 - 12, 2004

Neuron 2
Intro-5

From Biology to the Artificial Neuron, 1

axon

dendrite
synapse

Neuron 1

Neuron 2
w2,k

Neuron 1

, f1

w2,1

, f2

Neuron 2

w2,n
b
August 9 - 12, 2004

Intro-6

From Biology to the Artificial Neuron, 2

w2,k
Neuron 1

, f1

w2,1
w2,n

, f2

Neuron 2

The weight w models the synapse between two biological neurons.

Each neuron has a threshold that must be met to activate the neuron,
causing it to fire. The threshold is modeled with the transfer function, f.
Neurons can be
excitatory, causing other neurons to fire when they are stimulated; or
inhibitory, preventing other neurons from firing when they are
stimulated.
August 9 - 12, 2004

Intro-7

Applications of Neural Networks

Aerospace: aircraft autopilots, flight path simulations, aircraft control

systems, autopilot enhancements, aircraft component simulations

Banking: credit application evaluators

Defense: guidance and control, target detection and tracking, object

discrimination, sonar, radar and image signal processing including
data compression, feature extraction and noise suppression,
signal/image identification

Financial: real estate appraisal, loan advisor, mortgage screening,

stock market analysis, stock trading advisory systems

Manufacturing: process control, process and machine diagnosis,

visual quality inspection systems, computer chip quality analysis
August 9 - 12, 2004

Intro-8

Applications of Neural Networks, cont.

Medical: cancer cell detection and analysis, EEG and ECG analysis,
disease pathway analysis

Communications: adaptive echo cancellation, image and data

compression, speech synthesis, signal filtering

Robotics: Trajectory control, manipulator controllers, vision systems

Pattern Recognition: character recognition, speech recognition,

voice recognition, facial recognition

August 9 - 12, 2004

Intro-9

Problems Suitable for Solution by NNs

Problems for which there is no clear-cut rule or

algorithm
e.g., image interpretation

Problems that humans do better than traditional

computers
e.g., facial recognition

Problems that are intractable due to size

e.g., weather forecasting

August 9 - 12, 2004

Intro-10

Math Notation/Conventions

Scalars: small italic letters

e.g., p, a, w

Vectors: small, bold, non-italic letters

e.g.,
p, a, w

Matrices: capital, bold, non-italic letters

e.g., P, A, W

Assume vectors are column vectors (unless stated

otherwise)
1
e.g., p =
2

August 9 - 12, 2004

Intro-11

Network Architecture and Notation

Single-Input Neuron
Neuron
Scalar
Input
n
w
p

Scalar
Output

b
1

Network Parameters (weight: w, bias: b)

a = f(n) = f(wp + b)

Adjusted via learning rule

Net Input: n
Transfer Function, f (design choice)

August 9 - 12, 2004

Intro-12

Transfer Functions Hard Limiter

a
1
0

a = f(n) = 0, n < 0
1, n 0
MATLAB: a = hardlim(n)
(often used for binary classification problems)
August 9 - 12, 2004

Intro-13

Transfer Functions Symmetric Hard Limiter

a

1
n

0
-1
a = f(n) = -1, n < 0
1, n 0

MATLAB: a = hardlims(n)
(often used for binary classification problems)
August 9 - 12, 2004

Intro-14

Transfer Functions - Linear

a
slope: 1
y-intercept: 0

a = f(n) = n
MATLAB: a = purelin(n)
(often used in network training for classification problems)
August 9 - 12, 2004

Intro-15

Transfer Functions Log-Sigmoid

a
1
0.8
0.6
0.4
0.2
0
-4

-2

a = f(n) =

1
1 + e n

MATLAB: a = logsig(n)
(often used for training multi-layer networks with backpropagation)
August 9 - 12, 2004

Intro-16

Transfer Function Summary

Function

Equation

Output Range

MATLAB

Hard limiter

Discrete: 0 or 1

hardlim

Symmetric
Hard Limiter

a = 0,
1,
a = -1,
1,

Discrete: 1, -1

hardlims

Linear

a=n

Continuous: range of n

purelin

Log-Sigmoid

a= 1
1 + en

Continuous: (0, 1)

logsig

Continuous: (-1, 1)

tansig

Discrete: 0, 1

compet

Hyperbolic
Tangent
Sigmoid
Competitive

August 9 - 12, 2004

n<0
n0
n<0
n0

(
en e n )
a=
(en + en )
a = 1, neuron w/
max n
(0 else)

Intro-17

Multiple-Input Neurons

Consider a network with R scalar inputs: p1, p2, , pR :

Scalar
Inputs
p1
w1,1
p2
w1,2
.
.
.
w1,R
pR

Multiple-Input
Neuron
n

Scalar
Output

b
1

a = f(n) = f(Wp + b)

where W is the weight matrix with one row: W = [w1,1 w1,2 w1,R]

p1
and p is the column vector: p = M

pR

August 9 - 12, 2004

Intro-18

Multiple-Input Neurons: Matrix Form

Multiple-Input
Neuron

Input
p
Rx1
1

W
1xR

n
+

f
1x1

1x1

1x1
R

August 9 - 12, 2004

a = f(n) = f(Wp + b)

Intro-19

Binary Classification Example [Ref. #5]

Consider the patterns:

Features: # of vertices*, # of holes, symmetry (vertical or

horizontal)
yes(1) or no (0)

Noise-free observation vectors or feature vectors

4
2
A 1,
F 0
0

1

* Here a vertex is an intersection of 2 or more lines.
August 9 - 12, 2004

Intro-20

Binary Classification Example, continued

Targets (desired outputs for each input):

t1 = -1
(for A)

t2 = 1
(for F)

Neural Network (Given):

p1

.42

p2

-1.3

p3

-1.3

f = hardlims

a = hardlims(Wp)

W = [.42 -1.3 -1.3]

August 9 - 12, 2004

Intro-21

Binary Classification: Sample Calculation

p1

.42

p2

-1.3

p3

-1.3

f =hardlims

a = hardlims(Wp)

W = [.42 -1.3 -1.3]

Calculating the neuron output if the input pattern is the letter A:

Wp = [.42 -1.3 -1.3] 4 = -.92
1
1

a = hardlims(Wp) = -1
Find Wp and the neuron output a if the input pattern is F?
August 9 - 12, 2004

Intro-22

A Layer of 2 Neurons (Single-Layer Network)

Sometimes we need more than one neuron to solve a problem.

Example consider a problem with 3 inputs and 2 neurons:

w1,1
w2,1

p1

n1

a1

b1

w1,2
p2

p1
p = p2
p
3

1
w2,2

p3

w1,3
w2,3

n2
b2

1
August 9 - 12, 2004

w1,1 w1,2 w1,3

W = w

2,1 w 2,2 w 2,3

a2

a f( W : row(1) p + b1 )
a = 1 =
a2 f( W : row(2) p + b2 )
= f( Wp + b)
Intro-23

Weight Matrix Notation

Recall for our single neuron with multiple inputs, we used weight
matrix W with one row: W = [w1,1 w1,2 w1,R]

General Case (multiple neurons): components of W are weights

connecting some input element to the summer of some neuron

Convention (as used in Hagan), for component wi,j of W

First index (i) indicates the neuron # the input is entering
the to index
Second index (j) indicates the element # of input vector p that
will be entering the neuron
the from index

wi,j = wto,from
August 9 - 12, 2004

Intro-24

Single Layer of S Neurons: Matrix Form

Layer of S Multiple-Input Neurons
Input
p
Rx1
1

W
SxR
b
Sx1

August 9 - 12, 2004

n
+

f
Sx1

Sx1

S
a = f(Wp + b)

Intro-25

Multiple Layers of Neurons

Allow each layer to have its own weight matrix (W), bias vector (b),
net input vector (n), output vector (a), and # of neurons (S)

Notation: superscript on the variable name indicates the layer #:

1
2
e.g., W : weight matrix for layer #1, b : indicates bias vector for
layer #2, a3: output vector for layer #3
4
th layer
e.g., S : # of neurons in the 4

Output of layer 1 is input to layer 2, etc.

The last (right-most) layer of the network is called the output layer;
the inputs are not counted as a layer at all (per Hagan); layers
between the input and output are called hidden layers.

August 9 - 12, 2004

Intro-26

2-Layer Network: Matrix Form

Layer 1
(Hidden)

Layer 2
(Output)

Input
p
Rx1
1

W1
S1xR

b1
S1x1

n1
+

a1

f1

S1x1

S1x1
S1

S2xS1

b2
S2x1

R
a1 = f1(W1p + b1)

August 9 - 12, 2004

W2

n2
+

f2

S2x1

a2
S2x1

S2

a2 = f2(W2a1 + b2)

Intro-27

Introduction: Practice Problem

p1

n1

a1

1) For the neural

network shown,
find the weight
matrix W and the
bias vector b.

a2

2) Find the output if f

= compet and
the input vector is
p = p1 = 1 .
p2 2

3
1
5

p2

n2

-2
1
a = compet(Wp + b)
where compet(n) = 1, neuron w/max n
0, else
August 9 - 12, 2004

Intro-28

Project Description

August 9 - 12, 2004

Intro-29

Is that a tank
or a tree?

Computer Neural Network

Project Overview

August 9 - 12, 2004

Intro-32

RC
Tank/platform/
clutter

Image Capture
Software (Image
Acquisition
Toolbox
MATLAB)

Data Processing
to obtain NN
Inputs

Neural Network

Movement
direction for
camera

Image from Camera

Camera to
Computer
Interface

Video
Camera

Tilt/Pan
Servos

Servo
Controller

Computer
Interface to
Servo Controller

These components may be combined

in one or more physical units

Phase 1: How do we get from A to B?

Phase 2 (main project): How to we get from B to A?