AFOSR

SENSORY INFORMATION
SYSTEMS
14 March 2011
WILLARD LARKIN
Program Manager
AFOSR/RSL
Air Force Office of Scientific Research
Distribution A: Approved for public release; distribution is unlimited. 88ABW-2011-0780
 2307/C PORTFOLIO OVERVIEW

Program Manager: Willard Larkin

BRIEF DESCRIPTION OF PORTFOLIO:

• Auditory modeling for acoustic analysis

• Biological polarization optics & vision
• Sensori-motor control of bio- flight & navigation

SUB-AREAS IN PORTFOLIO:
Sensory Information Systems (2307/C)

2
 Program Trends and Strategy:
TOPIC AREA OVERVIEW
3 +6
42%
Advanced Auditory Modeling:
Scientific Question: How does the auditory brain parse acoustic
landscapes, bind sensory inputs, adapt its filters, hear through noise and
distortion? Could autonomous listening devices emulate neurology to match or
exceed human auditory analysis, e.g., to detect and identify speech targets in
noise and reverberation?
11%
Polarization Vision & Optics:
Scientific Question: How do natural photoreceptors detect and how do
animal brains interpret polarization information? How is it used for nocturnal
navigation or recognition of obscured targets? Can these unique bio-optical
structures be emulated?
47%
Sensorimotor Control of Flight & Navigation:
Scientific Question: How does neural control make natural, low-Reynolds
No. flight autonomous, efficient, and robust? Discover principles of
multisensory fusion, distributed sensors and actuators. Develop control laws
for emulation in MAVs.

Primary Strategy: Forge useful connections between math and biology 3

 Other Organizations
That Fund Related Work
5

Funds 5 current AFOSR P.I.s in a new MURI, “Animal Inspired Robust

ONR Flight with Outer and Inner Loop Strategies.” (M. Steinberg)

Program, “Bio-Inspired Autonomous Systems,” (T. McKenna) has focus

ONR on aquatic environment, held joint review with AFOSR in May 2010

Coordinates psychoacoustics with AFRL/RH, and extends our

WRAMC 6.1 research to hearing-impaired populations. (D. Brungart)
Micro Autonomous Systems and Technology program for
ARL “MAST” urban and complex terrain funds 6.2 work on MAVs.

NRL Applies AFOSR 6.1 to MAV development; funds related 6.2, 6.3

Partners with AFOSR to support the annual workshop on

NSF Neuromorphic Modeling, involving several AFOSR P.I.s

Broad programs in biomathematics (M. Horn), perception, action &

NSF cognition (B. Tuller), NSF-NIH computational neuroscience

. . . plus international & 6.2 coordination 4

 International and 6.2
Coordination
U.K. U.S.
AFRL-Dstl Working Group
Biologically-Motivated Micro-Air-Vehicles

G. Taylor
“STATE OF THE ART REVIEW”
M. Willis
Georgia Tech
15-18 June, 2010

H. Krapp
S. Humbert

Organizers: M. Wehling, AFRL. P. Biggins, Dstl

30 Participants from UK, US, Industry, Academia, & Gov.

Presentations:
https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294 5
J. Niven &objAction=browse&viewType=1 T. Daniel
 Recent Highlights

Best Paper, 2010

AFRL Fellow Journal of Special Issue
2010 Experimental Hearing Research
Biology

Richard McKinley Chiu, et al., Paper on

Human Effectivenes Directorate competitive target capture Mechanics of
Acoustic Research by echolocating bats Hearing Workshop

6
 Two Recent Discoveries
Visual Photoreceptor Receptor Membrane
Discriminates L vs R Imparts a Uniform 1/4
Circular Polarization Wave Delay, 400 – 700 nm

Photoreceptor
rows 1-6 in O.
Scyllarus QUARTZ
compound eye OPTIMAL GRATING

2 Photoreceptor
Rows have
membrane for ¼
wave phase delay
Crossed parallel
microvilli in lower
section confer
linear pol
sensitivity
7
Comparison with Best Available Materials
T. Cronin, (UMBC), J. Marshal (Queensland), N. Roberts (Manchester)
 AFOSR Young Investigator Develops
Transformative Science

NORMAL
ACTION
POTENTIALS

• Math predicts a previously unknown &

unexpected mode of neural behavior:
prolonged electrical silencing, thought to
be physiologically unsustainable.
ELECTRICALLY
• Math captures details of intracellular SILENT
gene expression / transcription and ~ 5 Hrs.
membrane electrochemistry.

• Confirmed during 2010 in several labs

MEMBRANE
• Finding revises basic assumptions for DEPOLARIZATION
neural computation in cells & networks.
8
D. Forger, AFOSR YIP. Science 326, 9 Oct. 2009
 Recent Transitions
7
To Army: VLSI implementation of insect visual sensory fusion for a motion-
sensitive guidance system. Contract W56HZV-100192. (Dr. P. Shoemaker)
To AFRL 6.2: Spatial audio sorting, annotation, and retrieval system enhances
voice communications for multiple sources. Victor Finomore, RHCB. (Dr. N. Iyer)

To AFRL 6.2: Speaker recognition method, based upon binary mask technique,
boosts performance in additive noise. Brett Smolenski, RADC/Rome Lab. (Dr. Wang)

To Oticon A/S: Method to boost directional hearing via monaural tandem

analysis of pitch and voice segmentation. Dr.Ulrik Kjems, J. Woodruff. (Dr. Wang)
To EmergentViews.com: New technique for polarization imaging, based
upon 6.1 AFOSR work accomplished under the BioInspired Theme. (Dr. N. Engheta)

To NRL: Methods for cooperative steering of autonomous surface & air vehicles,
from control law mathematics based on bats & dragonflies. Dr. Justh, NRL/TEMD
To NSMRL: Simulators and technical method to calibrate underwater acoustic
stimulation of the human head. Dr. Michael Qin. (Creare, Inc. STTR, Dr. A. Dietz.)

To NSA & CIA: Method to sort speech from non-speech in noisy electronic 9
signals, based upon cortical model from experiments with ferrets. (Dr. S. Shamma)
 A 50 dB Technical Transition
6.1 Research in Support of DTO HS-33

Research
Goal: ≥ 150 dBA SPL
Discover how high-level sound
transmits through air, bone, and
tissue to the human cochlea.
Enable 50 dB acoustic isolation,
with no sacrifice of voice
communications.
(Legacy level was 30 dB)

AFOSR Launched Most AF Flight-Line Crews

suffer Hearing Loss
Multiple Efforts:
• Measure nonlinear loudness compression in bone conduction 6.1
• Model acoustic wave propagation through skull 6.1
• Develop a physiologically realistic, instrumented human head simulator STTR
• Measure and model dynamics of middle ear transduction STTR
• Develop new techniques for noise cancellation, active and passive STTR10
 Technical Transition
Achieved 50 dB Attenuation Goal of DTO HS-33

Coordinated 6.1, 6.2

efforts enabled a major
breakthrough in hearing
protection

Wright-Patt Bioacoustics Lab tests New

Helmet for Navy Carrier Crew

Instrumented Head Simulator

enabled acoustic tests without
Hanover, N.H. risk to human listeners

11
Program Coordination with: NATO, ONR, ARO, AFRL 711th HPW
 Auditory Modeling for Acoustic Analysis
and Audio Displays 3 +6

ResearchTopics
• 3D spatial audio displays to optimize human performance
REPORTED
IN RECENT • Cortical theory for speech detection and recognition
AFOSR
REVIEWS • Computational auditory scene analysis
• Modulation analysis of acoustic signals
0.1

0.05
NEW IN 2011:
Amplitude 0
Biophysical basis of
3D spatial hearing. -0.05

(Wm. Hartmann, MSU)

-0.1
0 0.2 0.4 0.6 0.8 1 1.2
Time (s)

Speech signals encode information in low-frequency

envelopes modulating high-frequency carriers
NEXT SLIDES:
Innovative signal processing ideas
based upon neural mechanisms 12
 “Synchrony Capture” in Auditory Nerve:
inspires method for multi-frequency tracking
COCHLEAR HAIR CELLS

INNER
ROW

OUTER
ROWS

Triplet Tuning

Adjacent groups of neural fibers

Cochlea behaves as a system of respond strongly to the same key
adaptive, tunable oscillators. frequencies in a vowel sound

13
R. Kumaresan, V. Peddinti (U. Rhode Island) & P. Cariani (Harvard), ICASSP 2011
 Auditory Model emulates Synchrony
Capture to achieve multi-pitch tracking
New architecture for
auditory signal processing
employs synchrony capture
via adaptive band-pass
filters that emulate cochlear
mechanics.

Spectrogram of a speech fragment overlaid

with multiple frequency tracks obtained via
synchrony capture.

NEXT GOAL: Use model to explain other phenomena, e.g., distortion

products, two-tone suppression, gain control. 14
R. Kumaresan, V. Peddinti (U. Rhode Island) & P. Cariani (Harvard), ICASSP 2011
 Neural Model generates Auditory Nonlinear
Resonances to match Neural Brain Signals
INPUT ENERGY
Background: F1 F2
Conventional auditory modeling
does not account for highly F2 / F1 = 1.7
nonlinear neural response
patterns, e.g. in inferior colliculus
or brainstem EEG.

Hypothesis:
These nonlinear patterns are
key to auditory cognition.

Progress:
Arithmetic combinations of F1 and F2,
arise here in brainstem data and in a
nonlinear coupled oscillator model, Neural Responses to a two-tone complex
fitted with one free parameter (gain).

STTR:
Circular 15
Florida Atlantic Univ., E. W. Large, F. Almonte, unpublished (used with permission.) Data from Lee, et al. 2009
Logic
 AFOSR / Rome Lab Symposium on
Coherent Modulation Analysis

. . . A key problem in decomposition of speech & other acoustic waveforms

Eight harmonics
3000

Complex Modulators Carriers

2500

K K
x(t )   xk (t )   mk (t )  ck (t )
2000

Frequency
1500

k 1 k 1
1000

Acoustic Signal
500

0
• Coherent Demodulation is a new 0.1 0.2 0.3 0.4 0.5 0.6
Time
0.7 0.8 0.9 1 1.1

approach for feature extraction and

8-Component Modulation Spectrum for a
improved intelligibility. Speech Sample
• Out-performs conventional incoherent,
AM/FM, and phase vocoding methods. Dr. Les Atlas, University of Washington

Participants J. Grieco, D. Harris, B. Pokines, J. Parker, S. Wenndt, K. Godin, A. Noga,

16
8 July 2010 J. Cupples, S. Johns, B. King, P. Clark, L. Atlas, W. Larkin
 NEXT: BIOSENSORY
Flight and Navigation

ResearchTopics
REPORTED Nocturnal navigation by echolocation or optical polarization
IN RECENT Neuromorphic emulation of inner- and outer-loop control
AFOSR
REVIEWS Airfoil mechanosensors in bats

NEW IN 2010 & 2011:

• Airfoil, antennae mechanosensors in
hawkmoths, insects
• Sensorimotor reflex basis for
cooperative formation control
• Bio-clock compensated optical
compass navigation

NEXT SLIDES -- RECENT DISCOVERIES:

• Insect flight stabilization & self-motion tuning
• Target tracking and pursuit in the dragonfly 17
 Discovery: Insect Brain Tracks Theta
Motion

Natural scenes combine “first-order”

motion with “second-order” motion

Discovery: Experimental Setup

• Fly can track second-order motion in
direction opposite to first-order motion

Theory:
• Two superimposed systems govern
flight steering: one phase-leading and
tuned to velocity, one phase-lagging
and tuned to position. Optokinetic
18
Theobald, Shoemaker, Ringach & Frye 2010, Behavioral Neuroscience Tracking
 Discovery: Insect’s Sensorimotor
Altimeter Ignores Ventral Angular Velocity

Scientific Question:
How do insects control
flying altitude?

Background:
• Flies were assumed to match
flying height to a preferred rate of
optic flow on the ground.
• Caltech’s “Velocity Clamp” lab
enables experimental control of
dynamic optic flow, independent of Virtual Reality wind tunnel
allows simultaneous tracking of
flight velocity in a wind tunnel. free flight & computer-projected
imagery on the walls and floor
Discovery:
• Flies adjust height to match nearby horizontal features.
• They do not regulate optic flow rate for this purpose.
• They rely on optomotor and collision-avoidance reflexes.
19
AFOSR Young Investigator. A. Straw, Caltech. Current Biology 19 Aug. 2010.
 Discovery: Receptive Field Self-Motion
Tuning is Conserved across Species

Maps of motion sensitivity in right visual field show a strong response

to yaw motion to the right and inhibition to forward translation

• Self-motion tuning is key to understanding gaze stabilization

• Tuning organization is conserved across these 3 species.

• Gaze stabilization is essential for flight control – keeps visual
sensors aligned with the inertial reference frame 20
H. Krapp, et al., AFOSR Report Nov. 2010. Imperial College, London
 Flight Control relies on Sensor Fusion in
Insect Optical Processing
Yaw
3 Ocelli
Rotation Fusion Site Discovered in
(polarization
receptors) Lobula Plate Neurons
• Slow Inputs from compound
eye combine with fast inputs
from polarization receptors
(reflex latencies are 20 to 30
Lobula Plate ms versus 6 ms for ocelli.)
VS Neuron
• Both are tuned to specific,
Rotation but different axes of rotation
Tuning Curve
• Ocellar system has lower
angular precision than the
Primary Rotation
compound eye system.
Axes of Ocellar
Neurons
• Sensory information aligns
with axes of natural modes of
insect’s flight instability 21
M. Parsons, et al., Current Biology 2010. Cambridge University.
 Insect Gaze Stabilization via
Feedback/Feedforward Sensor Fusion
Compound eyes plus halteres and ocelli
THORAX
ROLL Head Compound eyes plus halteres
Roll
HALTERES Compound eyes only
Head
Deflection
COMPOUND EYES
NECK MOTOR
SYSTEM

Theory:
• Short latency mechanosensors first
detect body rotation, then feed forward
to induce compensatory head roll via
neck motor system …
• Long latency visual system detects
residual optic flow feedback from
incomplete compensation.
• Data fit preliminary linear model.

22
H. Krapp, et al., AFOSR Report Nov. 2010. Imperial College, London
 Dragonfly Attacks Moving Target
21 sec. VIDEO Target trajectory

Tracking?

OR
Interception?

23
R. Olberg, Union College, A. Leonardo, Howard Hughes Medical Institute
 Evidence for “Interception”

Dragonfly moves Launch angle depends on

t = -28ms
gaze prior to launch target velocity prior to
takeoff

Launch angle
leads target

Dragonfly plans launch direction while still on its perch

24
R. Olberg, Union College, Schenectady, NY
 Dragonfly Predicts Prey Flight Path,
Updates Interception Path in Real Time

Neural System Analysis: Dragonfly Eyes

Relates pursuit time &
accuracy to small-target
selective descending (TSD)
neurons in ventral nerve cord.

~ 300 ms
Dragonfly perch
25
R. Olberg, Union College, Schenectady, NY
 Dragonfly visual neurons code for
both past and future target positions

Coding Past Position Predictive Coding

Neurons tuned to target

locations in the visual field
also convey information
about past, current, or
future target positions
Receptive field Receptive field
Information peaks 30 ms Information peaks 30 ms
before cell spikes after cell spikes

Neurons have nearly

orthogonal sensitivity
to direction of motion 26
R. Olberg, Union College, Schenectady, NY
 Next Research Stage:
Neural Telemetry in Free Flight
Flight Arena

Cameras

Multi-contact
probe in Perch Platform
mesothoracic
ganglion

Telemetry chip
mounts behind legs,
recharges on perch
platform.
27
R. Harrison, Univ. of Utah R. Olberg, Union College, A. Leonardo, Howard Hughes Medical Institute
 SUMMARY:
Transformational Impacts & Opportunities
4

Hearing protection:
• Massive improvements in high-noise attenuation.
Advanced auditory modeling:
• Mathematics for coherent modulation analysis
• Neural-Inspired analyses to parse acoustic scenes

Optical processing:
• Polarization vision and signaling adapted from biology
• Achromatic 1/4 wave optical retarders
• Emulating compound eye in new optical devices

Autonomous flight control:

• Adaptive airfoils based upon bio-sensory mechanisms
• Guidance from neural systems, not from networks
• Discover sensorimotor basis of formation flight
28
 Questions?

Thank you for your attention

Willard Larkin, Program Manager, AFOSR/NL
703-696-7793

Willard.Larkin@afosr.af.mil

29