Você está na página 1de 32

BlueEyes

Human
Operator
Monitoring
System

•Motivation
•What is
BlueEyes ?
BlueEyes
•Physiological Human-Operator Monitoring System
foundations
•System
design
•System Team members Paweł Kowalik
implementation
•Future
Piotr Kubiaczyk
improvements Krystian Nowak
•Project work Stanisław Osiński
summary
•System
Tomasz Pużak
demonstration
Team mentor Jan Kniat, Ph. D.

Poznań University of Technology, Poland


http://www.put.poznan.pl
BlueEyes
Human
Operator

Agenda
Monitoring
System

•Motivation
•What is
BlueEyes ?
• Motivation
•Physiological
foundations • What is BlueEyes (and what is not) ?
• Physiological foundations
•System
design
•System
implementation • System design
•Future
improvements
•Project work
• System implementation
summary
•System
• Future improvements
demonstration
• Project work summary
• System demonstration
BlueEyes
Human
Operator

Motivation
Monitoring
System

•Motivation
•What is • Human error - a frequent reason of
BlueEyes ?
•Physiological catastrophes and ecological disasters
foundations
•System – temporal indisposition
design weariness

•System oversight

implementation Monitoring of conscious

brain involvement
•Future
improvements Capable sensors available

•Project work
summary
•System
demonstration

• Lack of complex solution exploiting wireless


link technology, database systems and AI
algorithms
BlueEyes
Human
Operator

What is BlueEyes ?
Monitoring
System

•Motivation
•What is • The complex solution for human-operator
BlueEyes ?
•Physiological monitoring:
foundations
•System
– conscious brain involvement monitoring
design
•System
– triggering user-defined actions
implementation
•Future
– data recording
improvements
•Project work
summary
•System
demonstration
What is BlueEyes not?
• Doesn’t predict nor interfere with
operator’s thoughts
• Cannot force directly the operator to work
BlueEyes
Human
Operator

Who is BlueEyes for ?


Monitoring
System

•Motivation
•What is • Generic control rooms
BlueEyes ?
•Physiological – Power station
foundations
•System – Captain bridge
design
•System – Flight control centers
implementation
•Future – Operating theatres – anesthesiologists
improvements
•Project work
summary
•System
• Common application
demonstration
– A simpler system version for drivers
BlueEyes
Human
Operator

BlueEyes - benefits
Monitoring
System

•Motivation
•What is • Prevention from dangerous incidents
BlueEyes ?
•Physiological • Minimization of
foundations
•System – ecological consequences
design
•System – financial loss
implementation
•Future – a threat to a human life
improvements
•Project work
summary • The reconstruction
•System
demonstration of the course of
operator’s work
BlueEyes
Human
Operator

Physiological foundations
Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological Human brain
foundations
•System
design
•System Preconscious brain Conscious brain
implementation
•Future • alogical • manages the
improvements
•Project work • „automatic” knowledge
summary
•System
• logical reasoning
demonstration
• makes decisions
BlueEyes
Human
Operator

Conscious brain - operation modes


Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

Exploratory mode Planning mode


BlueEyes
Human
Operator

Eye movements
Monitoring
System

•Motivation
•What is • Retina - central vision (macula)
BlueEyes ?
•Physiological • Central vision and
foundations
•System the conscious brain
design
•System
implementation Saccade - a rapid
•Future
improvements eye jump to a new
•Project work
summary
location assigned
•System by the conscious
demonstration
attention process.

Saccades presence ⇔ Exploratory mode


BlueEyes
Human
Operator

The eye movement sensor


Monitoring
System

•Motivation
•What is • Eye position measuring - direct infrared
BlueEyes ?
•Physiological oculography
foundations
•System
• Oxy- and deoxyhemoglobin measurment
design
•System
• Two axial
implementation
accelerometer
•Future
improvements • Ambient light
•Project work
summary sensor
•System
demonstration

Jazz Multisensor
BlueEyes
Human
Operator

Wires, wires, wires...


Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological
foundations A serious limitation of:
•System
design – mobility
•System
implementation – comfort
•Future
improvements – safety
•Project work
summary
•System
demonstration
BlueEyes
Human
Operator

Personal Area Network !


Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological
foundations
Solution:
•System
design
•System
A wireless Personal Area
implementation
•Future
Network
improvements
•Project work
summary
•System
demonstration
BlueEyes
Human
Operator

Design objectives
Monitoring
System

•Motivation
•What is • A mobile data acquisition device
BlueEyes ?
•Physiological • A reliable real-time buffering, processing
foundations
•System and recording
design
•Objectives • A clear visualization interface
•System
overview • Mechanisms for creating custom data
•DAU analyzers
•CSU
• To ensure a possibility of distributing the
•System
implementation processing among a number of computers
•Future
improvements
•Project work
summary
•System
demonstration
BlueEyes
Human
Operator

System overview
Monitoring
System

•Motivation
•What is Data Acquisition Unit Central System Unit
BlueEyes ?
•Physiological
foundations
Atmel 89C52 Connection Visualization
•System Bluetooth Bluetooth Manager Module
design microcontroller
•Objectives
Jazz Data
•System Data Analysis
Multisensor Logger
overview
•DAU
•CSU

•System Bluetooth technology provides


implementation
•Future
means for creating a Personal
improvements Area Network linking the
•Project work
summary operators and the central
•System
demonstration
system.
BlueEyes
Human
Operator

DAU - features
Monitoring
System

•Motivation
•What is Data Acquisition Unit
BlueEyes ? • Lightweight
•Physiological
foundations • Runs on batteries - low
Atmel 89C52
•System
design microcontroller
Bluetooth power consumption
•Objectives
Jazz • Easy to use - does not
•System
overview Multisensor disturb the operator
•DAU
working
•CSU
• ID cards for operator
•System
implementation authorization
•Future
improvements • Voice transmission using
•Project work
summary
hardware PCM codec
•System
demonstration
BlueEyes
Human
Operator

CSU - features
Monitoring
System

•Motivation
•What is Central System Unit
BlueEyes ? • Connection
•Physiological
foundations
management
Connection Visualization
•System
design
• Data processing Bluetooth Manager Module

•Objectives
• Visualization Data Analysis
Data
•System Logger
overview
• Data recording
•DAU
•CSU

•System
implementation
• Access
•Future verification
improvements
•Project work • System maintenance
summary
•System
demonstration
BlueEyes
Human
Operator

CSU - saccade detector


Monitoring
System

•Motivation
•What is • Saccade detector and conscious brain
BlueEyes ?
•Physiological involvement level calculation
foundations
Eye movement velocity
•System
Raw eye 250 saccade
design 200
movement data 150
•Objectives 100

Velocity
50
•System 0
-50
overview -100
-150
-200
•DAU -250

•CSU Differentiation 0 8000 16000 24000 32000


Time [ms]

•System
implementation
•Future
improvements Conscious
Visual
•Project work
brain
summary involvement
level
•System 60
demonstration
Lva)

50
BlueEyes
Human
Operator

CSU - Visualization Module


Monitoring
System

•Motivation
•What is • Raw and processed data visualization
BlueEyes ?
•Physiological using:
foundations
•System – VU-meters
design
•Objectives – pie-charts
•System
overview – time series
•DAU
•CSU

•System
implementation • audio and video
•Future
improvements data playback
•Project work
summary
•System
demonstration
BlueEyes
Human
Operator

Data security
Monitoring
System

•Motivation
•What is • Only registered mobile devices
BlueEyes ?
•Physiological can connect to the system
foundations
•System
• Bluetooth connection authentication
design
•Objectives • Bluetooth connection encryption
•System
overview • Access rights restrictions
•DAU
• Personal and physiological data encryption
•CSU

•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
BlueEyes
Human
Operator

Implementation - DAU
Monitoring
System

•Motivation
•What is • Made from scratch by the team
BlueEyes ?
•Physiological • Bluetooth module inside
foundations
•System • Jazz sensor
design
•System
interface
implementation

•DAU
• Microphone and
•CSU earphone sockets
•Future
improvements • 5-key keyboard
•Project work
summary • Two lines, 8
•System characters LCD
demonstration
• Powered from
6 AA batteries
BlueEyes
Human
Operator

Implementation - DAU
Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological

PCM codec board Microcontroller board


foundations
•System
design
•System
implementation

•DAU
•CSU
•Future
improvements Batteries and
•Project work
summary
voltageBluetooth module
level monitor
•System
demonstration
BlueEyes
Human
Operator

Implementation - CSU
Monitoring
System

•Motivation
•What is • Standard MDI user
BlueEyes ?
interface
•Physiological
foundations • Custom-built
•System
design
Bluetooth
•System communication
implementation
routines
•DAU
•CSU • Intel USB camera
•Future
improvements
• ID card programmer
•Project work
summary
• Developed using Visual C++ 6.0
•System • Deployment environment:
demonstration
Windows 2000,
MS SQL Server
BlueEyes
Human
Operator

Future improvements
Monitoring
System

•Motivation
•What is • DAU
BlueEyes ?
•Physiological – small CMOS camera to monitor the
foundations
•System
operator’s point of gaze
design
•System
– single PCB (SMD technlogy)
implementation
•System
– low voltage ICs - LiIO batteries power
demonstration
•Future
improvements
• CSU
•Project work
summary
– data mining algorithms
– advanced database encryption using e.g.
AES algorithm
BlueEyes
Human
Operator

Project work summary


Monitoring
System

•Motivation
•What is
BlueEyes ? • Bluetooth and PAN technology acquired
•Physiological
foundations • New skills on designing software and
•System
design
hardware systems gained
•System
implementation
• A mobile device implemented
•System • 31 KLOC of C++ code and
demonstration
•Future 4 KLOC of 8051 assembler code written
improvements
•Project work
• All parts of the Project Kit efficiently used
summary
• New approach to system safety proposed
and implemented
BlueEyes
Human
Operator
Monitoring
System

•Motivation
•What is

BlueEyes
BlueEyes ?
•Physiological
foundations
•System
design
Human-Operator Monitoring System
•System
implementation
•Future
improvements
•Project work

System demonstration
summary
•System
demonstration
BlueEyes
Human
Operator
Monitoring
System

•Motivation
•What is
BlueEyes ?
•Physiological
foundations
Thank you for your attention
•System
design
•System
implementation
•System
demonstration
•Future
improvements
BlueEyes
•Project work
summary
Human-Operator Monitoring System

Poznań University of Technology, Poland


http://www.put.poznan.pl
BlueEyes
Human
Operator

DAU - components
Monitoring
System

microphone MC 145483
earphone PCM codec

Jazz Multisensor Bluetooth


module
Beeper Atmel 89C52

UART
microcontroller
LCD display
UART MAX232
LED indicators
BlueEyes
Data Acquisition Unit
ID card Simple
interface keyboard
ID card
BlueEyes
Human
Operator

CSU - Connection Manager


Monitoring
System

Operator
Data Initializes
••Data demultiplexer Bluetooth
Manager modules
•Data producer queues

• Inquires for new


Separated
Bluetooth devices
Operator
Raw Data
Buffer
in range
data streams
Manager
• EstablishesAlert buffer Alerts

connections
... Bluetooth ...
Connection • Supports operator
... Manager roaming
...
• Feeds Operator
Raw Data
Manager
Operator Buffer with
Manager
Bluetooth physiological
Alert buffer data
Alerts
BlueEyes
Human
Operator

CSU - Data Analysis


Monitoring
System

• Custom data analyzers - decision tree


based classifiers

Example: Operator gesture recognition


(simplified)

Vertical Acceleration variance


value high
Horizontal Y N
Acceleration variance Horizontal Acceleration
high variance high
N Y N Y

“YES” No answer “NO”


BlueEyes
Human
Operator

CSU - Data Analysis


Monitoring
System

• Custom data analyzers - decision tree


based classifiers
• Automatic decision tree coefficients
generation - C4.5 based algorithm
Assigned
Data samples category

“Yes”

No
... answer
... Decision tree
“Yes” coefficients
“No”

No
answer
BlueEyes
Human
Operator

CSU - intermodule communication


Monitoring
System

• Uses buffered thread-safe queues and


producer-consumer scheme
register at

Data
consumer
creates

Data
Data Data Data
Processing
consumer & Data
consumer
producer queue producer queue

Data
consumer
Data
Data Data
Processing
consumer &
consumer
producer queue
BlueEyes
Human
Operator

CSU - intermodule communication


Monitoring
System

Connection
Manager

Operator manager (OM) ... OM OM

Alerts Separated physiological


data streams

Visualization Data
Data Analysis
Module Logger

Processed data

Recorded (off-line) data

Você também pode gostar