INTRODUCTION

People express their mental states, including emotions, thoughts, and desires, all the
time through facial expressions, vocal nuances and gestures. This is true even when they are
interacting with machines. Our mental states shape the decisions that we make, govern how
we communicate with others, and affect our performance. The ability to attribute mental
states to others from their behavior and to use that knowledge to guide our own actions and
predict those of others is known as theory of mind or mind-reading.
Existing human-computer interfaces are mind-blind oblivious to the users mental
states and intentions. A computer may wait indefinitely for input from a user who is no longer
there, or decide to do irrelevant tasks while a user is frantically working towards an imminent
deadline. As a result, existing computer technologies often frustrate the user, have little
persuasive power and cannot initiate interactions with the user. Even if they do take the
initiative, like the now retired Microsoft Paperclip, they are often misguided and irrelevant,
and simply frustrate the user. With the increasing complexity of computer technologies and
the ubiquity of mobile and wearable devices, there is a need for machines that are aware of
the users mental state and that adaptively respond to these mental states.

WHAT IS MIND READING?

A computational model of mind-reading
Drawing inspiration from psychology, computer vision and machine learning, the
team in the Computer Laboratory at the University of Cambridge has developed mindreading machines computers that implement a computational model of mind-reading to
infer mental states of people from their facial signals. The goal is to enhance humancomputer interaction through empathic responses, to improve the productivity of the user and
to enable applications to initiate interactions with and on behalf of the user, without waiting
for explicit input from that user. There are difficult challenges:

Fig: Processing stages in the mind-reading system

Using a digital video camera, the mind-reading computer system analyzes a persons
facial expressions in real time and infers that persons underlying mental state, such as
whether he or she is agreeing or disagreeing, interested or bored, thinking or confused.

Prior knowledge of how particular mental states are expressed in the face is combined
with analysis of facial expressions and head gestures occurring in real time. The model
represents these at different granularities, starting with face and head movements and
building those in time and in space to form a clearer model of what mental state is being
represented. Software from Nevenvision identifies 24 feature points on the face and tracks
them in real time.
Movement, shape and colour are then analyzed to identify gestures like a smile or
eyebrows being raised. Combinations of these occurring over time indicate mental states. For
example, a combination of a head nod, with a smile and eyebrows raised might mean interest.
The relationship between observable head and facial displays and the corresponding hidden
mental states over time is modeled using Dynamic Bayesian Networks.

WHY MIND READING?

Current projects in Cambridge are considering further inputs such as body posture and
gestures to improve the inference. We can then use the same models to control the animation
of cartoon avatars. We are also looking at the use of mind-reading to support on-line
shopping and learning systems

Fig:Monitoring a car driver

The mind-reading computer system presents information about your mental state as
easily as a keyboard and mouse present text and commands. Imagine a future where we are
surrounded with mobile phones, cars and online services that can read our minds and react to
our moods. How would that change our use of technology and our lives? We are working
with a major car manufacturer to implement this system in cars to detect driver mental states
such as drowsiness, distraction and anger.
Current projects in Cambridge are considering further inputs such as body posture and
gestures to improve the inference. We can then use the same models to control the animation
of cartoon avatars. We are also looking at the use of mind-reading to support on-line
shopping and learning systems.
The mind-reading computer system may also be used to monitor and suggest
improvements in human-human interaction. The Affective Computing Group at the MIT
Media Laboratory is developing an emotional-social intelligence prosthesis that explores new
technologies to augment and improve peoples social interactions and communication skills.

HOW DOES IT WORK?

Fig: Futuristic headband

The mind reading actually involves measuring the volume and oxygen level of the
blood around the subject's brain, using technology called functional near-infrared
spectroscopy (fNIRS).
The user wears a sort of futuristic headband that sends light in that spectrum into the
tissues of the head where it is absorbed by active, blood-filled tissues. The headband then
measures how much light was not absorbed, letting the computer gauge the metabolic
demands that the brain is making. The results are often compared to an MRI, but can be
gathered with lightweight, non-invasive equipment.

Wearing the fNIRS sensor, experimental subjects were asked to count the number of
squares on a rotating onscreen cube and to perform other tasks. The subjects were then asked
to rate the difficulty of the tasks, and their ratings agreed with the work intensity detected by
the fNIRS system up to 83 percent of the time.
"We don't know how specific we can be about identifying users' different emotional
states," cautioned Sergio Fantini, a biomedical engineering professor at Tufts. "However, the
particular area of the brain where the blood-flow change occurs should provide indications of
the brain's metabolic changes and by extension workload, which could be a proxy for
emotions like frustration."
"Measuring mental workload, frustration and distraction is typically limited to
qualitatively observing computer users or to administering surveys after completion of a task,
potentially missing valuable insight into the users' changing experiences.
A computer program which can read silently spoken words by analyzing nerve signals
in our mouths and throats has been developed by NASA.
Preliminary results show that using button-sized sensors, which attach under the chin
and on the side of the Adam's apple, it is possible to pick up and recognize nerve signals and
patterns from the tongue and vocal cords that correspond to specific words.
"Biological signals arise when reading or speaking to oneself with or without actual
lip or facial movement," says Chuck Jorgensen.

HEAD AND FACIAL ACTION UNIT ANALYSIS

Twenty four facial landmarks are detected using a face template in the initial frame,
and their positions tracked across the video. The system builds on Facestation [1], a
feature point tracker that supports both real time and offline tracking of facial features on a
live or recorded video stream. The tracker represents faces as face bunch graphs [23] or
stack-like structures which efficiently com- bine graphs of individual faces that vary in
factors such as pose, glasses, or physiognomy. The tracker outputs the position of twenty
four feature points, which we then use for head pose estimation and facial feature extraction.
EXTRACTING HEAD ACTION UNITS

Natural human head motion typically ranges between 70-90o

55o

of upward pitch, 70o of yaw (turn), and 55o

as a combination of all three rotations [16].

of downward pitch,

of roll (tilt), and usually occurs

The output positions of the localized

feature points are sufficiently accurate to permit the use of efficient, image-based head pose
estimation. Expression invariant points such as the nose tip, root, nostrils, inner and outer
eye corners are used to estimate the pose. Head yaw is given by the ratio of left to right eye
widths. A head roll is given by the orientation angle of the two inner eye corners. The
computation of both head yaw and roll is invariant to scale variations that arise from moving
toward or away from the camera. Head pitch is determined from the vertical displacement
of the nose tip normalized against the distance between the two eye corners to account for
scale variations. The system supports up to 50o , 30o

and 50o

of yaw, roll and pitch

respectively. Pose estimates across consecutive frames are then used to identify head action
units.

For example, a pitch of 20o degrees at time t followed by 15o at time t + 1

indicates a downward head action, which is AU54 in the FACS coding.

EXTRACTING FACIAL ACTION UNITS
Facial actions are identified from component-based facial features (e.g.mouth)
comprised of motion, shape and color descriptors. Motion and shape-based analysis are
particularly suitable for a real time video system, in which motion is inherent and places a
strict upper bound on the computational complexity of methods used in order to meet time
constraints. Color-based analysis is computationally efficient, and is invariant to the scale or
viewpoint of the face, especially when combined with feature localization (i.e. limited to
regions already defined by feature point tracking). The shape descriptors are first stabilized
against rigid head motion. For that, we imagine that the initial frame in the sequence is a
reference frame attached to the head of the user. On that frame, let (Xp , Yp ) be an
anchor point.
2D projection of the approximated real point around which the head rotates in 3D
space. The anchor point is initially defined as the midpoint between the two mouth corners
when the mouth is at rest, and is at a distance d from the line joining the two inner eye
corners l. In subsequent frames the point is measured at distance d from l, after accounting

for head turns.

Fig 2: Polar distance in determining a lip corner pull and lip pucker
On each frame, the polar distance between each of the two mouth corners and the
anchor point is computed. The average percentage change in polar distance calculated with
respect to an initial frame is used to discern mouth displays. An increase or decrease of 10%
or more, determined empirically, depicts a lip pull or lip pucker respectively (Figure 2). In
addition, depending on the sign of the change we can tell whether the display is in its onset,
apex, offset. The advantages of using polar distances over geometric mouth width and
height

(which is what is used in Tian et al [20]) are support for head motion and

resilience to inaccurate feature point tracking, especially with respect to lower lip points.

Fig 3 : Plot of aperture (red) and teeth (green) in luminance-saturation space

The mouth has two color regions that are of interest: aperture and teeth. The extent of
aperture present inside the mouth depicts whether the mouth is closed, lips parted, or jaw
dropped, while the presence of teeth indicates a mouth stretch. Figure 3 shows a plot of teeth
and aperture samples in luminance-saturation space .Luminance given by the relative
lightness or darkness of the color, acts as a good discriminator for the two types of mouth
regions. A sample of n=125000 pixels was used to learn the probability distribution
functions of aperture and teeth. A lookup table defining the probability of a pixel being
aperture given its luminance is computed for the range of possible luminance values (0% for
black to 100% for white). A similar lookup table is computed for teeth. Online
classification into mouth actions proceeds as follows: For every frame in the sequence, we

compute the luminance value of each pixel in the mouth polygon. The luminance value is
then looked up to determine the probability of the pixel being aperture or teeth. Depending
on empirically determined thresholds the pixel is classified as aperture or teeth or neither.
Finally, the total number of teeth and aperture pixels are used to classify the mouth region
into closed (or lips part), jaw drop, or mouth stretch. Figure 4 shows classification results of
1312 frames into closed, jaw drop and mouth stretch.

Fig 4: Classifying 1312 mouth regions into closed, jaw drop or stretch

COGNITIVE MENTAL STATE INFERENCE

The HMM level outputs likelihood for each of the facial expressions and head
displays .However, on their own, each display is a weak classifier that does not entirely
capture an underlying cognitive mental state. Bayesian networks have successfully been
used as an ensemble of classifiers, where the combined classifier performs much better
than any individual one in the set [15].
In such probabilistic graphical models, hidden states (the cognitive mental states in

10

our case) influence a number of observation nodes, which describe the observed facial and
head displays. In dynamic Bayesian

networks (DBN), temporal dependency across

previous states is also encoded. Training the DBN model entails determining the param- eters
and structure of a DBN model from data. Maximum likelihood estimates is used to learn the
parameters, while sequential backward elimination picks the (locally) optimal network
structure for each mental state model. More details on how the parameters and structure are
learnt can be found in [13].

EXPERIMENTAL EVALUATION
For our experimental evaluation we use the Mind reading dataset (MR) [3]. MR is a
computer-based guide to emotions primarily collected to help individuals diagnosed with
Autism recognize facial expressions of emotion. A total of 117 videos, recorded at 30
fps with durations varying between 5 to 8 seconds, were picked for testing. The videos
conveyed the following cognitive mental states: agreement, concentrating, disagreement,
thinking and un- sure and interested.

There are no restrictions on the head or body

movement of actors in the video. The process of labeling involved a panel of 10 judges who
were asked could this be the emotion name. ? When 8 out of 10 agree, a statistically
significant majority, the video is included in MR. To our knowledge MR is the only
available, labeled

11

Fig 5: ROC curves for head and facial displays

resource with such a rich collection of mental states and emotions, even if they are posed.
We first evaluate the classification rate of the display recognition layer and then the overall
classification ability of the system.
DISPLAY RECOGNITION
We evaluate the classification rate of the display recognition component of the system
on the following 6 displays: 4 head displays (head nod, head shake, tilt display, turn
display) and 2 facial displays (lip pull, lip pucker). The classification results for each of the
displays are shown using the Receiver Operator Characteristic (ROC) curves (Figure 5).
ROC curves depict the relationship between the rate of correct classifications and number of
false positives (FP). The classification rate of display d is computed as the ratio of correct
detections to that of all occurrences of d in the sampled videos. The FP rate for d is given
by the ratio of samples falsely classified as d to that of all non-d occurrences. Table 2
shows the classification rate that the system uses, and the respective FP rate for each display.
A non-neutral initial frame is the main reason behind undetected and falsely detected
displays. To illustrate this, consider a sequence that starts as a lip pucker. If the lip pucker

12

persists (i.e. no change in polar distance) the pucker display will pass undetected. If on the
other hand, the pucker returns to neutral (i.e. increase in polar distance). It will be falsely
classified as a lip pull display. This problem could be solved by using the polar angle and
color analysis to approximate the initial mouth state. The other reason accounting for
misclassified mouth display is that of inconsistent illumination. Possible solutions to dealing
with illumination changes include extending the color-based analysis to account for overall
brightness changes or having different models for each possible lighting condition.
MENTAL STATE RECOGNITION
We then evaluate the overall system by testing the inference of cognitive mental
states, using leave-5-out cross validation. Figure 6 shows the results of the various stages
of the mind reading system for a video portraying the mental state choosing, which belongs
to the mental state group thinking. The mental state with the maximum likelihood over the
entire video (in this case thinking) is taken as the classification of the system.
87.4% of the videos were correctly classified. The recognition rate of a mental
class m is given by the total number of videos of that class whose most likely class (summed
over the entire video) matched the label of the class m.

The false positive rate for class

m (given by the percentage of files misclassified as m) was highest for agreement (5.4%)
and lowest for thinking (0%). Table 2 summarizes the results of recognition and false
positive rates for 6 mental states.
A closer look at the results reveals a number of interesting points. First, onset
frames of a video occasionally portray a different mental state than that of the peak. For
example, the onset of disapproving videos were misclassified as unsure .Although this
incorrectly biased the overall classification to unsure, one could argue that this result is not
entirely incorrect and that the videos do indeed start off with the person being unsure.
Second, subclasses that do not clearly exhibit the class signature are easily
misclassified. For example, the assertive and decided videos in the agreement group were
misclassified as concentrating, as they exhibit no smiles, and only very weak head nods.
Finally, we found that some mental states were closer to each other and could co-occur.
For example, a majority of the unsure files scored high for thinking too.

13

WEB SEARCH

For the first test of the sensors, scientists trained the software program to recognize
six words - including "go", "left" and "right" - and 10 numbers. Participants hooked up to the
sensors silently said the words to themselves and the software correctly picked up the signals
92 per cent of the time.
Then researchers put the letters of the alphabet into a matrix with each column and
row labeled with a single-digit number. In that way, each letter was represented by a unique
pair of number co-ordinates. These were used to silently spell "NASA" into a web search
engine using the program.
"This proved we could browse the web without touching a keyboard.

14

MIND-READING COMPUTERS TURN HEADS AT HIGH-TECH FAIR

Devices allowing people to write letters or play pinball using just the power of their
brains have become a major draw at the world's biggest high-tech fair.
Huge crowds at the CeBIT fair gathered round a man sitting at a pinball table,
wearing a cap covered in electrodes attached to his head, who controlled the flippers with
great proficiency without using hands."He thinks: left-hand or right-hand and the electrodes
monitor the brain waves associated with that thought, send the information to a computer,
which then moves the flippers," said Michael Tangermann, from the Berlin Brain Computer
Interface. But the technology is much more than a fun gadget, it could one day save your life

Scientists are researching ways to monitor motorists' brain waves to improve reaction
times in a crash. In an emergency stop situation, the brain activity kicks in on average around
200 milliseconds before even an alert driver can hit the brake. There is no question of braking
automatically for a driver -- "we would never take away that kind of control,"

15

"However, there are various things the car can do in that crucial time, tighten the seat
belt, for example," he added. Using this brain-wave monitoring technology, a car can also tell
whether the driver is drowsy or not, potentially warning him or her to take a break. At the
g.tec stall, visitors watched a man with a similar "electrode cap" sat in front of a screen with
a large keyboard, with the letters flashing in an ordered sequence.
The user concentrates hard when the chosen letter flashes and the brain waves
stimulated at this exact moment are registered by the computer and the letter appears on the
screen. The technology takes a long time at present -- it took the man around four minutes to
write a five-lettered word -- but researchers hope to speed it up in the near future. Another
device allows users to control robots by brain power. The small box has lights flashing at
different

16

ADVANTAGES AND USES

Mind Controlled Wheelchair
1. This prototype mind-controlled wheelchair developed from the University of Electro
Communications in Japan lets you feel like half Professor X and half Stephen
Hawkingexcept with the theoretical physics skills of the former and the telekinetic
skills of the latter.
2. A little different from the Brain-Computer Typing machine, this thing works by
mapping brain waves when you think about moving left, right, forward or back, and
then assigns that to a wheelchair command of actually moving left, right, forward or
back.
3. The result of this is that you can move the wheelchair solely with the power of your
mind. This device doesn't give you MIND BULLETS (apologies to Tenacious D) but
it does allow people who can't use other wheelchairs get around easier.
4. The sensors have already been used to do simple web searches and may one day help
space-walking astronauts and people who cannot talk. The system could send
commands to rovers on other planets, help injured astronauts control machines, or aid
disabled people.
5. In everyday life, they could even be used to communicate on the sly - people could
use them on crowded buses without being overheard
6. The finding raises issues about the application of such tools for screening suspected
terrorists -- as well as for predicting future dangerousness more generally. We are
closer than ever to the crime-prediction technology of Minority Report.
7. The day when computers will be able to recognize the smallest units in the English
languagethe 40-odd basic sounds (or phonemes) out of which all words or
verbalized thoughts can be constructed. Such skills could be put to many practical
uses. The pilot of a high-speed plane or spacecraft, for instance, could simply order by
thought alone some vital flight information for an all-purpose cockpit display.
17

DISADVANTAGES AND PROBLEMS

Tapping Brains for Future Crimes

1. Researchers from the Max Planck Institute for Human Cognitive and Brain Sciences,
along with scientists from London and Tokyo, asked subjects to secretly decide in
advance whether to add or subtract two numbers they would later are shown. Using
computer algorithms and functional magnetic resonance imaging, or fMRI, the
scientists were able to determine with 70 percent accuracy what the participants'
intentions were, even before they were shown the numbers. The popular press tends to
over-dramatize scientific advances in mind reading. FMRI results have to account for
heart rate, respiration, motion and a number of other factors that might all cause
variance in the signal. Also, individual brains differ, so scientists need to study a
subject's patterns before they can train a computer to identify those patterns or make
predictions.

2. While the details of this particular study are not yet published, the subjects' limited
options of either adding or subtracting the numbers means the computer already had a
50/50 chance of guessing correctly even without fMRI readings. The researchers
indisputably made physiological findings that are significant for future experiments,
but we're still a long way from mind reading.

3. Still, the more we learn about how the brain operates, the more predictable human
beings seem to become. In the Dec. 19, 2006, issue of The Economist, an article
questioned the scientific validity of the notion of free will: Individuals with particular
congenital genetic characteristics are predisposed, if not predestined, to violence.

4. Studies have shown that genes and organic factors like frontal lobe impairments, low
serotonin levels and dopamine receptors are highly correlated with criminal behavior.
Studies of twins show that heredity is a major factor in criminal conduct. While no
one gene may make you a criminal, a mixture of biological factors, exacerbated by
environmental conditions, may well do so.

18

5. Looking at scientific advances like these, legal scholars are beginning to question the
foundational principles of our criminal justice system.

6. For example, University of Florida law professor Christopher Slobogin, who is

visiting at Stanford this year, has set forth a compelling case for putting prevention
before retribution in criminal justice.

7. It's a tempting thought. If there is no such thing as free will, then a system that
punishes transgressive behavior as a matter of moral condemnation does not make a
lot of sense. It's compelling to contemplate a system that manages and reduces the
risk of criminal behavior in the first place.

8. Max Planck Institute, neuroscience and bioscience are not at a point where we can
reliably predict human behavior. To me, that's the most powerful objection to a
preventative justice system -- if we aren't particularly good at predicting future
behavior, we risk criminalizing the innocent.

9. We aren't particularly good at rehabilitation, either, so even if we were sufficiently

accurate in identifying future offenders, we wouldn't really know what to do with
them.

10. Nor is society ready to deal with the ethical and practical problems posed by a system
that classifies and categorizes people based on oxygen flow, genetics and
environmental factors that are correlated as much with poverty as with future
criminality.

11. In time, neuroscience may produce reliable behavior predictions. But until then, we
should take the lessons of science fiction to heart when deciding how to use new
predictive techniques.

12. The preliminary tests may have been successful because of the short lengths of the
words and suggests the test be repeated on many different people to test the sensors
work on everyone.
13. The initial success "doesn't mean it will scale up", he told New Scientist. "Smallvocabulary, isolated word recognition is a quite different problem than conversational
speech, not just in scale but in kind."

19

14. that genes and organic factors like frontal lobe impairments, low serotonin levels and
dopamine receptors are highly correlated with criminal behavior. Studies of twins
show that heredity is a major factor in criminal conduct. While no one gene may make
you a criminal, a mixture of biological factors, exacerbated by environmental
conditions, may well do so.

15. Using computer algorithms and functional magnetic resonance imaging, or fMRI, the
scientists were able to determine with 70 percent accuracy what the participants'
intentions were, even before they were shown the numbers.

CONCLUSION

Tufts University researchers have begun a three-year research project which, if

successful, will allow computers to respond to the brain activity of the computer's user. Users
wear futuristic-looking headbands to shine light on their foreheads, and then perform a series
of increasingly difficult tasks while the device reads what parts of the brain are absorbing the
light. That info is then transferred to the computer, and from there the computer can adjust it's
interface and functions to each individual.

20

One professor used the following example of a real world use: "If it knew which air
traffic controllers were overloaded, the next incoming plane could be assigned to another
controller."
Hence if we get 100% accuracy these computers may find various applications in
many fields of electronics where we have very less time to react.

BIBILOGRAPHY

www.eurescom.de/message/default_Dec2004.asp
blog.marcelotoledo.org/2007/10
www.newscientist.com/article/dn4795-nasa-develops-mindreading-system
http://blogs.vnunet.com/app/trackback/95409

21

22