INTRODUCTION

Sixth sense is a wearable gestural interface that augments the physical world around
us with digital information and let us use natural hand gestures to interact with that
information.

We have evolved over millions of years to sense the world around us.When we
encounter something, someone or some place, we uses our five natural senses to
perceive information about it; that information helps us make decisions and chose
the right action to take. But arguably the most useful information that can help us
make the right decision is not naturally perceivable with our five senses, namely the
data, information and knowledge that mankind has accumulated about everything
and which is increasingly all available online.Although the miniaturization of
computing devices allows us to carry computers in our pockets, keeping us
continually connected to the digital world, there is no link between our digital
devices and our interactions with the physical world.Information is confined
traditionally on the tangible world, and allowing us to interact with this information
via natural hand gestures.Sixth sense frees information from its confines by
seamlessly integrating it with reality, and thus making the entire world your
computer.

1
 Gesture Recognition

Gesture recognition is a topic in computer science and language

technology with the goal of interpreting human gestures via

mathematical algorithms. Gestures can originate from any bodily motion

or state but commonly originate from the face or hand. Current focuses

in the field include emotion recognition from the face and hand gesture

recognition. Many approaches have been made using cameras and

computer vision algorithms to interpret sign language. However, the

identification and recognition of posture, gait, proxemics, and human

behaviors is also the subject of gesture recognition techniques.

Gesture recognition can be seen as a way for computers to begin to

understand human body language, thus building a richer bridge between

machines and humans than primitive text user interfaces or even GUIs

(graphical user interfaces), which still limit the majority of input to keyboard and
mouse.

2
Gesture recognition enables humans to interface with the machine (HMI) and
interact naturally without any mechanical devices. Using the concept of gesture
recognition, it is possible to point a finger at the computer

screen so that the cursor will move accordingly. This could potentially make
conventional input devices such as mouse, keyboards and even touch-screens
redundant.

 GESTURE AND ITS RECOGNITION

Gesture recognition is useful for processing information from humans which is not
conveyed through speech or type. As well, there are various types of gestures which
can be identified by computers.

1. Sign language recognition. Just as speech recognition can transcribe

speech to text, certain types of gesture recognition software can
transcribe the symbols represented through sign language into text.

2. For socially assistive robotics. By using proper sensors

(accelerometers and gyros) worn on the body of a patient and by reading
the values from those sensors, robots can assist in patient rehabilitation.
The best example can be stroke rehabilitation.

3. Directional indication through pointing. Pointing has a very

specific purpose in our society, to reference an object or location based on
its position relative to ourselves. The use of gesture recognition to
determine where a person is pointing is useful for identifying the context
of statements or instructions. This application is of particular interest in
the field of robotics.

4. Control through facial gestures. Controlling a computer through

facial gestures is a useful application of gesture recognition for users who
may not physically be able to use a mouse or keyboard. Eye tracking in
particular may be of use for controlling cursor motion or focusing on
elements of a display.

3
 5. Alternative computer interfaces. Foregoing the traditional
keyboard and mouse setup to interact with a computer, strong gesture
recognition could allow users to accomplish frequent or common tasks
using hand or face gestures to a camera.

6. Immersive game technology. Gestures can be used to control

interactions within video games to try and make the game player's
experience more interactive or immersive.

7. Virtual controllers. For systems where the act of finding or acquiring a

physical controller could require too much time, gestures can be used as
an alternative control mechanism. Controlling secondary devices in a car,
or controlling a television set are examples of such usage.

8. Affective computing. In affective computing, gesture recognition is

used in the process of identifying emotional expression through computer
systems.

9. Remote control. Through the use of gesture recognition, "remote

control with the wave of a hand" of various devices is possible. The signal
must not only indicate the desired response, but also which device to be
controlled.

 INPUT DEVICES
The ability to track a person's movements and determine what gestures they may be
performing can be achieved through various tools. Although there is a large amount
of research done in image/video based gesture recognition, there is some variation
within the tools and environments used between implementations.

1. Depth-aware cameras. Using specialized cameras such as time-of-

flight cameras, one can generate a depth map of what is being seen through
the camera at a short range, and use this data to approximate a 3d
representation of what is being seen. These can be effective for detection of
hand gestures due to their short range capabilities.

2. Stereo cameras. Using two cameras whose relations to one another are
known, a 3d representation can be approximated by the output of the

4
 cameras. To get the cameras' relations, one can use a positioning reference
such as a lexian-stripe or infrared emitters.

3. Controller-based gestures. These controllers act as an extension of

the body so that when gestures are performed, some of their motion can be
conveniently captured by software. Mouse gestures are one such example,
where the motion of the mouse is correlated to a symbol being drawn by a
person's hand, as is the Wii Remote, which can study changes in
acceleration over time to represent gestures.

4. Single camera. A normal camera can be used for gesture recognition

where the resources/environment would not be convenient for other forms
of image-based recognition. Although not necessarily as effective as stereo
or depth aware cameras, using a single camera allows a greater possibility of
accessibility to a wider audience.

 CHALLENGES
There are many challenges associated with the accuracy and usefulness of gesture recognition
software. For image-based gesture recognition there are limitations on the equipment used and
image noise. Images or video may not be under consistent lighting, or in the same location. Items in
the background or distinct features of the users may make recognition more difficult.

The variety of implementations for image-based gesture recognition may also cause issue for
viability of the technology to general usage. For example, an algorithm calibrated for one camera
may not work for a different camera. The amount of background noise also causes tracking and
recognition difficulties, especially when occlusions (partial and full) occur. Furthermore, the distance
from the camera, and the camera's resolution and quality, also cause variations in recognition
accuracy.

In order to capture human gestures by visual sensors, robust computer vision methods are also
required, for example for hand tracking and hand posture recognition or for capturing

5
movements of the head, facial expressions or gaze direction.

COMPONENTS OF THE DEVICE

The Sixth Sense prototype comprises of the followings :-

1. A Pocket Projector
2. A Mirror
3. A Camera
4. A Mobile component(Smart Phone)

6
5. Coloured Markers

TECHNOLOGY BEHIND SIXTH SENSE

It includes the following things :-

1. HARDWARE SETUP

7
 (a) MOBILE
A mobile phone is used rather say smart phone which has multi-tasking
capability. Built-in camera provides execution of both “gesture tracking engine”
and “gesture enabled application”.

(b) PROJECTOR
A LED based projector is used which augments the near by
surface.It is suitable for mobile usage.

2. SOFTWARE SETUP
The software for sixth sense prototype is developed on a Microsoft Windows
platform using C#, WPF and open CV.

The software works on the basis of Computer vision.Computer vision library is

written in symbian C++(used in Gesture tracking).

The software recognizes three kind of gestures namely Multi-touch gesture, Free-
hand gesture and Iconic gesture.

3. EVENT AND GESTURE

 Event and gestures are detected using coloured markers
attached to the fingers.
 The user can zoom in or zoom out by moving his/her
hand/fingers farther or nearer to each other respectively.
 The user can draw on any surface using the movement of
his/her fingers used as a pen.
 System also supports freehand gestures (postures). One
example is to touch both the index fingers with the

8
 opposing thumbs forming a rectangle or framing gesture.
This gesture starts the photo taking application.

Some of the gestures are

9
 HOW IT WORKS???

 The hardware that makes the Sixth Sense work is a pendant like mobile wearable interface.
 It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth smart
phone that can slip comfortably into one’s pocket.
 The camera recognizes the individuals, images, pictures, gestures one makes with their
hands.

10
 Information is sent sent to yhe smartphone for processing.
 The downward facing projector projects the output image on to the mirror.
 Mirror reflects image on to the desired surface.
 Thus, digital information is freed from its confines and placed in the physical world.

APPLICATIONS

The device has a huge number of applications.

 Firstly, it is portable and easily to carry as you can wear it in your neck.
 The drawing application lets user draw on any surface by observing the movement
of index finger.
 Mapping can also be done anywhere with the features of zooming in or zooming
out.
 The camera also helps user to take pictures of the scene he is viewing and later he
can arrange them on any surface. That’s not it.
 Some of the more practical uses are reading a newspaper. Imagine reading a
newspaper and viewing videos instead of the photos in the paper. Or live sports
updates while reading the newspaper.
 The device can also tell you arrival, departure or delay time of your air plane on your
tickets.
 For book lovers it is nothing less than a blessing. Open any book and you will find
the Amazon ratings of the book. To add to it, pick any page and the device gives
additional information on the text, comments and lot more add on features.
 While picking up any good at the grocery store, the user can get to know whether
the product is eco friendly or not.
 To know the time, all one has to do is to just gesture drawing circle on the wrist and
there appears a wrist watch.
 The device serves the purpose of a computer plus saves time spent on searching
information.

Technology has a role in society of making life better, and while it does this it’s still an
inconvenience to use. A whole set structure of systems and languages need to be adopted
for someone to begin to use the technology. Technology is supposed to link the user with

11
 the required result as efficiently as possible, but we can never escape the fact that
technology is always going to be a third wheel in an otherwise personal experience.

Below are in my opinion 5 of the most exciting possibilities

1. Photographs Become Personal

It goes without saying because of just how natural it is to make the framing hand gesture to
signify a view. It seems as though we were always meant to take photos this way, we just had
to go through a process of holding a box of plastic and metal in the area first. One of the most
difficult things about taking photos is the de-personalisation of the photographer. Fair enough
they’re not hidden under a black hood like in the old days, but they’re still completely on the
other side of the camera. For such a personal moment, it’s always an inconvenience to have
an inanimate object like a camera dictate where you sit, how you smile and inevitably be the
object which records you. But by using your hands to take the photo it’s now the user who is
responsible for recording you, in the same way that being sketched by a friend is much more
comfortable than having a stranger analyse the contours of your face.

2 . Sharing Your Digital World

As much as social media has done for connecting the world, it’s still a very remote
experience. It’s ironic that such a social activity is usually done in isolation, with retweeting
or tagging taking precedence over calling someone over to your screen. Sixth Sense
Technology would allow social media to become social in the offline environment too –
photos could be shared naturally and intuitively rather than in a method more akin to long
distance mail. Remember crowding around a photo album, laughing and smiling with friends
and families as you chat about the pictures? It’s a distant memory to the frequently inane
comments we find left on our Facebook photos that we’re beginning to get used to today.

3 . Overlay of the World

The advantages of performing tasks in a digital manner  have been great enough to justify
sacrificing some of the natural benefits we have been used to. Instead of reading the free
paper on the bus, I’ll check my RSS feeds for up-to-date and filtered content, but it’s just not
as natural a way to interact with the written word. Also the enjoyment of purchasing a new
item is lost when done so remotely online, but the added benefit of product research is
enough for us to go without the full thrill of the purchase. But maybe the uses that Mistry and
his team are investigating may be able to give something back to the offline world. Could the
web’s murderous relationship with printed media and high street outlets actually just be a
phase?

4 .Dispensable Computers

It’s a shame that technology is always so expensive, because it holds it back from really
being integrated into the user’s life. How many pens do you own that you actually paid for?
There’s an unspoken system of pen exchanges connecting the entire world which means you
may be using a Biro which has had many, many other owners in the past. Imagine if iPads

12
were as frequently passed around – you would maybe find one sitting on the bus or ask to
borrow one and never give it back. My point is that the more expensive the technology and
the more protective you are over it, the more it becomes a block to the task at hand. You
should be playing a game, booking tickets or listening to music – not using an iPad. If you
can take away your relationship with technology, then you can begin to get on with the real
aspects of your life.

5 .Merging Hardcopies with Digital

Now this the real clincher, and something incredibly beautiful I think. In the past 50 years so
many of us have been forced to adopt a very respectful relationship with computers –
speaking in their language, following their rules and generally doing things there way. While
it certainly speeds up some processes, as soon as we’re away from a computer we
immediately go back to how we intuitively work, and the systems exist entirely separate from
one another. The intuitive way that humans work is the most efficient and enjoyable, but
we’re being squeezed into a way which suits the computer. As much as GUI and usability
specialists are trying to imitate the experience, we still have a long way to go before we can
say we’ve really reclaimed dominance over the computer and made the computer work for
us. Sixth Sense Technology is a tantalising tease that we might be about to make a significant
step closer towards this moment of singularity.

13
 ADVANTAGES OF SIXTH SENSE

Following are the advantages of sixth sense technology:-

1. It is portable.
2. It supports multi-touch and multi-user interaction.
3. It is very much cost effective.
4. Connectedness between the world and information.
5. Data access directlyfrom machine in real time.
6. Mind map the idea anywhere.
7. It is an open source.

DISADVANTAGES OF SIXTH SENSE

It has following demerits:-

14
1. The device hangs around the users neck. Future versions could be
made compact.
2. Personal documents can’t be opened in public.
3. Needs high Band width.
4. Applicable only in Smart phones and 3G technology.
5. For operating this device lots of physical activities involved.
6. Any surface which is empty can only be used.
7. Little bit of training is required to operate.

CONCLUSION

15
16
 REFERENCES

 P. Mistry. The thrilling potential of SixthSense technology. TEDIndia 2009.

Mysore, India 2009
 P. Mistry, P. Maes, L. Chang. WUW - Wear Ur World - A Wearable Gestural
Interface. In the CHI '09 extended abstracts on Human factors in computing
systems. Boston, USA. 2009
 P. Maes, P. Mistry. Unveiling the "Sixth Sense," game-changing wearable tech.
TED 2009. Long Beach, CA, USA 2009
 P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the
Proceedings of SIGGRAPH Asia 2009, Sketch. Yokohama, Japan. 2009
 www.pranavmistry.com/projects/sixthsense
 en.wikipedia.org/wiki/sixthsense

17