Human Activity Recognition based

Home Automation System

Sushant Kumar Singh
Department of Electrical Engineering, Delhi
Technological University
Delhi, India

Abstract Home automation allows us to control household

appliances like light, door, fan, AC etc. Home automation not
only provides us convenience but also saves energy and time.
This paper presents a design and prototype implementation of a
new home automation system that uses computer vision and
embedded microcontrollers to accomplish two tasks; home
automation and monitoring. The primary objective of the
prototype is to implement control that requires minimum user
intervention. The system will recognize the activity being
performed by the user and turn on/off the electrical appliances to
create an optimal ambience suitable for that particular activity.
This context aware system enables us to extend the capability of
home automation. A software application is also provided to
monitor and control the home.
Keywordshome automation; computer vision; embedded
processor; visual interface; context awareness

I. INTRODUCTION
A. Current State of Home Automation.
The dream of a fully automated home has existed for a long
time. Despite the interest of media and researchers and the
widespread availability of motion sensors, programmable
lighting and mobile applications, home automation itself has
not been widely adopted.
B. Types of Home Automation
Home automation has been dived into two categories, user
controlled and rule based [1]. In user controlled automation the
household member explicitly takes a single action which
causes several things to happen. This requires constant inputs
from the user even for actions that could have been actuated
reliably based on situation based on contextual awareness. For
example, a user turns off all the appliances while leaving the
house. This could have been automated using motion sensors
and computer vision. Whereas in user controlled home
automation he presses the All Off lighting button.
In rule-based automation, rules trigger automation based,
rules trigger automation based on events or at certain times.
These rules are configured by the professional installer. They
definitely represented an additional level of automation
inflexibility`.

C. Current Challenges of Home automation

While the majority of households who installed home
automation in partial or full capacity had positive experiences,
there are barriers that need to be overcome before the general
population considers adopting home automation. They include
poor manageability, complex user interfaces, inflexibility and
lack of centralized control [1].
D. Need for new paradigm for broad adoption.
The people who have adopted the home automation are
mainly classified as Do-it-yourselfers (DIY). They have
installed the automation themselves. But the majority of the
households will outsource the installation to professionals.
Therefore, it is vital that the system is self-reliant on making
control decisions right from installation. Smart appliances
today ask the user to provide extensive preference settings or
need a certain period of time to learn users activity pattern.
Since these devices do not perform optimally in the initial
interaction it leads to dissatisfaction or discomfort.
E. Advantages of context aware systems
Context Aware systems cam mitigate most of the
abovementioned challenges.
The systems can be preprogrammed to recognize
commonly performed human activities like work,
sleep, socializing, dining etc. and therefore work out
of the box efficiently.
Even when additional users are added or the system
is expanded, the human activates remain the same
thus making the system scalable.
Context Aware systems can take home automation to the
next level
F. Low Cost
Traditionally, Human activity is recorded using sensors or
makers on body. This methodology is complex and expensive.
The prototype uses hand and face detection using image
processing to reduce costs. Also, hand and face detection has
been implemented with high accuracy in consumer devices
like webcams, Kinect etc.

II. RECOGNIZING HUMAN PRESENCE AND ACITVITY STATUS

A. Face Detecion Methodolgy
Face Detection is implemented using OpenCVs Haar
feature based cascade classifier It is a machine learning
based approach where a cascade function is trained from a
lot of positive and negative images. It is then used to detect
objects in other images. This approach to detecting objects
in images combines four key concepts

Simple rectangular features, called Haar features

An Integral Image for rapid feature detection

The AdaBoost machine-learning method

A cascaded classifier to combine many features

efficiently

The features that Viola and Jones used are based on Haar
wavelets[2]. Haar wavelets are single wavelength square
waves (one high interval and one low interval). In two
dimensions, a square wave is a pair of adjacent rectangles one light and one dark.
The actual rectangle combinations used for visual object
detection are not true Haar wavlets. Instead, they contain
rectangle combinations better suited to visual recognition
tasks. Because of that difference, these features are called
Haar features, or Haar like features, rather than Haar
wavelets.

In general, "integrating" means adding small units together. In

this case, the small units are pixel values. The integral value
for each pixel is the sum of all the pixels above it and to its
left. Starting at the top left and traversing to the right and
down, the entire image can be integrated with a few integer
operations per pixel.
AdaBoost selects a set of weak classifiers to combine and
assigns a weight to each. This weighted combination is the
strong classifier.
B. Experimanetal Work
We will now utilize OpenCVs buit in .xml classifier
file haarcascade_frontalface_alt.xml for face detection.
When the users face is detected, a set of events can be triggered
automatically. For instance, when user enters a room, the
computer detects a face and sends command to the
microcontroller to turn on the lights as shown in Figure 2.
It is important to note that we are using Face detection and
not Face recognition. Face recognition requires much more
complex algorithms as it tries to exactly match the users
features. As a consequence it requires more processing power.
Face detection has been proven to be simple and efficient with
high accuracy. It has also been successfully implemented in
low power embedded microcontroller devices such as webcam.
Face detection can be extrapolated to face recognition such
that only certain group of users can access the house, providing

The presence of a Haar feature is determined by

subtracting the average dark-region pixel value from the
average light-region pixel value. If the difference is above a
threshold (set during learning), that feature is said to be
present.
To determine the presence or absence of hundreds of
Haar features at every image location and at several scales
efficiently, Viola and Jones used a technique called an
Integral Image.

extra level of security.

Fig. 2. The Users face is detected and the light bulb is turned on

Fig. 1. The two features are shown in the top row and then overlayed on a
typical training face in the bottom row. The first feature measures the
difference in intensity between the region of the eyes and a region across the
upper cheeks. The feature capitalizes on the observation that the eye region is
often darker than the cheeks. The second feature compares the intensities in
the eye regions to the intensity across the bridge of the nose.

Another application of face detection is that we can detect the

number of faces thus determine the number of humans in a
given room .In Figure 3 when two faces i.e two users are
detected the two light bulbs are turned on. The light bulbs can
be replaced by any electrical appliance for example an electric
fan.

A. Background Construction
The first step towards accurate detection is to determine the
background such that only the hand is extracted. Given the feed
from the camera We use running average over a sequence of
images to get the average image which will be the background
So as and when we keep receiving image, we construct the
new background average image as
CurBG[i][j] = CurBG[i][j] + (1- )CurFrame[i][j]
Fig. 3. When two users are detected two light bulbs are turned on to
increase illumination

C. Eye Detection for actitvity status

Eye detection uses "haarcascade_eye_tree_eyeglasses.xml"
to detect eyes even with spectacles. Eyes provides us
information about whether the user is awake or sleeping as
shown in Figure 4. The OpenCv

Fig. 3. In comparison to Figure 2 the users face is detected

but the eyes are not detected. Therefore, the user must be
resting or sleeping and the light bulb is turned off.

III. RECONGIZIN HUMAN ACTION USING HAND TRACKING

The primary purpose of home automation is to provide
convenience. True automation can only be achieved if the
home automation system can adapt according to the activity
being performed by a user at a given time most home
automation systems can detect human presence but cannot
interpret what the user is doing.
Human use their hands for most activities. If we could
detect the hand orientation and the object being held, we can
provide an accurate guess on the activity being performed.

(1)

This equation works because of the assumption that the

background is mostly static. Hence for those stationary item,
those pixels arent affected by this weighted averaging and
(2)
Hence those pixels that are constantly changing isnt a part
of the background, hence those pixels will get weighed down.
The stationary pixels or the background gets more and more
prominent with every iteration while those moving gets
weighed out. Thus after a few iterations, we get the above
average which contains only the background. Even the face is
part of the background therefore it detects only the hands [3].
B. Background Substraction
A simple method to start with is we can subtract the pixel
values. However this will result in negative values and values
greater than 255, which is the maximum value used to store an
integer. Instead we use an inbuilt background subtractor based
on a Gaussian Mixture-based Background/Foreground
Segmentation Algorithm. Background subtraction involves
calculating a reference image, subtracting each new frame from
this image and thresholding the result which results is a binary
segmentation of the image which highlights regions of nonstationary objects . This reference image is the background
image constructed[3].
C. Contour Extraction
Contour extraction is performed using OpenCVs inbuilt
edge extraction function. It uses a canny filter to get a list of
contours. It works by convoluting a filter with the image so
that gradually changing components gets cancelled out while
sudden changing components like at the edge or borders are
enhance. Thus the edges become visible. It does the following
steps to extract the edges.
Filter out any noise. The Gaussian filter is used for this
purpose.

Apply a pair of convolution masks (in x and y

directions)

Find the gradient strength and direction. The direction

is rounded to one of four possible angles (namely 0,
45, 90 or 135).

Non-maximum suppression is applied. This removes

pixels that are not considered to be part of an edge.
Hence, only thin lines (candidate edges) will remain.

We are now going to detail the algorithm for hand detection.

Canny does use two thresholds (upper and lower ) :

I.

If a pixel gradient is higher than the

upper threshold, the pixel is accepted as
an edge

II.

If a pixel gradient value is below the

lower threshold, then it is rejected.

III.

If the pixel gradient is between the two

thresholds, then it will be accepted only
if it is connected to a pixel that is above
the upper threshold.

This gives us a list of set of points, each set representing a

contour. We can filter out small contours as they will be noise.
So we set a threshold for the area for the contour about which
we consider for the following steps[4].

F. Experimental Work
We applied simple background elimination techniques
and set of filters to get the hands contour and the number
of finger. After the determination of hand contour , we can
detect the activity being performed.
We will now discuss the implementation of this idea on
one commonly performed activity i.e. reading. A book or
even an electronic device is usually held in hand and the
hand position is adjusted for appropriate eye position. The
fingers are spread apart on the back of the book to give
adequate support.
The advantage of the contour detection algorithm is that
once the background has been suppressed, the book and
the hand form a closed concave structure which is detected
as a contour as shown in figure 5 and figure 6.

D. Convex Hull and Defects

Now given the set of points for the contour, we find the
smallest area convex hull that covers the contours .The
Sklanskys algorithm was used to find the convex hull which
has a complexity of O(nlogn). The observation here is that the
convex hull points are most likely to be on the fingers as they
are the extremeties and hence this fact can be used to detect no
of fingers[5]. But since our entire arm is there, there will be
other points of convexity too. So we find the convex defects ie,
between each arm of the hull, we try to find the deepest point
of deviation on the contour.
E. Tracking and Finger Detection
For each maxima defect point which will be the finger
tip, there will be 2 minimal defect points to indicate the
valleys. Hence the maxima and the 2 minimal defects should
form a triangle with the distance between the maxima and the
minimas to be more or less same. we get the list of maximal
defect points that satisfy the above conditions and thus we find
the no of fingers using this. If no of fingers is 0 , it means the
user is showing a fist.

Fig. 4. Hand Contour is detected The green line bounding the hand
is convex hull .Basically its a convex set ; means if we take any two
points inside the red region and join them to form a line then the line
entirely lies inside the set

Fig. 5. The green line bounding the hand and the book is convex hull
.Convex hulls basically means if we take any two points inside the
green region and join them to form a line then the line entirely lies
inside the set.

Fig. 6. Once the contour is detected the two lights bulbs are
turned on to reduce eye strain.

Illumination of the room is increased to reduce eye strain.

If a fist is detected then user is holding some tool and other
appropriate actions can be initiated. In most cases tool
work also requires increase of illumination.
IV. SOFTWARE APPLICATION FOR CENTRALIZED CONTROL
A Windows Software Application has been developed
which not only controls the various fans and lighting
equipment but also displays the total energy saved with use of
a home automation system. The application calculates energy
that would have been consumed if appliances would have
been left ON in the absence of a user. For example, when a
user leaves the room, the system turns off the lights after a set
delay. If the system would not have been present the lights
would have been kept on until the user returned. The energy
that is consumed, if the lighting would have been kept ON, is
calculated using basic power equations
P = V*I*T1-T2
Where
P is the total power consumed in KWh
V is the voltage across the appliance
I is the current across the device
T1 is the time when user left the room
T2 is the time when user returned
We can use the rated voltage and current values eliminating
the need for complex current and voltage measurement.
Total financial savings can then be calculated by multiplying
the energy saving with cost per unit.

V. EXPERIMENTAL SETUP
The prototype system consists of a PC running Microsoft
Visual Studio 2013 running OpenCV library, a webcam for
feature detection and an ARM 7 based microcontroller LPC
2148. The feed from the camera is processed and an command
is generated which is send to the microcontroller. The
Microcontroller controls switching relays. Various electrical
appliances are connected to these relay. OpenCV can be
installed on the ARM board itself by running embedded linux
to eliminate the need for PC and reduce costs.

Fig. 9. Picture of Implemented Prototype.

A. System Architecutre and Process Exectuion

We have discussed the algorithms used for feature
detection and the appropriate action to be executed. The block
diagram in figure 10 details the procedure for process
execution from monitoring to execution. Figure 11 illustrates
the flowchart for the algorithm.
PC with
Video
Receiver

LPC 2148
Max
232

Fig. 7. User interface for control and monitoring

U
A
R
T

GPIO

Relay

Bulb

Fig. 10. Block Diagram of Implemented Prototype

Fig. 8. Cost of electricity Worldwide (2011)

Bulb

Video Feed from

Camera

Face
detected?

No

Turn Lights off

if previously
on

Yes

Number
of faces

>1

Increase
Illumination

B. Microcontroller LPC 2148

This study adopts ARM7 LPC2148, which is a cost
effective and high performance and very low power
consumption 32-bit microcontroller for the development of
system. The ARM architecture is based on Reduced Instruction
Set Computer (RISC) principles, the users can take the
advantages of the rich and open programming platform
resources, to develop different applications. It is based on a 32bit ARM7TDMI-S CPU with real-time emulation and
embedded trace support that combine the microcontroller with
32 kB, and 512 kB of embedded high-speed flash memory.
A128-bit wide memory interface and unique accelerator
architecture enable 32-bit code execution at maximum clock
rate. Due to their tiny size and low power consumption, these
microcontrollers are ideal for applications. Key features for
LPC2148[6]
1.

32-bit ARM7TDMI-S microcontroller a tiny LQFP64

or HVQFN64 package.

2.

32 kB of on-chip static RAM and 512 kB of on-chip flash

program memory.
Low power Real-time clock with independent power and
dedicated 32 kHz clock input.
Up to forty-seven 5 V tolerant general purpose I/O pins in
tiny LQFP64 package.
CPU operating voltage range of 3.0 V to 3.6 V (3.3 V 10
%) with 5 V tolerant I/O pads.
On-chip integrated oscillator operates with external crystal
in range of 1 MHz to 30 MHz and with external oscillator
up to 50 MHz
Two 32-bit timers/external event counters (with four
capture and four compare channels each), PWM unit (six
outputs) and watchdog.
Single 10-bit DAC provides variable analog output
(LPC2132/34/36/38).

3.
Turn Lights off
if on previously

=1

4.
5.

.
No

Eyes
detected?

6.
7.

Yes
Lights are turned on if
off previously

No

Hand
Contour
Detected?

Yes
Increase Illumination

Fig. 11. Flow of Control

8.

C. SPDT Relay
A relay is an electrically operated switch. Many relays use an
electromagnet to mechanically operate a switch, but other
operating principles are also used, such as solid-state relays.
Relays are used where it is necessary to control a circuit by a
low-power signal (with complete electrical isolation between
control and controlled circuits)
Since relays are switches, the terminology applied to
switches is also applied to relays; a relay switches one or more
poles, each of whose contacts can be thrown by energizing the
coil in one of three ways:
Normally-open (NO) contacts connect the circuit when the
relay is activated; the circuit is disconnected when the relay is
inactive. It is also called a Form A contact or "make" contact.
NO contacts may also be distinguished as "early-make" or
NOEM, which means that the contacts close before the button
or switch is fully engaged.
Normally-closed (NC) contacts disconnect the circuit when
the relay is activated; the circuit is connected when the relay is
inactive. It is also called a Form B contact or "break" contact.

NC contacts may also be distinguished as "late-break" or

NCLB, which means that the contacts stay closed until the
SPDT Single Pole Double Throw. A common terminal
connects to either of two others. Including two for the coil,
such a relay has five terminals in total.

[6]

D.Naresh1, B.Chakradhar2, S.Krishnaveni3, Bluetooth Based Home

Automation and Security System Using ARM9. International Journal
of Engineering Trends and Technology (IJETT) Volume 4 Issue 9Sep 2013.

VI. RESULTS AND CONCLUSION

The prototype system controls two light bulbs according to the
user action. We can connect up to two 4-channel relays that
means up to 8 electrical appliances can be connected to just
one LPC 2148.
TABLE I.

S.no

SYSTEM RESPONSE TO DIFFERENT ACTIVITIES

Activity

1.

User enters the room

Another User enters the room

User is Working/Reading

4.

User is sleeping

5.

User leaves the room

Feature
Detected

Face is
detected ,
number of
user also
detected
Two
Faces are
detected
Hand
contour is
detected
Eyes are
not
detected
No face
detected

Action
Taken

One bulb is
turned on

Two bulbs
are turned
on
Illumination
is increased
.Two light
bulbs are
turned on
Lights are
turned off
Lights are
turned off

The system performed reliably turning light bulbs on/off in

line with expected behavior. The prototype is being improved
by addition of parameters such as temperature, past activity etc.
to further improve accuracy
ACKNOWLEDGMENT (Heading 5)
We wish to thank Delhi Technological university for their
support and guidance.
REFERENCES
[1]

A.J. Bernheim Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal,

Stefan Saroiu, Colin Dixon*, Home Automation in the Wild:
Challenges and Opportunities,CHI 2011.

[2]

[3] P. KaewTraKulPong and R. Bowden, An Improved Adaptive

Background
Mixture Model for Realtime Tracking with Shadow Detection,d
European Workshop on Advanced Video Based Surveillance Systems,
AVBS01.Sept 2001.
Aniket
Tatipamula,
Hand
Guesture
using
OpenCV,
http://anikettatipamula.blogspot.in/2012/02/hand-gestureusingopencv.html
Lakshminarayanan, Hand Tracking And Guesture Recognition
Paul Viola, Michael Jones, Rapid Object Detection using a Boosted Cascade of Simple

[3]

[4]

[5]
[2]