Escolar Documentos
Profissional Documentos
Cultura Documentos
CST 499
Joe Sarabia
Executive Summary
This project seeks to enhance the usability of home automation by correlating user location and
user activity within a home to build behavioral models of individual users which can then be
applied to automatically apply state based on detection of a user and other environmental factors.
This project affects all home automation users. The expected outcome is a functional prototype
demonstrating how machine learning and home automation can combined to achieve enhanced
usability.
3
Executive Summary 2
Overview 4
Project Description 4
Phase I: Research-based 5
Phase II: Project-based 5
What This Project Will Do 6
Who Will Benefit and How 6
Why Is It Important 6
Background Information 9
Literature Review 10
Paper 1: Bluetooth Low Energy for Smart Home Energy Management 10
Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation 11
Methodology 13
Milestone 1: Market Analysis and Feasibility Assessment 13
Milestone 2: Procure Equipment 14
Milestone 3: Prototype Implementation 14
Milestone 4: Prepare Demonstration 15
Ethical Considerations 15
Ethical Concerns 15
Negative Impacts 17
Environmental Impacts 18
References 19
Appendix 20
4
Overview
initially advised by Dr. Tao. The first part of the the project will be research-based and will
incorporate a survey of the industry to hone in on a specific area of focus for the second part of
the capstone. This second part will be project-based with a focus on implementing a prototype as
Project Description
While substantial progress has been made towards delivering on the promise of the smart
home of tomorrow, even the modest smart home capabilities of today generally require a
thereby excluding vast portions of this technologys potential user base. This research intends to
investigate the extent to which todays technology can be used to create a learning home by
implementing an artificial intelligence (AI) which is able to learn the behavior of its occupants
and derive the desired state of the home at any given time based upon a combination of
parameters such as time of day, biometrics, weather conditions and the specific location of a
particular occupant or combination of occupants within the home. This state engine should
ultimately be able to control common Internet of Things (IoT) devices such as lighting, music
speakers, electrical outlets, appliances and televisions. The research intends to create a functional
prototype of this state engine demonstrating control of a subset of these devices based on a
Phase I: Research-based
home? This research-based portion of the capstone project will seek to answer whether particular
hardware such as IoT devices and presence detection devices can be used to accurately and
precisely track the location of specific individuals within a home. Furthermore, this research will
seek to answer whether a software implementation can be created to detect and track the change
of state of IoT devices within the home and correlate that to the location of a specific individual
within the home. Finally, this research will seek to answer whether an AI can be built to learn the
behaviors of a homes occupants and automatically apply desired state based upon applicable
parameters such as time of day, weather conditions and other factors which influence a particular
behavior.
How well can todays existing technology be used to create an early, functional prototype
of a learning home? Based upon the findings from the research-based portion, the project-based
portion of this capstone will seek to implement a prototype which is capable of precisely tracking
indoor location of occupants within a home, observing state changes of IoT devices within that
home and then correlating these data together to produce a recommendation engine for desired
state of a home based upon the location of a particular occupant and at least one other desirable
demonstrates the ideas embodied within this research as outline above. Ultimately, it should
enable a home to exhibit the capability to accurately respond to particular combinations of inputs
based upon previous learned behaviors. The idea is to facilitate the transition from quasi-smart
Potentially any current or future user of home automation based technologies will benefit
from this research. In particular, classes of users who are otherwise excluded from or
substantially disadvantaged in leveraging home automation and its capabilities today would
benefit.
Why Is It Important
ubiquitous and its capabilities are increasingly impressive. There is frequent conversation in the
national dialogue today amongst policymakers and others about the role of automation and
artificial intelligence in our future economy. On one level, this research is important to
demonstrate the power of artificial intelligence in a tangible way that is readily accessible and to
close the gap between the sci-fi based depictions of a smart home, such as those depicted in Back
to the Future and The Jetsons with the real capabilities of todays technology. On another level,
7
the realization of these capabilities opens up more possibilities for societal and individual benefit
The three most important criteria to demonstrate this research are that a change of state
occurs based upon a combination of the following factors: a particular users location within a
home, some other pre-condition such as time of day, and that it exhibits control over more than
Phase I: Research-based
During the research based portion of the this capstone, the objective will be to survey the
industry and identify suitable technologies and their capabilities to fulfill the above use cases.
For purposes of precise location tracking, the objective will be to identify 1-2 technologies which
are able to determine specific location within a room, for example detecting an occupant in the
living room may not be sufficient, the technology should be able to detect the difference between
sitting on the couch in the living room and just walking through it. A requirement for these
technologies is that they have a public Application Programming Interface (API) which can be
used for integration in the broader solution. For purposes of detecting behaviors and ultimately
portion of the research will be to identify 2-3 IoT devices with APIs that can be used to read and
There are two sample use cases that I intend to explore as possible early prototypes based
upon findings in the research-based portion of the project. The ultimate goal with these would be
Objective 1: Follow Me. The objective with this use case will be to demonstrate the
feasibility and a prototype of some desired state that follows an occupants traversal through
various parts of a home. Similar to how light sensors work, this can be thought of a sensor with
persona, that is, the sensors know the difference between multiple occupants such that they only
activate for particular occupants under particular circumstances. One interesting use case to
demonstrate here is follow-me audio, where a multi-speaker, multi-room audio system is able to
continuously adapt its state based upon an occupants location. An example of this could be a
playlist that starts playing on a speaker when an occupant walks into the bathroom to get ready
for the day, then switches from the bathroom speaker to the bedroom speaker while the
occupants finishes getting ready and then switches to the living room speaker when the occupant
moves to couch to prepare a to-do list for the day. Its not necessary and indeed inefficient for
speakers to be playing in a room the occupant doesnt occupy, and this use case would
demonstrate the ability for a home to apply a state, in this case a playlist, and have it follow that
some combination or sequence of states across multiple IoT devices or systems applied by a
particular occupant in a discernible and predictable manner. The objective of this use case will be
9
to demonstrate the ability to learn and appropriately apply such a behavior. An example use case
of this is a breakfast making routine where every weekday morning between 8:15a and 8:45a an
occupant enters the kitchen, turns on two specific lights, requests to hear the news from a home
assistance device, and then plays a playlist consisting of 3-4 songs while making and eating
breakfast. Before leaving the kitchen, the occupant turns off the playlist and turns off the lights
while heading out the door for work. The objective of this use case would be to learn this
behavior and then appropriately apply it if the occupant is present during the learned timelines.
Importantly, this use case should also demonstrate that state changes dont occur for that same
occupant in that location at times previously unobserved, that is, it shouldnt apply the breakfast
behavior during dinner, and it should only apply the behavior for intended occupants and state
should be unaffected by the presence of some other occupant. For example a second occupant
passing through the kitchen during the same time should have no impact on the state of the
home.
Background Information
smart home technologies in my home over the last 3.5 years. Initially, my attempts at
implementing home automation yielded a very low wife acceptance factor, and I realized that not
everyone would find it appealing to control their home from their phone, or even have a phone
from which they could control their home. Ive observed numerous deficiencies in existing
products such as geofences that are orders of magnitude larger than my home and which dont
intelligently account for the presence of multiple people within a home. More than once, I might
leave to go to the store and all the lights in the house as drove away while my wife was sitting on
10
the couch watching television. This is where the initial ideas behind occupant detection came
into play.
between multiple apps on my phone to yield a desired state in my home could be improved upon.
I realized that I was typically taking the same set of actions from the same location within the
home, and that the this sequence of steps along with precise occupant detection should be
sufficient to create a home that can learn and apply my desired behaviors and truly be called a
smart home.
Literature Review
A brief review of existing research yielded two results which have some similar concepts
yet not entirely overlapping concepts and proposals. The two papers reviewed are covered
below.
appliances communicate to an energy management unit (EMU) and some intermediary system
between the appliances and the user of the appliance is able to make a recommendations about
the use of the appliance when the user attempts to use it, generally in an effort to reduce peak
load demand and consumptions charges (Collotta & Pau, 2015). For example, the EMU might
suggest that the electric clothes dryer be used at a later time when the usage fee for electricity is
lower. While this does involve a use case where home automation interacts with the user which
is something The Learning Home proposes, in the case it occurs through an intermediary system
11
and is not completed automated based on behavior. The Learning Home proposes that similar
activity with appliances be applied by presence detection. Nonetheless, there could certainly be
use cases where both applications might exist, or where the data collection by the EMU provides
an input into The Learning Homes state engine as a parameter to consider in determining
This paper proposes using Bluetooth Low Energy (BLE) to coordinate communication
between home appliances as a mechanism for home energy management (HEM). The choice of
wireless protocol is critical according to this paper. BLE connection setup and data transfer occur
in a range from 3ms to 1.28s. Tens or even hundreds of milliseconds to transfer data would likely
be sufficient to satisfy the perception of near real-time reaction in the cases proposed by The
Learning Home. The authors point out the substantial BLE penetration in the industry, for
example, its already technology included in commonly owned devices such as smart phones,
smart watches and tables. They also point out that reliability is key to achieve accurate
automation functionality. The paper asserts that BLE is the best choice in applications of home
automation.
This research by S. Shaikh, Attar, Pathan & R. Shaik (2016) sounds similar in that their
goal is to use Machine Learning (ML) to generate personalized and time-variant home
automation plans. It posits a solution to the same problem that The Learning Home seeks to
improve upon, namely the rigidity of usability. This research also proposes a system that will
While these concepts have some overlap with concepts that The Learning Home also
seeks to investigate, this paper is two pages long and is primarily conceptual in its nature with no
reference to the user differentiation detection capabilities or precise real-time location services
As previously mentioned, the stakeholders and community for this project are any current
or future home automation user. What this group of individuals stands to gain is an improved
realization of home automation benefits, many of which could be construed as comfort level
benefits. Nonetheless, home automation promises advances in many aspects of ones home
including security and energy efficiency. One common IoT device with interesting capabilities is
the smart camera which can be used to detect activity within zones, and more recently, human
presence in a specified area. This activity detection can then be used to trigger desired home
automation actions. While useful, today these are not personalized. For example, the cameras can
detect a person but not a specific person. This paper asserts that the ability to detect particular
people in particular spaces coupled with behavioral modeling will add greater persona and
reduced rigidity to the usability of home automation. On the other end of the spectrum, another
consideration for the home automation user community is one of privacy. There are some
prospective users of home automation who may have no interest in having cameras storing
images of their activity in some external storage service, or there might be objections to the use
of cameras in particular parts of the home, such as bedrooms and bathrooms. The lack of
cameras in these areas otherwise reduces home automation capabilities which might otherwise be
13
achieved. This paper seeks to overcome such privacy objections by researching feasible options
to trigger actions based on the presence of a particular person in a particular position within a
space without a camera while also providing greater location detection precision.
One particular area of interest that this paper seeks to investigate is to what extent the
techniques discussed herein might benefit disabled people using home automation. As an
example, great strides have been made in speech based home assistant type devices, but an
individual with a speech impediment may not benefit from some of the capabilities such devices
Methodology
The methodology I intend to follow in completing this project will be Agile, using
iterative and incremental development techniques and fast feedback loops to adjust as the
projects goes on. I intend to use real-world consumer devices combined with software to
implement prototypes of the topics discussed herein. I have identified four major milestones as
In order to meet this milestone, I intend to research various devices currently available in
the consumer marketplace. Specifically, I intend to research smart cameras which are suitable
candidates for real-time facial detection and devices which can be used to precisely track
real-time location of a user within a home. As part of the intended solution seeks to correlate the
14
activity of devices with the location of users in order to build behavioral models which could
later apply state, I further intend to research home automation smart hubs which aggregate the
control of multiple devices where some such model can be applied. One important part of
research that I intend to complete in this milestone is the assessment of APIs for all devices that I
research, as that will be important for later stages of project. Based on the results of this
assessment, I will make any required adjustments to scope which could be necessary based on
my findings.
To reach this milestone, I intend to choose some combination of devices based upon the
completed analysis. The intent here is to use this equipment to build prototypes of the topics
discussed in this paper. If the analysis turns up items which I already own, then that could
I intend to use the equipment previously procured to build a functional prototype of the
concepts discussed in this paper. Specifically, I intend to demonstrate the ability to correlate user
activity with user location to build a behavioral model. I further intend to demonstrate both a
camera based and non-camera based approach. Lastly, I intend to demonstrate how a learned
behavior can be applied to trigger some home automation activity. In order to reach this
milestone, I will need to conduct extensive research in the area of machine learning as its not an
area which I have previous exposure to. I will use iterative and incremental development during
15
this phase and will use tools such as Pivotal Tracker and GitHub to track my work. I expect that
adjustments to the scope of the project are likely to occur in this phase.
Most of the testing will be conducted in my home and I would like to find a way to
present this on a smaller scale on-campus, if possible. To that extent, during this portion of the
project I intend to experiment with a scaled down version or other mechanisms to effectively
Ethical Considerations
Given that this research deals with tracking user activity and location very precisely
within ones home, there are important ethical considerations to consider both in its design and
implementation.
Ethical Concerns
The primary ethical concerns relating to this capstone project relate to privacy. This
project seeks to investigate two fundamental methods as mechanisms to track user behavior and
build corresponding behavioral models using machine learning, each of which has a discrete set
of privacy considerations.
Location Services. While location tracking capabilities, which are commonly called
location services, provide substantial capabilities in a variety of mobile devices, they can be
viewed as useful or invasive. Mobile devices commonly leverage location services to provide
useful functions such as GPS based directions and geo-tagging of objects such as pictures.
Nevertheless, many device users are not comfortable with the notion of their every movement
16
and location being tracked and recorded. As a result of these concerns, some users opt to turn off
A common use of location services in home automation is for the control of lighting, for
example, to turn lights on or off when entering or exiting a region such as a geofence around
somebodys home. In the case of iOS devices, Apple Inc. (2017) states that Location Services
uses GPS and Bluetooth (where they're available), along with crowd-sourced Wi-Fi hotspots and
cellular towers to determine the approximate location of your device. One of the primary
mechanisms that this project plans to use for precisely tracking user location within a home is via
Bluetooth communications between a personal device such as a mobile phone, e.g. an iPhone, or
a wearable device, e.g. Apple Watch and beacons. Any user who wishes to completely turn off
all location services would not be able to benefit from the techniques discussed in this paper.
Nevertheless, the opposition to tracking everywhere a person goes outside of their home
is expected to be more more objectionable than tracking where they are inside their home. For
example, the former may expose illicit or morally objectionable behavior that a device user may
wish to conceal, while in the case of the latter, a visit to the closet or the living room is less likely
validating with potential users however should this research yield viable prototypes.
One method of mitigating this concern, should it become one, would be to use some
ancillary device other than mobile device as a tracking mechanism to represent the user, such as
Cameras. The other primary method this research intends to evaluate as part of its efforts
to build user behavioral models is using IoT cameras. As demonstrated by the Persirai IoT botnet
17
in May 2017, IoT cameras having proven to be a favorite target of malware authors and used as a
launching point for distributed denial of service (DDOS) attacks. Additionally, while users have
generally been accepting of storing streaming video of activity outside their homes with some
external service provider as a method of home security, it is expected that users would have
strong opposition to continuous video recording inside their home, particularly in certain rooms
within the home such as bedrooms and bathrooms where a generally high level of privacy should
rightfully be expected.
This research intends to use IoT cameras to perform facial recognition as a means to
discern and identify various actors within the home and eventually use those recognition
capabilities to apply learned behaviors. For example, this could mean the detection and
application of different lighting based upon who enters the front door. This research aims to
mitigate concerns about cameras within a home by positioning them to cover main points of
ingress and egress such as a front door or garage entry door and main living spaces such as a
family room and kitchen. This research will not consider applications of IoT devices in other
parts of the home, and therefore anticipates a hybrid approach between the bluetooth and camera
Negative Impacts
non-automated counterparts, and thus the barrier for entry could be very high for lower income
individuals and families. According to Smith (2017), while 77% of all Americans own a
smartphone, as many as 64% of low income owners also own smartphones. Bluetooth is a
18
leverage at least some of techniques which are part of this research and nearly all smartphones
users should theoretically, from a technology perspective, be able to leverage these techniques
using capabilities built-in to their phone. The bigger issue which may exclude low-income
groups would be the additional cost of the items required to track location inside a home such as
A long term goal of this research is to determine whether it would be useful or feasible to
incorporate BLE chips in more commonly used devices such as light switches and power plugs
to facilitate some of the techniques this research will examine. The ability to do so in a viable
manner may reduce the potential financial burdens associated with implementation of the
technology.
Environmental Impacts
One of the primary motivations behind this project and indeed home automation in
general is to have a positive environmental impact by enabling people to use their homes more
efficiently. Automation of lighting is a common entry point into home automation and this
research intends to build techniques that will make the use of homes and devices more efficient
References
Apple Inc. (2017, February 24). Turn Location Services and GPS on or off on your iPhone, iPad,
https://support.apple.com/en-us/HT207092
Collotta, M., & Pau, G. (2015). A Solution Based on Bluetooth Low Energy for Smart Home
Shaikh, S., Attar, A., Pathan, S., & Shaikh, R. (2016). Machine Learning On Real-Time Data to
Smith, A. (2017, January 12). Record shares of Americans now own smartphones, have home
http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/
20
Appendix
Sen.se provides sensors and and iOS application that potentially embody part of the
A critical aspect of whether any existing solution will be viable to include with
this research is an API, in particular one which will expose the ability to use proximity
detection that can be read by some external observer to record and store that information
in a separate system. This data can then be analyzed by machine learning algorithms to
potentially correlate positions, in this case proximity to a beacon, with state changes
occurring within the home. Upon cursory examination, Sen.se seems to have a
well-formed API for this purpose, though a more thorough examination will be required.