Evans: Solving Home Automation Problems Using Artificial Intelligence Techniques
SOLVING HOME AUTOMATION PROBLEMS USING ARTIFICIAL INTELLIGENCE TECHNIQUES
Grayson Evans
Director of Engineering, MTI Inc., Portland, OR
Abstract
The most difficult problem facing the success of
future consumer home automation products is
not a purely technical one: how to make the
products or systems intuitive to use, work
together, yet powerful enough to have real
perceived value to the consumer.
The paper describes how existing Artificial
Intelligence (AI) techniques are directly
applicable to solving the human interface
problem of home automation products even
when the "system intelligence" is widely
distributed among different products sharing a
common home network.
Introducton
Existing product user interfaces found in the
home still represent a formidable obstacle to
most people. With the arrival of new home
automation technology such as CEBus the
number of new products about to face the
consumer-that require yet another user
interface-will increase dramatically.
The home owner does not have any idea how
to operate the systems (heating/cooling,energy
management, security) in his home in the most
efficient yet most comfortable and safe way.
To be a truly useful, home automation
products must be able to adapt to their environ-
ment. They have to know what to do without
the home owner having to tell the device
everything and constantly "fuss" with it. The
best user interface is the one that requires the
least user interface.
Solving these problems will be one of the
best contributions of Artificial Intelligence (AI)
technology yet devised.
The AI Tools
The tools needed to solve these problems
already exist in many forms of AI technology
and have gained wide acceptance in commercial
areas. The following sections describe applic-
able techniques, examples of their implemen-
tation and use, the problems they solve, and
operation in a home network.
Voice Operation
Most people that learn about home automation
seem to assume that "automated"products have
(or should have) voice recognition capability.
Manuscript received June 7, 1991 0098 3063/91/0200 00395S01.00
395
In fact, most people feel that in the future all
home automation products will use voice as the
means of control. It's a feature that the
entertainment media has made a requirement in
the mind of the consumer.
Typical applications include:
Voice command - instructing a product or
system to perform a function
"Please raise the temperature in the living
room" (how much? for how long?)
Voice query - obtaining information from a
product or system
"How much have I spent this month on
electricity?"
It doesn't take an engineer to see the benefits of
using voice to eliminate other "unnatural" forrns
of product operation. Unfortunately, voice
recognition-the ability to recognize discrete
words, either speaker dependent or
independent-is only a first step in voice
operation. Voice recognition provides little
ability beyond voice driven "functions keys".
For example, the words: light, room, kitchen,
It
on" recognized in that order, might be used to
turn on lights in the kitchen. "Kitchen, room,
light, on" may produce no action or cause
unwanted results. While recognition is an
accomplishment in the right direction it is only
the first step in language processing needed to
make voice a useful tool. The remaining
requirements:
Syntactic analysis - word order and
modification
Semantic recognition (Context sensitive) -
word meaning in the context of
the home and systems addressed
Pragmatic analysis - testing word usage
and meaning for reasonableness in the
context of the home
In fact, the ability to recognize words
without the ability to provide the necessary
analysis of syntax and content only makes the
user interface problem worse. It simply gives
the home owner the ability to get frustrated
much faster.
But, oddly enough, the most difficult
Q 1991 IEEE
396 IEEE Transactions on Consumer Electronics, Vol. 37, No. 3, AUGUST 1991
problems do not lie with the more sophisticated
aspects of syntactic and semantic analysis but
with the more mundane requirement of simply
sorting out separated speech in a typical home
with a typical family. The current state of voice
recognition is not very useful for an
uncontrolled environment such as a home.
Voice recognition is a difficult problem in the
best of circumstances. In a home environment,
it is nearly impossible.
Y1it.h praducts o p ~ i a t i in
~ ~agdistributed
environment such as =Bus, all the needed
voice operation technology need not be
contained in any one product. A single voice
recognition product-say, the voice recognition
telephone-can place the recognized words on
the house network in a message to be picked up
and used by a product that only contains
syntactic or semantic analysis that in turn can
place a resulting control message on the
network.
Figure 1 shows how the various parts of a
complete home automation system employing
all the techniques proposed in this paper are put
together to form a working system.
Figure 2 shows how the same techniques can
be applied in a group of home automation
products using a network such as CEBus.
Expert System Techniques
The home will undoubtedly prove.to be the best
(and most difficult) applications for expert
system software. The family would like to
operate their home in an efficient, comfortable,
and safe manner. It is not intuitively obvious
how best to accomplish these sometimes
conflicting goals. For example, is it more
efficient-given the present outside and inside
temperature, humidity, desired comfort level,
and present electric rate-to operate the heat
pump to cool the house, use outside air, a
combination, or do nothing and wait for the
house to loose enough heat? Certainly the
home owner doesn't know. However, an expert
in thermal systems, HVAC equipment and
human physiology could determine the best
choice given temperature and humidity data and
a thermal model of the home. Expert system
software can easily embody this knowledge and
adapt it to a particular home through a learning
process. Temperature, humidity, and other
sensors can easily provide the basic infomation
needed by the software.
Typical home applications include
anticipation of family requirements
learning most efficient operation of the
home systems
achieving safer operation of the house and
its systems
prcviding advice on system operation
Expert systems are created from a knowledge
base, a set of rules, and an inference mechanism
for applying thc rules io die knowledge base.
There may or may not be a means of adding to
the knowledge base. The raw knowledge base
neededto suppport home automation consists of
two parts: the information about the house itself,
and information about the family living in it:
HOUSE KNOWLEDGE BASE
physical layout, openings, connectivity
types of system in house
heating, cooling,
water, gas, etc.
communications,phones, etc.
thermal characteristic
environment
costs: elec. rates, gas rates
OCCUPANTKNOWLEDGE BASE
names
relationships, age
authority
likes/dislikes
habits
The knowledge base is created in two ways:
simple manual entry of facts by the
manufacturer, dealer, and/or home owner, and
synthesis of the knowledge by observation and
learning.
The rule base contains the basic operating
rules to apply to the knowledge base. Unlike
knowledge about the house, most of the rules
are built in. That is, they are derived and
entered by the manufacturer based on "expert"
knowledge about how best to operate a house.
A learning process is applied to the rule base to
both modify the rules and derive new rules
based on existing rules-rules that only apply to
one particular family or house.
An inference mechanism applies the rules to
the knowledge base driven by information
needed by other parts of the system.
The rules and knowledge about the house
 Evans: Solving Home Automation Problems Using Artificial Intelligence Techniques 397

Environmental Data
House
Occupants

Figure 1. Elements of an artificial intelligence system arranged in one “central unit It.

Communication between elements is “hardwired Sensor inputs, flector outputs ,

‘I.

while remotely located are all wired to one location. Not explicitly shown is LEARNING since
this process is embodied in many places throughout the system. The WORKER software shown
represents most of the traditional home automation software used in existing systems to
perform routine tasks and interface to flector (output)type devices.

Knowledge Bsse

1 Smart
Security System
Knowledge Base
.Occupant data
TEMPLETS
Knowledge Base Voice
House Data LIGHTS,
VE&, etc. HEATER,etc.

Figure 2. Elements of an artificial intelligence system which are distributed over a home automation
bus such as CEBus. Elements must now share gained information by broadcasting the result of an
analysis ,recognition, inference, etc..
398 IEEE Transactions on Consumer Electronics, Vol. 37, No. 3, AUGUST 1991
form a working model of the house, its behavior
and the occupants behavior. This model
mimics a similar model that forms in the
occupants minds, a model of the behavior of the
home automation systems. The better the
models, the more synergistic will be the
relationship between the occupants and the
home automation system.
The "parts" that make up an expert system
need not reside in one physical place but also
can be distributed on the network. The
knowledge base, rule base, templets, and
inference mechanism can be located in different
products (assuming some division of knowledge
between different products) and still operate
successfully.
Pattern Recognition
Pattern recognition is usually used in the context
of machine vision- identifying the contents of
an image supplied from a camera or other
imaging device. Voice recognition is a special
case of pattern recognition. In the context of
home automation the term is used to describe
the recognition of living patterns. While very
different from machine vision, the techniques
used for identifying patterns are the same.
Typical home applications include:
determining the occupancy state of home
observing most likely way to operate the
home
looking for repetitive patterns to
determine:
-when to do things
-where people are
-abnormal or unsafe conditions
-if the occupants are in need of help
While the types of sensors available to detect
the presence of people in the home are limited
they may be used in combination-via the home
automation network-to achieve remarkable
results. People detectors (short range passive
infrared) throughout the house, magnetic door
and window switches on all openings, power
sensors, light sensors, temperature sensors, and
other simple analog and digital devices can
provide a crude "picture" of the state of the
home. Even such information as phone lines in
use, water flow, and what TV channel is being
watched can enhance the picture. This picture
can be analyzed using similar techniques used to
analyze more traditional digitized pictures but
because the sensor input is limited the patterns
do not require extensive processing.
The North American, single family,
detached home is basically the same throughout
the country. The activities that take place in the
home tend to follow a regular pattern and are
fairly limited. People get up, bathe, eat
breakfast, go to work, go to school, come home,
eat dinner, watch TV, go to bed, sleep, and get
up to go to the bathroom. The patterns that
need to be learned and remembered can be
observed in a somewhat short period.
In practice, multiple sensor input is feed to
a "feature extractor" employed to filter out
"false alarm" sensor activity, extract pertinent
information, and parameterize that information.
This preliminary information can be used to
figure out basic, but extremely useful, state
information about the house. For example, by
knowing only four basic states:
OCCUPIED - people are in the house
ACTIVE - people are moving around in the
house
INACTIVE - people are in the house but
are not moving around
UNOCCUPIED - the house is empty
we can greatly enhance the operation of so
called "programmable" devices. Products can
easily modify their behavior or operation to take
advantage of the extra information, such as
operating the dishwasher only when the house is
unoccupied, or operating lights only when the
house is occupied and active.
By applying time constraints to known
basic patterns and knowing something about
basic American family habits, additional
subcategories of states can be determined such
as: having breakfast, taking a bath, preparing
meals, having dinner, watching TV.
The processing elements for pattern
recognition also can be easily distributed among
many products. The sensors are already
distributed, feature extraction can be done on a
"need to know" basis by each product. For
example, a security system may only be
concerned with an UNOCCUPIED state. Once
this state is determined, a message is placed on
the network indicating UNOCCUPIED state
detection. The water heater may use this
information plus time message traffic to set back
the water temperature.
The processing elements for pattern
recognition also can be easily distributed among
Evans: Solving Home Automation Problems Using Artificial Intelligence Techniques
many products. The sensors are already
distributed, feature extraction can be done on a
"need to know" basis by each product, and the
detected state of one product can be "passed
along" to other products.
Learning
Learning is integrally tied to the capabilities of
the three previous techniques. Learning, for
purposes of home automation products, means
changing "behavior", or operation, in a
beneficial way based on experience. Learning
requires the altering or changing of the
knowledge base and rule base that the home
automation systems uses to perform tasks or
generate actions.
Typical home applications:
improving the "behavior" of products
freeing the family from "teaching"
products
building a knowledge base about the home
environment
adapting product behavior based on family
feedback
The process of learning requires two things.
First, a method of feedback. If behavior is to be
changed based on observed input, there must be
some way to know what needs to be changed. A
learning technique must decide (1) that it is not
sure whether a certain action was appropriate,
(2) that some particular input is relevant to
determining if an action was appropriate, (3)
what this relevance is, (4) what the actions
should have been, and (5) how to change its
knowledge base so that future actions will be
more appropriate. These choices involve some
extremely subtle decision making. There is no
obvious input to a home automation system
labeled "feedback". There must be some
method to observe ovemdes, contradictions, or
other actions (either from the house or
occupants) to determine that an action taken was
inappropriate.
Two forms of feedback in the home can
provide the majority of information needed for
learning.
1) The physical characteristics of the home
are learned primarily via sensor feedback. For
example, electric energy usage, temperature, and
humidity throughout the house can be used to
determine the thermal characteristics of the
house and the efficiency of the heatinglcooling
399
systems.
2) Owner feedback comes from requested
changes to schedules, ovemdes, cancellations,
manually requested actions, and family activity.
If a product performs an action and the action is
constantly cancelled by the owner it is easy to
learn not to do it again. If someone always
requests an action (turns on the TV for example)
when he arrives home from work, the TV can
simply learn to perform the task automatically.
The second requirement for learning is a
"weighting"of information. A learning
technique relies on the altering of weighting of
probable actions. When an action was
determined to be inappropriate, a decision must
be made to reduce the weighting function used
to generate that action. The true skill of a
learning algorithm can be judged by its ability to
determine the correct thing to change and by
how much. The difficult task in developing this
skill lies in setting feedback thresholds-that is,
determining how much "weight" to give
feedback information.
Two techniques can be used to help
improve weighting functions in the home: a
behavior predictability function developed by
pattern observation; and asking for assistance
via voice' synthesis.
This learning process should be applied to
all parts of the knowledge base: the
characteristics of the house, the habits of the
occupants, the rules of operation, the patterns
looked for to determine when to do things, etc.
Summary
Solving the problems inherent in the user
interface of home automation products and
systems will be one of the best applications for
AI ever invented. Home automation products
can employ techniques from all areas of AI:
language processing, expert systems, pattern
recognition, and learning. The author believes
that the long term market acceptance of home
automation products will, in fact, require the
ability to adapt to the family. An ability that
A.I. techniques can provide.
400 IEEE Transactions on Consumer Electronics, Vol. 37, No. 3, AUGUST 1991

Biography
Grayson Evans is currently the Director of
Engineering at MTI Inc. in Portland, Oregon
and a consultant to the EIA for the development
of the CEBus home automation standard. Prior
to MTI, Mr. Evans was Manager of Applica-
tions Engineering at Echelon Inc. From 1983 to
1987 he was founder and President of
Archinetics Inc., a leading manufacturer of
home automation systems.