Você está na página 1de 11

Received February 28, 2017, accepted April 10, 2017, date of publication June 14, 2017, date of current

version July 3, 2017.


Digital Object Identifier 10.1109/ACCESS.2017.2714647

Trusted Interconnections Between a Centralized


Controller and Commercial Building HVAC
Systems for Reliable Demand Response
C. BIRK JONES1 AND CEDRIC CARTER, Jr.2
1 Sandia National Laboratories, Photovoltaic and Distributed Systems Integration Department, Albuquerque, NM 87123 USA
2 Sandia National Laboratories, Resilient Control Systems Department, Albuquerque, NM 87123 USA
Corresponding author: C. Birk Jones (cbjones@sandia.gov)

ABSTRACT Dynamic smart grid operations require that utilities that incorporate intermittent renewable
energy resources provide creative and inclusive solutions to reduce and shift electrical demand. Commercial
building HVAC systems have been used as a dispatchable load, but are limited by the lack of interoperability.
Increased operability can be achieved using a secure and reliable interconnection device, which can
provide direct bidirectional communications between the utility and the building automation system (BAS)
controllers. This paper developed and tested a building automation intrusion detection system (BAIDS) that
can provide a cyber-secure connection between public and private BAS networks. The BAIDS was used
in a hardware-in-the-loop experiment that connected an actual photovoltaic array with a BAS control test
bed and a building zone model. The BAIDS device allowed for critical control signals to pass from the
public network directly to a fan controller in a BAS private network. At the same time, the BAIDS device
provided intrusion detection monitoring to identify malicious activity. The network traffic was evaluated
using an adaptive resonance theory (ART) artificial neural network. The ART algorithm was able to learn
normal traffic activity on the private and public networks. The algorithm was then used to detect unauthorized
attempts to access the interconnection device and a malicious cyber-physical attack on the BAS.

INDEX TERMS Demand response, cyber security, intrusion detection, power demand, data security,
centralized control, intelligent control.

I. INTRODUCTION Increased interoperability with commercial building heat-


Building infrastructure can support smart grid applications, ing, ventilating, and air-conditioning (HVAC) equipment can
including the integration of non-dispatchable renewable provide more predictable and accurate responses to grid
resources and resiliency measures. There have been a number needs [3]. Targeted investments to upgrade and expand the
of studies that describe various aspects of this paradigm, in use of automated control systems in commercial and indus-
which buildings are not seen only as passive loads. Building trial facilities have been estimated to double the ability of
systems can also be active participants in the energy delivery buildings to shed electrical loads quickly [4]. However, con-
and generation system. As contributors to the smart grid, necting different information technology systems, software
buildings can be managed intelligently to reduce the over- applications, and platforms to communicate and exchange
all electrical demand permanently through energy efficiency data offers many challenges. This increased functionality
measures, or at short timescales using demand response (DR) introduces new vulnerabilities that are susceptible to cyber
controls [1]. Siano [2] discusses the potential benefits of using attack vectors that can spoof data, manipulate control settings,
DR as an integral part of smart grid operations, including and modulate device disconnects [5]. Data and control signals
deferred upgrades and reduced capital costs. Enabling tech- must be transferred reliably and securely through complex
nologies for smart grid applications, such as smart meters, networks while completely avoiding cyber attacks [6].
communication systems and energy controllers, requires a Communication systems for DR activities require sophis-
high degree of interoperability. ticated security protocols. The exchange or transmission of

2169-3536
2017 IEEE. Translations and content mining are permitted for academic research only.
VOLUME 5, 2017 Personal use is also permitted, but republication/redistribution requires IEEE permission. 11063
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

data between devices must be trusted to avoid privacy leaks, analyzing inbound and outbound traffic on the local and
malicious denial-of-service attacks, and undetected modifi- public networks.
cations of information by unauthorized individuals or sys- The network traffic monitoring systems deployed within
tems [7]. Wang and Lu [8] warns that the operations of critical the BAIDS device included the collection and analysis of
systems can be delayed, blocked or corrupted by denial-of- network traffic. The traffic was collected using a network
service attacks. Also, targeted attacks, that disrupt sensor sensor that read message values and the associated metadata.
values, change system settings or modify control commands, It then inserted the information into an onboard elasticsearch
can create safety problems or equipment damage. database. The data was accessed by an Adaptive Resonance
Robust and trusted network connections are critical for Theory (ART) neural network that first learned normal
expanding DR capabilities in commercial buildings. This behavior. It was then presented with previously unseen traffic
work developed and tested a device that will improve net- and was able to detect two types of cyber attacks. The first
work connectivity for DR applications while minimizing attack was an unauthorized attempt to access the BAIDS
the potential for successful cyber attacks. The proposed device using SSH. The second malicious activity was a cyber-
Building Automation Intrusion Detection System (BAIDS) physical attack from inside the BAS that shut off lights and
bridges the private BAS local area network (LAN) with the HVAC fan. The ART algorithm successfully identified the
the internet or public network as shown in Figure 1. The attacks with minimal false positives.
private LAN interconnects devices within a limited area and
restricts outside access. Alternatively, the public network II. BACKGROUND
allows anyone access to other networks or the internet. The Connecting centralized controllers with BAS has been dis-
bridge between the public and private networks grants a cussed by Piette et al. [9]. Reliable connections can be
centralized controller the ability to establish two-way com- achieved using devices that comply with the Open Automated
munication with existing BAS control networks. This allows Demand Response (OpenADR) standard [10]. The open stan-
for more detailed control configurations that incorporate dard provides a non-proprietary interface for utility compa-
dispatchable loads from a large set of buildings. It can nies to send control signals to a building system with the
also consider occupant comfort as a control parameter. The intent of modifying electrical loads. The OpenADR architec-
BAIDS was tested in an hardware-in-the-loop experiment that ture uses the open internet to connect the utility or Indepen-
connected a building zone thermal model, a photovoltaic (PV) dent System Operator to a DR Automation Server (DRAS).
array, and a building automation testbed. The experiment An internet connection has been used to connect the DRAS
also tested passive cyber intrusion detection capabilities by with a Client Logic Integrated Relay (CLIR) or gateway [11].
The signal then penetrates the BAS using an internet relay
for the CLIR or through a web service client in the case of
the gateway application [12]. However, current research has
limited information regarding the security for the gateway
connection in DR applications. The BAIDS can provide the
added security necessary to create a fast and trustworthy
connection between the utility and the commercial building
control network.
A secure connection with the BAS is necessary because
cyber adversaries can easily exploit the BAS network to
attack both corporate network devices and HVAC systems.
For example, the corporate network at a Target retail store in
the United States was breached by international adversaries
who used the BAS network to access the corporate network
and were able inject code into the credit card machines [13].
The code read and delivered the card information of hundreds
of customers to the adversaries. Additionally, the common
HVAC communication protocol, known as BACnet, has many
vulnerabilities, and was not originally designed with cyber
security in mind.
Building automation professionals have recognized secu-
rity issues associated with BAS [14] and have implemented
FIGURE 1. The Building Automation Intrusion Detection System (BAIDS) various security measures to protect private BAS control
connects building automation controls devices located on a local area networks. Many of the methods include the segregation of the
network (LAN) directly with a centralized controller. The centralized
controller is then able to implement demand response activities at a
BAS network from the public network. In these cases, BAS
larger scale while considering a broad range of parameters such as professionals have built a secure firewall around the BAS
occupant comfort. networks. The firewall allows authorized users to access the

11064 VOLUME 5, 2017


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

controls network through virtual private network connections used to smooth cloud-driven intermittency as described by
or remote desktop access [15]. Proposals have also been made Mammoli et al. [22]. This synchronization requires a direct
to improve the communication protocol by wrapping security connection between the renewable energy resources and
into the BACnet messages [16]. However, BACnet embed- building HVAC equipment. The BAIDS provides this link
ded security measures are not robust. For example, session with a cyber secure, bidirectional connection that provides
keys used to encrypt transmitted data can be manipulated. reliable communication with existing HVAC devices for
The BACnet protocol is susceptible to man-in-the-middle advanced DR control. This paper reports on a hardware-
attacks, parallel interleaving attacks, and replay attacks [17]. in-the-loop experiment that used an actual PV array and a
Maintaining the integrity of the building’s public and BAS highly functional building automation testbed. The experi-
networks is a critical task that must be conducted in parallel ments tested the ability of the BAIDS to transfer DR signals
with the DR controls. An interconnection device, such as to the control network while providing a high level of cyber
the proposed BAIDS, can provide meaningful DR control oversight to prevent malicious activity.
capabilities while maintaining a high level of security.
The BAIDS connects centralized controllers directly with A. BUILDING AUTOMATION INTRUSION
individual BAS devices. The connection allows centralized DETECTION SYSTEM
control algorithms to modulate fan speeds, turn off lights, The BAIDS was deployed in a Raspberry Pi (RPi) linux
change thermostat states, and receive status updates. The con- computer that connected the public and private networks and
nection is achieved with a bridge between the BAS network had 1 GB of ram and 32 GB of memory. The RPI is a
and the public network. The interconnection requires robust cost effective, reliable, and energy efficient computer that
security measures that include intrusion detection algorithms. has been applied to various applications [23]. For example,
Intrusion detection methods have been developed for many the RPI has been used as a data management server [24], it
different applications, including building automation net- has been applied to smart grid applications [25], and it has
works. The algorithms must be adaptive to accurately detect been integrated into PV systems for advanced monitoring
different types of attacks. Attack methods are dynamic and and analysis [26]. In the present work, the RPI was used to
continually change with time and technology. Data mining control HVAC devices, monitor network traffic, and display
platforms have been developed to adapt and collect many performance results.
different types of features that describe network topologies The RPI was equipped with Python programming language
and traffic [18]. Anomaly detection frameworks have focused code to transmit control signals, monitor network activity, and
on the BACnet protocol and have used modern databases detect traffic anamolies. Custom Python code was developed
and advanced analysis methods such as an inductive rule to retrieve PV output data, calculate the fan motor control
learning algorithm to classify traffic with very low false signal, and then transmit the signal to the actual control
positive rates [19]. Other detection algorithms have identified device. The transmission of the control signal was achieved
anomalies at very low error rates using an artificial neural using custom Python code that could communicate with
network [20]. Tonejc et al. [21] compared clustering, random the BACnet protocol. The network monitoring code used
forests, one-class support vector machines, and support vec- Pcapy Python extension module, which uses the libcap packet
tor dimensionality reduction unsupervised machine learning capture library that can discover transmitted and received
algorithms for anomaly detection in BACnet networks. messages. The RPI device was also used to store and display
Past approaches have been shown to be effective. Yet, many system performance and network traffic data. This stor-
including the support vector machine, require training and age and visualization system was provided by elasticsearch
testing data sets in order to analyze new data streams. The database and Kibana HMI. The RPI was also equipped
ART algorithm, on the other hand, has the ability to learn with ART algorithm Python code that analyzed traffic every
behavior, store the memory, and then delete past data. This minute.
allows the BAIDS device to operate without a large amount
of data storage and limits its need for communication with B. TESTBED & PHOTOVOLTAIC ARRAY
an off-site storage system. The present work tested the ART Commercial buildings typically have multiple rooms for
algorithm’s ability to learn normal behavior and then identify office space, conference rooms, and other uses. The rooms
brute-force and cyber-physical attacks. are designed strategically for heating and cooling by defining
thermal zones. A thermal zone is a space within a building
III. METHODOLOGY that can include one or more rooms that are controlled by
Modernization of the electric grid can include the integration a single thermostat. The conditioning of the thermal zones
of large-scale, on-demand control of commercial building can be controlled using a BAS. The Building Automation
HVAC equipment. The simultaneous control of numerous Control System (BACS) testbed, located at Sandia National
HVAC equipment can be coordinated from a centralized Laboratory, was built to simulate common BAS used to heat
platform in a secure and reliable manner. This type of con- and cool thermal zones. The testbed has several controllers,
trol has the potential to synchronize building power demand boiler, lighting, energy meter, chiller, fan, duct work dampers,
with renewable energy output. For example, DR can be and thermostats (T-Stat). Figure 2 shows the infrastructure

VOLUME 5, 2017 11065


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

connected through as simple network topology that included


the BAIDS as a key link between the public and private
networks.

FIGURE 2. The building automation testbed includes various building


HVAC equipment and controls such as a chiller, boiler, fan, etc. This
experiment used the thermostat (T-Stat), zone damper, damper actuator,
fan motor, and the BAS controller. The BAS used the standard
communication protocol called BACnet to transmit message. This protocol
was used by the Building Automation Intrusion Detection Device (BAIDS)
to synchronize fan power demand with the local photovoltaic array
generation.
FIGURE 4. The demonstration had multiple network connections. First,
local weather and solar PV data was accessed through an API on the
public network. Second, control signals and performance feedback were
sent and received on the private network connection. Additionally, the
used in this experiment, including the T-Stat, zone damper, BAS and public network traffic were monitored on the spam and public
damper actuator, fan motor, and the BAS controller. ethernet connections respectively. The BAIDS used a Raspberry Pi (RPi) to
connect the two networks. The RPi ran Python programming language
The experiment used the commercially available control scripts to send signals and monitor traffic. It also used elastic-search
system, known as Delta Controls, to simulate a typical cool- database to store and view historical data.
ing of a single thermal zone. The system uses the BACnet
protocol to transfer control messages. It was originally oper-
ating on a secure LAN that was segregated from any public C. NETWORK TOPOLOGY
network. The experiment connected the BAIDS device to the The network topology connected the PV array with the BACS
BAS LAN and a public network. This connection allowed for testbed HVAC devices as shown in Figure 4. A hypertext
DR signals to be sent from a nearby PV power plant to the transfer protocol (HTTP) application programming inter-
HVAC fan. face (API) was used to transfer the data from the PV mon-
itoring system to the BAIDS device. The BAIDS device was
connected to the public network through the ethernet 1 (eth1)
connection provided by a network switch. The BAIDS device
was also connected to the BAS switch through the ethernet 0
(eth0) spam port and the ethernet 2 (eth2) private network.
The spam port, labeled as eth0, connection was used to
monitor all the traffic in the BAS network and perform intru-
sion detection evaluations. The eth2 network connection was
established to send control signals to the BAS devices. This
connection was also used to connect the zone thermal model
FIGURE 3. The data collection device for the actual photovoltaic array
measured the produced power and quickly sent the value to the Building
with the testbed. For example, the air flow rate, measured by
Automation Intrusion Detection System (BAIDS). The signal was then a pitot tube sensor, was sent to the thermal model in real-
processed by BAIDS and used to control the testbed fan motor. time. Finally, the weather, solar PV, HVAC performance, and
network traffic data could be visualized in a human machine
On the other side of the laboratory was a 10.8kW PV array interface (HMI). The HMI was developed using elasticsearch
(Figure 3) that had a monitoring system connected to a public database and Kabana visualization tools.
network. The PV array consisted of four strings each with
10 modules in series that terminated at a single inverter. The D. HARDWARE-IN-THE-LOOP SETUP
inverter was connected to the electric grid. The monitoring The experiment integrated an actual solar PV plant and a real
system collected DC data from a combiner box and AC data fan motor/damper system with a virtual thermal zone model.
from the inverter. The measured AC output from the PV array The setup created a hardware-in-the-loop situation where the
was sent to the BAIDS device and was used to control the fan actual damper position was dependent on the temperature
motor in the BACS testbed. The testbed and the PV array were output of the zone model. Also, the fan motor speed was

11066 VOLUME 5, 2017


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

a function of the power output produced by the solar PV where s was equal to the fan speed, calculated in Equation 1,
array. The experiment showed that an HVAC fan could be when the zone temperature was between 21.6 and 23.8◦ C
synchronized with a PV plant using a cyber secure connection during the occupied hours of 8 in the morning to 5 in the
without degrading the occupant comfort. afternoon. When the temperature was outside of the defined
threshold, no command was sent and the fan motor and
the system operated as instructed by the existing control
software.
The actual air flow from the fan travelled through duct
work to a damper controlled by an actuator. The flow that
passed the damper section was measured by a pitot tube
sensor and the value was sent to the zone model using BACnet
UDP/IP read commands. The zone model also made queries
of the HTTP API to collect the most recent ambient temper-
ature value. Based on these two actual inputs the zone model
ran a first order differential equation to compute the zone
temperature (3):
d
M Tz = Q̇ + ṁCp (Ts − Tz ) (3)
FIGURE 5. The hardware-in-the-loop experiment connected the BAIDS
dt
device to the BACS testbed. BAIDS controlled the testbed fan based on where M is the air mass inside the zone,
the solar PV power. The output from the fan and damper system was
input into the zone model located within the BAIDS device. The output Tz is the zone temperature,
from the model was then sent to the thermostat within the testbed. Q̇ is the thermal load,
ṁ is the mass flow rate of air to heat or cool the space,
The experiment had four significant connections that were Cp is the specific heat capacity of air, and
necessary to provide reliable DR. The connections, shown in Ts is the supply air temperature
Figure 5, included: The thermal load (Q̇) was computed for a zone that was
1) HTTP API: 3 meter long by 4 meters wide and 2.4 meters tall. Three of
a) Actual Solar PV → BAIDS fan control algorithm the wall surfaces had adjacent interior walls and one surface
b) Actual ambient temperature → Zone model had an exterior wall that faced west. The thermal load for the
zone was defined by (4):
2) BACnet UDP/IP Control Signal:
a) BAIDS fan control algorithm → Testbed fan Q̇ = Q̇interior + UA(Text,int − Tz ) (4)
motor
where Q̇interior is the interior load,
3) BACnet UDP/IP Sensor Message: UA is the heat transfer coefficient times the surface area,
a) Mass flow rate (ṁ) sensor → Zone model Text,int is either adjacent space temperature,
b) Sensor → elasticsearch database Tz is the zone temperature.
4) Database Query: The interior load was estimated based on a standard occu-
a) Zone model temperature → BAIDS fan control pancy of one person and one computer during normal busi-
algorithm (and BACS testbed thermostat) ness hours of 8 in the morning until 5 in the evening. The
The HTTP API connection provided the most current solar heat transfer coefficient was assumed to be 2W/m2 K for the
PV power production value (PPV ). The PPV value was used four walls. The Text temperature was the outside tempera-
as an input for the fan controller equation: ture extracted from the HTTP API and Tint was a constant
value of 20◦ C that represented the adjacent interior space
   31 temperatures.
PPV − min(PPV )
%Speed = (max(Pfan motor )) The model was run every minute and output the zone
max(PPV ) − min(PPV )
temperature at the particular instance. This output was stored
(1)
in the onboard database and was sent to the testbed control
Equation 1 provided a scaling of the PPV to define the fan system. The value was read on the actual thermostat and it was
motor speed. It applied the affinity law properties where the accessed by the damper control system. The damper actuator
fan power is equal to the cube of the speed. The control modulated independently based on the difference between the
signal (s) sent to the actual fan motor in the BASC testbed zone temperature and the set-point value.
using BACnet UDP/IP was defined by a simple if state-
ment (2): E. NETWORK TRAFFIC SENSOR
( Bidirectional connections with existing BAS networks
%Speed (Eqn. 1), if 21.6◦ C ≥ Tz ≤ 23.8◦ C require that control networks remain accessible to the out-
s= (2)
no command, otherwise side world. However, accessible internet connections allow

VOLUME 5, 2017 11067


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

adversaries to access critical data, devices, and equipment. classification. The ART algorithm incorporates a vigilance
Therefore, BAS are often connected on private networks that parameter (ρ), which can be characterized as a similarity
do not allow public access. The BAIDS provides access to the parameter. This parameter is used to judge the similarity
private BAS networks by creating a secure bridge between between all of the input patterns.
the public and private BAS network. The secure connection
required the implementation of a passive intrusion detection
system (IDS) that monitored traffic on both the public and
private networks.
Passive IDS provides security in networks and comput-
ers by identifying unauthorized access and misuse in real
time [27]. The monitoring provides critical alerts that identify
potential attacks. It can also help operators find vulnerabili-
ties within the system. In this experiment, IDS was embedded
inside of the BAIDS to monitor traffic along two network
segments. The monitoring was performed using custom
developed Python script.
The Python sensor monitored and classified TCP, UDP,
ICMP traffic based on the following rules:
1) Unknown source IP: Inbound and outbound traffic that FIGURE 6. Flow diagram of ART training algorithm where inputs are
originated at an unknown IP address. normalized in the F0 layer. Then category choice and vigilance are
2) Unknown destination IP: Inbound and outbound traffic processed in the F1 layer. The final F2 layer is where the templates are
created and stored.
that terminated at an unknown IP address.
3) Unknown source port: Inbound and outbound traffic The basic structure of the Fuzzy ART architecture is shown
that originated at an unknown port. in Figure 6. This flow chart describes the interconnection of
4) Unknown destination port: Inbound and outbound traf- the different layers that are designated as F0, F1, and F2. The
fic that terminated at an unknown port. F0 layer is for the normalization and complement coding of
5) BAS traffic: Frequency of inbound and outbound traffic the input values, and is considered a preprocessing operation.
from each device in the BAS network. The F1 layer, which includes the choice and vigilance calcu-
6) BAIDS traffic: Frequency of inbound and outbound lations, performs the recognition of features in the data. This
traffic from the public network. process includes a reset that is based on a vigilance a param-
The Python script used the existing pcapy library to capture eter (ρ), and is considered the orienting subsystem [30]. The
network packets. Pcapy is highly effective when used in con- final layer, F2, is where the categorization occurs. Similar to
junction with a packet-handling package such as Impacket. other ANN architectures, the Fuzzy ART has a weight (or
Impacket is a collection of Python classes for constructing template) matrix where the algorithm stores expectations or
and dissecting network packets. The network traffic data was memory.
stored in the database and accessed by the IDS algorithm. The template matrix computations occur between the
F1 and F2 layers. First, the free parameters α, ρ, and β must
F. INTRUSION DETECTION ALGORITHM be set. The choice parameter, α, can assume values between 0
Artificial neural networks (ANN) are a form of machine and infinity, but is typically very small. The learning rate, β,
learning that emulate a simplified version of an animal’s is often set to 1. The vigilance parameter, ρ, must be between
nervous system for the purposes of acquiring and storing 0 and 1, and determines the fineness of the clusters. The first
knowledge. ANN provide a high level learning system that step finds the templates that match best with the input pattern
can perform complex computations, and calculate many non- according to Equation 5:
linear problems. The BAIDS device used an ART ANN |X ∧ Tj |
to learn normal traffic behavior and then detect abnormal c= (5)
α + |Tj |
activity that could represent potentially harmful cyber attacks.
The unsupervised ANN algorithm was created by Grossberg Then, a vigilance test is computed using the best choice
and Carpenter and includes various versions (ART 1, ART 2, template from Equation 5 and the input using Equation 6.
ART 3, and Fuzzy ART). ART 1 is an architecture that can |X ∧ Tj |
be used for clustering of binary inputs only [28]. ART 2 ≥ρ (6)
|Tj |
improved upon the ART 1 architecture to support continuous
If the template (T) does not pass the vigilance test then the
inputs. Fuzzy ART, used in the present work, incorporates
algorithm checks to see if there are other templates available.
fuzzy set theory into the pattern recognition process [29].
If there are no templates available then a new template is
The Fuzzy ART approach can provide stable categorization of
created based on Equation 7:
analog input patterns. The fuzzy logic improves the general-
ization of the algorithm which increases its ability to perform Tjnew = X (7)

11068 VOLUME 5, 2017


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

If there are templates available then Equation 5 and 6 are com- TABLE 1. Training and testing data set sizes for the brute-force and
cyber-physical attack tests.
puted again. Finally, if the template and input pass vigilance
then the template is updated according to Equation 8:
update
Tj = β(X ∧ Tjold ) + (1 − β)Tjold (8)
where X is the preprocessed input matrix,
Tj is the template (weight) vector,
c is the choice function vector and
∧ is the fuzzy set theory conjunction or minimum operator.
The ART architecture is useful for pattern recognition and
categorization because it can adapt easily to significant vari-
ations through a normalization. It can also recognize subtle
differences in input patterns which is critical for accurate
decision making. Additionally, the algorithm can perform FIGURE 7. The network traffic packet information (frequency, packet
detailed recognition of features and is guaranteed to converge length, and checksum) was used to train the Adaptive Resonance Theory
algorithm so that it could identify cyber attacks. The known network
on a solution. The memory can easily be stored and accessed activity was labeled in classes 0 to N. However, unknown and potentially
in the onboard database. These attributes make it a good malicious traffic was labeled as -1.
candidate for evaluating BAS and other network traffic data
streams. and source IP address for each minute of the day. The pre-
processing also computed the one minute average packet
G. CYBER ATTACK TESTS length and checksum. Each instance of the three values were
The BAIDS device was presented with two types of attacks: presented to the ART algorithm as shown in Figure 7. During
(1) brute-force attack on the BAIDS device from the public training the ART algorithm learned inbound and outbound
network, and (2) cyber-physical attack that altered HVAC traffic on the public network for the brute-force test and BAS
and lighting states in the BAS. The brute-force attack is traffic for the cyber-physical test. The algorithm developed
where an attacker attempts to guess the correct passwords or a set of templates (or categories) that represented the traf-
passphrases with the intent to eventually gain access or shut fic characteristics for each connection. The templates were
down the system. In this experiment, the simulated attacker stored in the onboard database. Then, during testing the ART
attempted to gain entry using the secure shell (SSH) protocol. algorithm used the stored templates to analyze previously
The attacker made 4 consecutive unsuccessful attempts using unseen frequency, packet length, and checksum data points.
common usernames and passwords. The second cyber test The algorithm labeled each known instance with a label
was successfully executed on the BAS. between 0 and N that were created during the training stage.
The BAS is a type of cyber-physical system that has various Unknown instances were labeled with a -1. The -1 indicated
mechanisms (fans, lights, etc) controlled and monitored by that it could not find a class for the inputs and was considered
computer-based algorithms. The simulated adversary in the novel. The instances labeled with -1 were then compared with
test gained access to the system and was able to identify the time that the attacks were implemented to determine if the
critical control points in the BAS. The adversary was then ART algorithm had accurately detected malicious network
able to deploy code that sent signals to the fan motor and activity correctly.
light switches that altered the states in a negative manner.
This attack and the brute-force threat had unique network IV. RESULTS
traffic signatures that were analyzed by the unsupervised The BAIDS was used in a hardware-in-the-loop experiment.
ART algorithm. The experiment demonstrated the ability to perform advanced
The network traffic was archived in the onboard database HVAC fan control across multiple IT networks while main-
included a timestamp, packet length, header length, check- taining a trustworthy data connection. Fan control signals
sum, source address, destination address, source port, desti- were sent from a nearby PV array data acquisition system
nation port, ttl, protocol, and IP version. The brute-force test on the public network, through the BAIDS interconnection
used inbound and outbound data to detect abnormal behavior device, and finally to the fan motor controller in the local
(Table 1). The analysis included more than 38,000 training BAS network. The control signal was used to synchronize
data and over 2,400 testing data points for the inbound and the HVAC load with the electrical production from the PV
outbound traffic data points. The cyber-physical attack test power plant. The experiment used the actual fan and damper
used over 1 million training data points and attempted to system in conjunction with a building zone thermal model
detect anomalies within a testing data set that contained more with the intent to simulate the fan control performance. The
than 57,000 data points as shown in Table 1. IDS embedded in the BAIDS device passively monitored
The IDS algorithm on the BAIDS first pre-processed inbound and outbound traffic on the public and local net-
the collected data by computing the communication fre- works. The traffic patterns were evaluated successfully using
quency (hits/minute) that occurred between the destination the ART ANN.

VOLUME 5, 2017 11069


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

A. ADVANCED HVAC FAN CONTROL


The existing BAS controlled the fan motor speed based on
the duct static pressure. The static pressure sensor value
increased or decreased when the fan speed or the zone damper
position was modified. This fan control script, written in the
Delta Controls Software, was not changed for this experi-
ment. Instead, the BAIDS interconnection device performed
an override of the existing system and controlled the fan
motor directly. When the zone temperature exceeded the tem-
perature limits, described in Equation 2, BAIDS relinquished
the override and the fan operated under the Delta Controls
command.

FIGURE 9. A 3 hour period on September 4, 2106 where the fan power


remained below the PV power output during smooth and intermittent PV
electrical production.

FIGURE 8. The fan and PV power had similar profiles over this three day
span. The BAIDS device was able to use the PV data to control the fan so
that it matched with the power produced.

The advanced control of the HVAC fan synchronized the


fan motor power with the electrical output from the PV array.
The fan power matched the PV production patterns as shown
in Figure 8. The fan motor followed the PV profile during FIGURE 10. The modeled thermal zone used in the hardware-in-the-loop
smooth and intermittent PV electricity production. The con- experiment had reasonable temperatures during occupied hours. The
trol was able to modulate the fan quickly in response to the temperature did not exceed 21.6◦ C or drop below 23.8◦ C.

reduced PV power production caused by clouding conditions.


During clouding conditions the fan speed was reduced so that rate produced by the HVAC fan in the BACS testbed. The
the power profile remained below the PV power output. outside air temperature ranged from low of 18◦ C to a high of
The interconnection between the PV array monitoring 32◦ C over the six day period. The experiment required that
device and the fan motor control was not inhibited by the the building zone connected to the fan and damper system
BAIDS device. The signal was sent at a one minute interval maintain a comfortable temperature. The zone temperature
and allowed for the fan motor to adjust to PV output changes. was able to remain between 21.6 and 23.8◦ C during occupied
Fan speed adjustments to maintain synchronization during hours when outside air temperatures exceeded 32◦ C. For
intermittent PV power output can be seen over a 3 hour period example, the zone temperature results are plotted over the
on September 4, 2016 in Figure 9. Figure 9 graphs the fan same two day period as Figure 8 in Figure 10. The temper-
power and PV output at one minute intervals. The fan power ature in the zone remained between the required thresholds
was able to stay below the PV power profile for most of during the day and increased to above 25◦ C during the unoc-
the time. Additionally, the air flow rate provided by the fan cupied hours. The fan control and zone model simulation was
did not negatively impact the comfort level for the simulated conducted in parallel with the cyber security monitoring.
zone.
The fan control experiment was conducted over a six day B. NETWORK TRAFFIC PACKET COLLECTION
period in September. The zone model used the actual outside The BAIDS was used to connect the public and private
temperature measured by NOAA and the actual air flow network. The interconnection provided a necessary bridge

11070 VOLUME 5, 2017


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

that sent signals to and from the BAS private network. The and cyber-physical attacks. The attacks were detected through
connection allowed for fast, and reliable DR control of indi- the analysis of the collected and stored network traffic data.
vidual BAS devices. The trustworthy connection was made The one minute frequency, average packet length, and aver-
possible through the implementation of a passive IDS. The age checksum were presented to the ART neural network for
IDS was embedded in the interconnection device and was training and testing. The training process considered close
able to monitor, store, and analyze incoming and outgoing to 2 million data points collected on the public and private
messages on the private and public networks. networks. The IDS detected potential cyber-physical threats
The IDS successfully monitored and stored network traffic by analyzing the private network activity. The brute-force
in near real-time. The messages were recorded in an onboard threats were monitored on the public network.
elasticsearch database. The database was structured so that The ART algorithm evaluated the outbound and inbound
inbound and outbound traffic on the public and private net- traffic on the public network in order to detect a brute-force
works were stored as separate tables. In addition, the system threat. The training data set for the outbound traffic is shown
monitored all of the network traffic in the BAS and recorded on the left side of Figure 12. The different colors represent
all IP addresses and ports in real-time. At the same time, all the various IP addresses that were contacted by the BAIDS
the traffic that communicated directly with the BAIDS device device. The traffic behavior with the different IP addresses
on the public network was detected and stored. varied; some of the network communications between the
BAIDS and public network devices had scattered behavior
and others were consistent over time. The BAS traffic, used to
evaluate the cyber-physical attack, showed consistent behav-
ior for four devices as shown on the right side of Figure 12.
The training data was used by the ART algorithm to learn
normal network traffic activity. The gained knowledge was
then applied to cyber threat situations with the intent to detect
unwanted attacks accurately.

FIGURE 11. Activity that originated from unknown IP addresses on the


BAS network. Each color represents a different IP address.
FIGURE 12. The monitored traffic on the public and private networks
The incoming traffic observed from unknown IP addresses, were evaluated using the traffic frequency, average packet size, and
shown in Figure 11, revealed the frequency at which each IP average checksum attributes. The different colored markers describe the
behavior for each device. The public network data defines the
address was contacted over a 15 minute period on the private communication between various devices and BAIDS. The private network
network. The different color bars in Figure 11 represent the data defines the behavior for four different devices.
particular IP address that sent a message. The frequency for Cyber attacks were detected accurately by the ART algo-
each of the IP addresses remained below 20 hits per minute rithm in the public and private networks. The ART algo-
from hour 19:21 to 19:25 and then went up to above 50 hits rithm trained on each of the normal data set using a free
per minute at 19:26. The lowest communication frequency parameter ρ value equal to 0.92. It identified the brute force
for the IP address was around 10 hits per minute and the attack by labeling outbound and inbound traffic data as nor-
highest was above 80. The large peaks could be a potential mal or abnormal. For example, the ART algorithm flagged
indication of malicious cyber behavior. Hackers could be abnormal behavior in the outbound data set as shown in
attempting to alter the state of the system by connecting with Figure 13. In this case, the traffic data had a higher frequency,
the particular device at a higher rate than usual. However, larger than the normal packet size, and a normal checksum
more detailed analysis would be required to make these kind for a two minute period which occurred from hour 16:57
of determinations. The present work used the ART neural to 16:59. This matched exactly with the simulated hacking
network to analyze network messages and detect potentially event. The successful identification of the brute-force attack
malicious activity. was achieved with only one false positive result. The same
approach was used to detect the cyber-physical attack on the
C. INTRUSION DETECTION private network.
The cyber security capabilities of the BAIDS interconnection The cyber-physical attack impacted three HVAC control
device where tested against artificially induced brute-force devices in the private network. The malicious code altered the

VOLUME 5, 2017 11071


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

power with the PV electrical output without disturbing the


thermal comfort levels inside a simulated thermal zone of a
building. At the same time, one second monitoring of network
traffic on the private and public networks was implemented
to perform passive intrusion detection. The network data was
collected and made available through a HMI database inter-
face. The network data was then used by an ART algorithm
to identify anomalies and potentially malicious activity.
Future work will expand on this work to upgrade the
controls and provide active intrusion detection capabilities.
The goal is to improve interoperability between the utility
and commercial building operators. Improved interoperabil-
ity will allow for DR to occur without imposing discomfort
to the building occupants. For example, utilities could work
FIGURE 13. The intrusion detection system was able to detect the with building operators to connect with a large amount of
brute-force attack using the Adaptive Resonance Theory neural network.
It quickly identified abnormal behavior.
dispatchable loads, receive feedback to adjust controls, and
target the best loads for the utility and the users at any given
moment in time. This most be done with cyber security in
mind.
Real-time cyber threat analysis using learning based algo-
rithms has the potential to monitor, evaluate, and flag traffic
at specific ports and IP addresses. The current work tested a
single ANN algorithm. The next step is to perform detailed
experiments that test the abilities of various algorithms to
identify threats on BAS. The experiment will evaluate the
algorithms ability to analyze network traffic anomalies and
state changes in the cyber-physical system.
ACKNOWLEDGMENT
Sandia National Laboratories is a multimission laboratory
managed and operated by National Technology and Engineer-
ing Solutions of Sandia, LLC., a wholly owned subsidiary
FIGURE 14. The intrusion detection system was able to detect the of Honeywell International, Inc., for the U.S. Department
cyber-physical attack. The Adaptive Resonance Theory neural network of Energy’s National Nuclear Security Administration under
identified abnormal behavior by evaluating traffic that originated from
various devices. The upside down triangles show the abnormal traffic that contract DE-NA0003525.
came from the hackers computer.
REFERENCES
[1] P. Palensky and D. Dietrich, ‘‘Demand side management: Demand
states of numerous systems connected to the HVAC control response, intelligent energy systems, and smart loads,’’ IEEE Trans. Ind.
system. Unfortunately, the BAIDS device was not able to Informat., vol. 7, no. 3, pp. 381–388, Aug. 2011.
[2] P. Siano, ‘‘Demand response and smart grids—A survey,’’ Renew. Sustain.
stop it from occurring, nor was it able to react to the attack Energy Rev., vol. 30, pp. 461–478, Feb. 2014.
and return the states back to normal. However, it was able to [3] S. Kiliccote, D. Olsen, M. D. Sohn, and M. A. Piette, ‘‘Characterization of
accurately detect the event and identify the devices impacted. demand response in the commercial, industrial, and residential sectors in
the United States,’’ Wiley Interdisciplinary Rev., Energy Environ., vol. 5,
The ART algorithm detected abnormal behavior from the no. 3, pp. 288–304, May 2016.
adversary’s device as shown in Figure 14 by evaluating the [4] N. M. Watson, J. Page, S. Kiliccote, and M. A. Piette, ‘‘Fast auto-
source IP information. The destination IP address results pro- mated demand response to enable the integration of renewable resources,’’
Lawrence Berkeley National Lab., Berkeley, CA, USA, Tech. Rep. LBNL-
duced by the ART algorithm identified each of the impacted 5555E, Jun. 2012.
devices. The algorithm had a 100% true positive rate and [5] S. Clements and H. Kirkham, ‘‘Cyber-security considerations for the smart
flagged only two false alarms. grid,’’ in Proc. IEEE PES Gen. Meet., Jul. 2010, pp. 1–5.
[6] Y. Yan, Y. Qian, H. Sharif, and D. Tipper, ‘‘A survey on smart
grid communication infrastructures: Motivations, requirements and chal-
V. CONCLUSION lenges,’’ IEEE Commun. Surveys Tuts., vol. 15, no. 1, pp. 5–20,
Direct control of HVAC devices from a centralized loca- Feb. 2013.
[7] Y. Yan, Y. Qian, H. Sharif, and D. Tipper, ‘‘A survey on cyber security for
tion for DR can be achieved using an intelligent and secure smart grid communications,’’ IEEE Commun. Surveys Tuts., vol. 14, no. 4,
interconnection device. The BAIDS successfully bridged two pp. 998–1010, Oct. 2012.
networks to provide direct communication between an HVAC [8] W. Wang and Z. Lu, ‘‘Cyber security in the Smart Grid: Survey and
challenges,’’ Comput. Netw., vol. 57, no. 5, pp. 1344–1371, Apr. 2013.
control devices and a solar PV array. The hardware-in-the- [Online]. Available: http://www.sciencedirect.com/science/article/
loop experiment successfully synchronized the fan motor pii/S1389128613000042

11072 VOLUME 5, 2017


C. B. Jones, C. Carter: Trusted Interconnections Between a Centralized Controller and Commercial Building HVAC Systems

[9] M. A. Piette, O. Sezgen, D. S. Watson, N. Motegi, C. Shockman, and [27] B. Mukherjee, L. T. Heberlein, and K. N. Levitt, ‘‘Network intrusion
L. ten Hope, ‘‘Development and evaluation of fully automated demand detection,’’ IEEE Netw., vol. 8, no. 3, pp. 26–41, May 1994.
response in large facilities,’’ Lawrence Berkeley Nat. Lab., Berkeley, [28] G. A. Carpenter and S. Grossberg, ‘‘A massively parallel architecture for a
CA, USA, Tech. Rep. LBNL-55085, Mar. 2004. [Online]. Available: self-organizing neural pattern recognition machine,’’ Comput. Vis. Graph.
http://escholarship.org/uc/item/4r45b9zt Image Process., vol. 37, no. 1, pp. 54–115, 1987.
[10] G. Ghatikar and R. Bienert, ‘‘Smart grid standards and systems interoper- [29] G. A. Carpenter, S. Grossberg, and D. B. Rosen, ‘‘Fuzzy ART: Fast
ability: A precedent with OpenADR,’’ in Proc. Grid Interop Forum, 2011, stable learning and categorization of analog patterns by an adap-
pp. 1–7. tive resonance system,’’ Neural Netw., vol. 4, no. 6, pp. 759–771,
[11] C. McParland, ‘‘OpenADR open source toolkit: Developing open source 1991.
software for the Smart Grid,’’ in Proc. IEEE Power Energy Soc. Gen. Meet., [30] M. Georgiopoulos, I. Dagher, G. L. Heileman, and G. Bebis, ‘‘Proper-
Jul. 2011, pp. 1–7. ties of learning of a fuzzy ART variant,’’ Neural Netw., vol. 12, no. 6,
[12] M. A. Piette, G. Ghatikar, S. Kiliccote, D. Watson, E. Koch, and pp. 837–850, Jul. 1999.
D. Hennage, ‘‘Design and operation of an open, interoperable automated
demand response infrastructure for commercial buildings,’’ J. Comput.
Inf. Sci. Eng., vol. 9, no. 2, p. 021004, May 2009. [Online]. Available:
http://dx.doi.org/10.1115/1.3130788
[13] N. Research. (2016). How To Protect Corporate Building Networks
From Cyber Attacks. [Online]. Available: http://www.forbes.com/sites/
pikeresearch/2016/09/13/cybersecurity-and-intelligent-buildings/
[14] M. Peacock and M. N. Johnstone, ‘‘An analysis of security issues in build- C. BIRK JONES was born in Santa Fe,
ing automation systems,’’ in Proc. 12th Austral. Inf. Secur. Manage. Conf., NM, USA. He received the B.S. degree in civil and
Jan. 2014, pp. 100–104. [Online]. Available: http://ro.ecu.edu.au/ism/170 environmental engineering from the University of
[15] C. Neilson, ‘‘Securing a control systems network,’’ BACnet Today Suppl. California at Davis, Davis, CA, USA, in 2004, and
ASHRAE J., vol. 55, no. 11, pp. b18–b22, Nov. 2013. the M.S. degree in construction engineering and
[16] D. G. Holmberg, ‘‘Secure messaging in BACnet,’’ BACnet Today Suppl. the Ph.D. degree in mechanical engineering from
ASHRAE J., vol. 47, no. 11, pp. B23–B26, Nov. 2005. the University of New Mexico, Albuquerque, NM,
[17] W. Granzer, F. Praus, and W. Kastner, ‘‘Security in building automation in 2009 and 2015, respectively.
systems,’’ IEEE Trans. Ind. Electron., vol. 57, no. 11, pp. 3622–3630, He was a Structural Engineer involved in
Nov. 2010. designing trusses and inspected bridges. He spent
[18] W. Lee, S. J. Stolfo, and K. W. Mok, ‘‘A data mining framework for
some time in construction retrofitting large and small commercial building.
building intrusion detection models,’’ in Proc. IEEE Symp. Secur. Privacy,
He was also a Mechanical Engineer involved in assessing building perfor-
May 1999, pp. 120–132.
[19] Z. Pan, S. Hariri, and Y. Al-Nashif, ‘‘Anomaly based intrusion detection mance using models and advanced learning algorithms to identify faults. He
for Building Automation and Control networks,’’ in Proc. IEEE/ACS 11th is currently a Senior Member of the Technical Staff with the Sandia National
Int. Conf. Comput. Syst. Appl. (AICCSA), Nov. 2014, pp. 72–77. Laboratories, Photovoltaics and Distributed System Integration Department.
[20] M. Johnstone, M. Peacock, and J. D. Hartog, ‘‘Timing attack His research interests include photovoltaic system reliability, building HVAC
detection on BACnet via a machine learning approach,’’ in Proc. fault detection, demand response, and renewable energy integration.
Austral. Inf. Secur. Manage. Conf., Jan. 2015. [Online]. Available:
http://ro.ecu.edu.au/ism/181
[21] J. Tonejc, S. Guttes, A. Kobekova, and J. Kaur, ‘‘Machine learning methods
for anomaly detection in BACnet networks,’’ J. Univ. Comput. Sci., vol. 22,
no. 9, pp. 1203–1224, 2016.
[22] A. Mammoli, H. Barsun, R. Burnett, J. Hawkins, and J. Simmins, ‘‘Using
high-speed demand response of building HVAC systems to smooth cloud- CEDRIC CARTER, Jr., was born and raised in
driven intermittency of distributed solar photovoltaic generation,’’ in Proc. Valdosta, GA, USA, and later moved to Durham,
IEEE PES Transmiss. Distrib. Conf. Expo. (T D), May 2012, pp. 1–10. NC, USA, at the age of 15. He received the B.S.
[23] W. Anwaar and A. Shah, ‘‘Energy efficient computing: A comparison of and M.S. degrees in computer science (informa-
raspberry PI with modern devices,’’ Int. J. Comput. Inf. Technol., vol. 4,
tion assurance) from North Carolina A&T State
no. 2, pp. 410–413, Mar. 2015.
University (NCAT).
[24] P. Paethong, M. Sato, and M. Namiki, ‘‘Low-power distributed NoSQL
database for IoT middleware,’’ in Proc. 15th ICT Int. Student Project While at NCAT, he conducted research at
Conf. (ICT-ISPC), May 2016, pp. 158–161. Indiana University Bloomington in parallel com-
[25] H. G. Jamuna and S. Hugar, ‘‘Smarter grid embedded in an Internet of puting with multi-core machines. He also interned
Things,’’ Int. J. Eng. Technol., vol. 2, no. 3, p. 1, Jun. 2016. at Shell Oil Company, Houston, TX, USA, and
[26] C. B. Jones et al., ‘‘Wondering what to blame? Turn PV performance Cisco Systems, Raleigh, NC. He is currently with the Sandia National
assessments into maintenance action items through the deployment of Laboratories, Resilient Control Systems Department. His research interests
learning algorithms embedded in a Raspberry Pi device,’’ in Proc. IEEE include cyber-physical assessments and threat deterrent development. He
43rd Photovoltaic Specialists Conf. (PVSC), Jun. 2016, pp. 0261–0266. was a recipient of the National Science Foundation Scholarship for Service.

VOLUME 5, 2017 11073