Unleashing the Power of Big Data and

Analytics for the Utility Industry

WHITE PAPER
Sponsored by: HP

J i l l F e bl o wi t z
S ep tem b er 2 01 2
www.idc-ei.com

IDC OPINION
The proliferation of smart meters and sensors along with new channels
for customer interaction have provided North American utilities with
F.508.988.7881

unprecedented access to data from the grid and from their customers.
The challenge that utilities now face is how to take advantage of that
data to create value for the business. Fortunately, advances in
technology have made it possible to quickly gain intelligence from
large volumes and a variety of data types. Areas where Big Data and
P.508.935.4400

analytics can provide value for operations and customer relations are
emerging, along with some best practices in assembling an analytics
approach. Big Data and analytics technologies describe a new
generation of technologies and architectures designed to economically
extract value from very large volumes of a wide variety of data by
Global Headquarters: 5 Speen Street Framingham, MA 01701 USA

enabling high-velocity capture, discovery, and/or analysis. Key

takeaways include:

Utilities recognize the significance of having an analytics strategy

and view investment in business intelligence and analytics as
important.

Utilities in North America are considering the use of Big Data and
analytics primarily for analysis of operations data.

The current conversation is about using all the attributes of Big

Data and analytics to support grid and network reliability. In the
future, the conversation may extend to customer engagement and
management.

Applications of Big Data and analytics in utilities that show

promise are predictive asset management, capital investment
planning, fraud and theft detection, sentiment analysis, and social
media scans.

To gain full advantage from existing data sources, utilities need to

formulate a Big Data strategy that includes evaluation of decision
makers' requirements, decision processes, existing and new technology,
and the availability and quality of data.

September 2012, IDC Energy Insights #EI236948

SITUATION OVERVIEW

The Utilities Environment

Utilities in North America are faced with declining revenue. For

example, the U.S. Department of Energy's Energy Information
Administration expects total electricity consumption in the United
States to fall by 1.3% in 2012 and grow by 1.3% in 2013, whereas in
the past, consumption grew at a fairly steady rate of 23%. Sales are
expected to be down by 3.2% this year. The reduction in revenue is in
part due to the slow recovery from the recession as commercial and
industrial consumption is down. The reduction may also be attributed
to the influence of low gas prices on revenue for gas and electric
utilities. While these may be short-term situations, greater energy
efficiency achieved with equipment standards, energy efficiency
measures, and changes in behavior are expected to impact
consumption, thus revenue, going forward. It is no longer a case of
balancing supply and demand but optimizing supply- and demand-side
resources to meet demand.

Aging infrastructure, unusual weather patterns, and cybersecurity and

data privacy threats are challenging the utility's ability to maintain
reliability of service. The industry has known for the past decade that
aging delivery infrastructure is in need of new investment. For
example, according to the American Water Works Association
(AWWA), much of the drinking water infrastructure in the United
States, comprising more than 1 million miles of pipes, is nearing the
end of its useful life and approaching the age at which it needs to be
replaced. This aging infrastructure is leaky, is prone to breaks, and is
resulting in degraded water service and increased water service
disruptions. Utilities need to understand the operational thresholds that
can safely be achieved by older infrastructure to avoid equipment
failure. North America has also experienced an uptick in flooding,
fires, and storms that have left utility customers without access to
energy and water. Regulators are demanding that utilities be better
prepared to restore services quickly in the event of extreme weather or
other natural disasters. Today, long-term or seasonal weather models
are insufficient; unusual weather patterns require better near-term
forecasting capability. A new breed of viruses attacking process
control systems has raised awareness of security at utilities. Utilities
need to continuously scan for potential attacks and stay on top of the
next generation of security risks.

The past five years have seen the installation of new "smart"
technologies by electric, gas, and water utilities. Smart grid
investments are made to improve grid reliability, support distributed
and renewable energy, reduce operations and maintenance costs,
increase energy delivery efficiency, improve system security, and
enable energy efficiency and demand response. A growing movement

Page 2 #EI236948 2012 IDC Energy Insights

toward the deployment of smart grid technologies was bolstered by
stimulus efforts such as the American Recovery and Reinvestment Act
(ARRA). IDC Energy Insights estimates that the installation of smart
meters will reach a peak in 2012, and by 2015, there will be over 90
million smart meters installed in North America. ARRA funding has
also introduced new technologies such as synchrophasors to the grid.
Now that the ARRA funding has been allocated and largely spent,
utilities are building out IT capabilities as the focus shifts to using the
data to justify return on investment. With new data available and
existing data more accessible, utilities are asking, "Now that we have
all this data, what do we do with it?"

Today's consumers expect a rich "customer experience," based on their

experience with retailers, banks, and other service providers. They
want to have their needs quickly understood and met and want to
interact through multiple devices and social media. Customers are
already coming to expect this same level of care and insight from their
utilities. For example, customers may want to be notified of an outage
on their phone and have that notice followed up with an update on
expected restoration. Customers may use tools provided by the utility
on a customer portal to develop a budget plan for their usage and have
notices sent to their phone as consumption nears the target budget for
the month. They want to receive information only about those
efficiency programs or demand response programs where they meet
eligibility requirements. They may want utilities to respond to
questions posed to them in their tweets. The ability to get rate
increases approved by regulators depends on demonstrating customer
satisfaction. Customer care requires continuous attention on the part of
the utility, yet care must be cost effective.

New and disruptive technologies are gaining ground. In response to

renewable portfolio standards and emissions regulations, utilities are
diversifying their portfolios and exploring energy storage. Residential
and commercial/industrial customers continue to invest in distributed
energy resources such as solar and wind, despite some setbacks in the
industry. The expectation is that 120,000 plug-in electric vehicles
(EVs) will be sold in North America in 2012. Utilities are looking
toward demand response technologies to shift consumption to lower-
cost periods. Commercial and industrial customers are evaluating the
use of smart building technology to lower costs and reduce their
carbon footprint. As penetration of new technologies grows, there will
be an impact on operations, commercial relationships, and distribution
infrastructure. Resources like distributed wind and solar come on and
off the grid. EV charging will be done at home, at work, at the store,
and at fueling stations. Utilities will need to prepare for a more
dynamic distribution grid through informed planning based on
simulations. In the future, utilities will need to respond quickly to
supply and demand coming on and off the grid.

2012 IDC Energy Insights #EI236948 Page 3

The Attributes of Big Data and Analytics

Big Data and analytics technologies describe a new generation of

technologies and architectures designed to economically extract value
from very large volumes of a wide variety of data (structured and
unstructured) by enabling high-velocity capture, discovery, and/or
analysis. Big Data and analytics include infrastructure (servers, storage,
clustering software, networks, etc.), data organization and
management (software that cleanses, normalizes, tags, and integrates
data), analytics, business intelligence and discovery (software for ad
hoc discovery and deep analytics); real-time analysis; and automated,
rules-based transactional decision making), and decision support
(collaboration, scenario evaluation, risk management, and decision
capture and retention). Software associated with Big Data and
analytics includes graph or network analysis, text mining or sentiment
analysis, streaming data monitoring and analysis, noSQL databases,
relational databases, text mining, and graph or network analysis.

Here's where volume, variety, velocity, and value (discussed in the

sections that follow) line up with the conditions at utilities.

Volume
Smart grid and smart network devices are generating more data than
ever before. For example, one distribution company with 2 million
meters estimates that it processes 35GB of data a day based on 15-
minute interval data collected four times a day from each smart meter.
Utilities are investing in powerline sensors, advanced remote terminal
units, and intelligent electronic devices at the edges of the network. It
is not just smart grid and network investments that are generating data.
Any facility that has SCADA systems or data historians has had access
to control data. In aggregate, these devices and control systems throw
off an even greater amount of data than smart meters. If the utility
were to make all this data available and accessible to process, volumes
would meet the Big Data volume thresholds.

Variety
The bulk of the data that utilities are generating smart meter and
grid-generated data is structured data. It is time series data recorded
at uniform time intervals ranging from seconds to hours. Applications
at the utilities, such as customer care and billing, work and asset
management, energy trading and risk management, enterprise resource
planning, project management, geospatial information systems, and so
forth, are also sources of structured data that are valuable for analysis.
External data services, such as weather, market prices, and futures, can
be helpful as well. Traditional sources of unstructured data at the
utility include emails from utility customers, recorded contact center
calls, contracts with suppliers in PDF format, engineering drawings,
specifications, Web traffic, and interactions via social media.

Page 4 #EI236948 2012 IDC Energy Insights

Velocity
Today, velocity can be associated with discovery/analysis on
operational data from control system and equipment or grid sensors in
near real time. The idea is to allow the utility to monitor conditions,
send fault or tamper alarms, or predict potential incidents. The interval
consumption data collected by the smart meter is collected only
periodically. However, "last gasp" data from the meter, especially in
outage situations, is returned to the headend system in near real time.
To support the dynamic distribution network of the future, high-
velocity discovery and/or analysis based on interactions between
utilities and smart homes and buildings will be required. Google and
Facebook have been strong proponents of Big Data and analytics by
monitoring Web traffic and social media to push advertisements. The
same techniques could be applied by utilities to customer data.

Value
Value is the fourth attribute of Big Data and analytics and must be
present to justify investment in the infrastructure, data organization
and management, discovery, and decision support. In general, Big
Data and analytics provide the ability to filter through large amounts
of data, unlock insights available through new data sources, and
leverage and optimize untapped data. There are potential problems and
opportunities that need time and resources to address. For example,
there may be a specialized transformer that is likely to fail, but the
transformer needed is not available off the shelf and requires months
from order to delivery. If these kinds of problems can be identified
quickly, then a company can avoid potential problems or get ahead of
its competitors in the marketplace.

FUTURE OUTLOOK
Utilities look for information technology investments to provide value
in improving operations and customer satisfaction. More specifically
this means:

Protecting revenue (customer)

Improving customer engagement (customer)

Maintaining reliability and securing the grid (operational)

Reducing the cost of operations (operational)

Complying with environmental regulations (operational and

customer)

2012 IDC Energy Insights #EI236948 Page 5

Business Intelligence and Analytics
at Utilities

Utilities have come to recognize the importance of having an analytics

strategy and view investment in business intelligence and analytics as
important. According to IDC's 2012 Global Technology and Industry
Research Organization IT Survey, 30.2% of utility respondents have a
stated analytics strategy. Utilities see the importance of investment in
this area. Fifty-four percent of United Statesbased utility respondents
rate investment as important or very important; 87.4% have invested in
business intelligence and business analytics; and 36.9% plan to invest
in a new solution or make major enhancements to existing solutions in
the coming year. The top motivations for investment are cost control
or reduction, financial reporting and analysis, optimization of
operations, and workforce optimization.

Utilities are using business analytics to support:

Customer satisfaction: Target marketing for energy efficiency

and demand response, evaluation of the effectiveness of demand
response and energy efficiency programs, product development of
time-based rates or demand response incentives, analytical tools
provided to customers to help them reduce their energy bills,
understanding how customers prefer to receive communications
about outages

Improving operations: Pinpointing the location of outages for

workforce deployment, scanning for potential security breaches,
detecting fraud and theft and identifying unbilled accounts,
performing analysis of faults or voltage irregularities

Probably the most immediate savings are being realized with fraud and
theft detection. Utilities are also starting to realize savings through
analytics used to prioritize capital investments in transformers and
other equipment.

Still, a business intelligence/analytics strategy is not the same as

making a commitment to Big Data and analytics. Big Data and
analytics could be a part of the strategy but will involve investments
different from business intelligence/analytics in technology, process,
data, and personnel. For example, companies will likely need to hire
data scientists or invest in streaming analytics. There is still some
confusion over what Big Data and analytics entail. Close to half of
utility respondents were not familiar with the term. Whether or not
utilities will choose to implement Big Data and analytics will depend
on the answers to questions such as:

What business objective (s) do we want to achieve?

What decisions do we want to make using the technology?

Page 6 #EI236948 2012 IDC Energy Insights

 How quickly do we need to make decisions?

Are decisions to be automated or involve human interaction?

How much data is required to arrive at an answer?

How varied is the data involved?

Can we modify existing applications and infrastructure to get the

answers we want?

Does it make sense to build a common analytics infrastructure to

handle all our analytics needs?

Big Data Use Cases for Utilities

Today, utilities in the United States are considering the use of Big Data
and analytics primarily for analysis of operations data (see Figure 1).

FIGURE 1

Drivers of Big Data in the Utility Industry

Q. What are your organization's drivers for using Big Data technologies and approaches?

Analysis of operations-related data

Analysis of online customer

behaviorrelated data
Analysis of transactional data f rom
sales systems

Service innovation

Analysis of machine or device data

Nonanalytic workloads

0 5 10 15 20 25 30 35 40
(% of respondents)

n = 124 (utilities)
Source: IDC's Global Technology and Industry Research Organization IT Survey, 2012

The current conversation is about using all the attributes of Big Data and
analytics to support grid and network reliability by analyzing structured
time series data from SCADA and control systems, sensors, and smart
meters, and that is where most investment is just now being made.

2012 IDC Energy Insights #EI236948 Page 7

Probably the most publicized use of Big Data and analytics is by the
independent system operators (ISOs) that must handle large quantities of
data every day just to keep the power markets functioning and
transmission functioning. For example, PJM has contracted with a public
power authority to process synchrophasor data to improve transmission
grid management and will use Hadoop to manage the data.

In fact, data management in the sense of being able to quickly access

data is the first thing that utilities think about when presented with the
concept of Big Data and analytics. Twenty-five percent think of this
term as referring to a very large amount of data, while 11.8% thought
real-time streaming data, 12.8% said data of multiple types, and 2.6%
said high-performance computing. It is likely that the initial focus will
be on accessing data more quickly, but the value comes in executing
on the results of the analytics. So business users should be able to run
their own analytics and use the data for decision making or produce an
automated action.

Applications of Big Data and analytics that are showing promise in

utilities include:

Fraud and detection. One transmission and distribution company is

comparing customer load profiles with data from smart meters and
line transformer loads that are separately metered to identify leakage.

Sentiment analysis. Analyzing the voice patterns of the customer

during a call to indicate the level of customer satisfaction is just
starting to be used in other industries and may help CSRs reorient
the approach to the call. Sentiment analysis may also be a rich
source of data for marketing.

Predictive asset management. Predictive analytics are being used

by a few utilities to forecast potential performance or equipment
failures by studying the load on each asset along the distribution
network and the rated capacities of devices (switch, reclosure,
breaker, transformer, etc.). Detection of water leakages can also
help predict potential water pipeline failures.

Capital investment planning. One ISO with the ultimate goal of

informing capital investment strategy and improving grid
reliability has recently invested in in-memory analytics to gain
access to timely information about grid and asset status.

Social media scans. There may be value to utilities in analyzing

unstructured data found in social media or emails. Some utilities
are already doing scans of the social networks to understand how
their customers are reacting to events such as unplanned outages.

As the distribution network sees the addition of customer-owned

distributed resources, energy storage, electric vehicles, and automated

Page 8 #EI236948 2012 IDC Energy Insights

demand response, the attributes of Big Data and analytics velocity,
volume, variety, and value will be called upon to support the virtual
power plant (VPP). Virtual power plants are a way to aggregate
distributed energy resources and demand response as if they were a
single power plant. The VPP can be used to balance the grid or enhance
revenue. Some utilities are looking at the potential for developing a
distribution marketplace, similar to an ISO, to help manage the high
penetration of solar in some neighborhoods in order to take strain off
feeders when abundant solar power is available. Big Data and analytics
will be needed to support this type of market.

Challenges for Utilities

Probably the biggest barrier to the adoption of Big Data and analytics
at utilities is the lack of awareness. Other barriers are:

Concern for privacy. Smart meter installations have generated

backlash from customers about safety (RF frequency) and concern
about privacy of data. While utilities will likely invest in analytics to
do social media scans to gauge sentiment, because of privacy
concerns, they are not likely trawl for individual customer preferences
on social media sites.

Lack of competition. U.S. utilities are still largely regulated, with

the customer not having much supplier choice, compared with the
Western European utilities market, which is deregulated and
competitive. Therefore, utilities in the United States are less
motivated to make investments in customer-related analytics.

Need to understand how Big Data works with real-time

operational systems. Utilities have already made a considerable
investment in real-time systems to run the transmission grid, and
while there is some movement to apply IT to operational data, the
business value is yet to be determined, especially around where
speed to answer is essential.

Uncertainty about costs and requirements. Utilities are

uncertain about the costs and requirements for building a Big Data
and analytics infrastructure. There is also uncertainty about how
Big Data and analytics will fit in with legacy systems.

Overview of HP Big Data Offerings

HP offers a wide range of information management and analytics

(IM&A) capabilities drawn from its enterprise services, software
products, and cloud and security platforms to address Big Data
challenges and opportunities in the utility industry.

2012 IDC Energy Insights #EI236948 Page 9

HP IM&A services include information strategy and organization,
information management and architecture, business analytics and
information delivery, and social intelligence. HP provides these
consulting services around its own Big Data software assets, Autonomy
and Vertica, and third-party software assets, in particular SAP HANA
and Microsoft SharePoint/BI platform. HP supports this with a variety of
software and solution delivery models including on-premise installation,
hosted, software as a service (SaaS), cloud computing, and multitenant
SaaS (cloud deployment).

HP focuses its capabilities to enable utilities to proactively manage

information-related business risk, enhance customer experiences, and
optimize business performance to maintain reliability, reduce the costs
of operation, and protect revenue.

HP's Autonomy software asset helps utilities develop connected

intelligence from structured and unstructured data for actionable
decisions that improve business performance.

HP's Vertica analytics database delivers scalable performance on Big

Data queries enabling real-time decision making to be embedded in
utility processes in order to optimize business performance.

Vertica and Autonomy deliver a powerful combination for real-time

analytics and decision making using structured and unstructured data
across the enterprise.

Strengths and Challenges

HP offers a product and services portfolio that is consistent with what
IDC Energy Insights expects from a market-leading technology
provider to the utility industries. In addition to the software and
services capabilities noted previously, HP provides the following
infrastructure component to enable Big Data application deployments:

Cloud computing. HP can offer clients a variety of enterprise-

grade cloud computing solutions, including public, hybrid, and
private. The ability to offer private cloud to utilities is considered a
strength for a number of reasons, not the least of which is a tighter
security model.

Security. HP offers clients a security strategy through the HP

Security Framework, designed to offer end-to-end information
security plans and execution road maps. Because of the sensitive
nature of customer data and the requirements and penalties imposed
by the regulators, security is top of mind for industry IT executives.

Page 10 #EI236948 2012 IDC Energy Insights

 Mobility. HP's approach to enabling enterprise mobility is suited for
organizations that wish to reach their constituents across multiple
networks and devices by delivering applications, content, and
services in a scalable, secure, and reliable way. This approach
leverages HP's global applications services capabilities to provide the
architecture, systems engineering, development, and support services.
Combined, they help an organization simplify its applications and
extend them where necessary as well as build mobile business-to-
business, business-to-consumer, and business-to-employee
applications. This approach also leverages HP's service-oriented
architecturebased integration architecture and is enabled by
development and security frameworks that help create
componentized building blocks from monolithic legacy applications
to develop and deploy mobile applications.

HP faces several unique market challenges as well as many of the

same market challenges as other enterprise vendors servicing the Big
Data marketplace:

Demonstrating leadership in the transition from information

management to Big Data. HP has made inroads into utilities with
Exastream, a platform for creating, managing, and delivering
communications to the utility customer. HP had a very compelling
case for the use of OpenView in the management of smart meter
data but was not able to gain significant traction in utilities. HP
must more effectively demonstrate leading capabilities at the
component level with high-profile Autonomy and Vertica wins as
well as highlight its ability to deliver end-to-end Big Data software,
services, and hardware with lighthouse and marquee clients.

Offering competitive, differentiated solutions portfolios.

Utilities are not early adopters of Big Data and analytics, but they
are now focused on analytics to make use of smart grid data. Many
of the major professional service firms are offering to help with a
business intelligence/analytics strategy. HP's competitors are also
focused on expanding their solution portfolios on the Big Data
trend in terms of the breadth and depth of product capabilities, IT
infrastructure, and professional services. HP must leverage its
industry position and demonstrate the value of a Big Data
relationship with HP to gain Big Data market share.

Channel network management. HP's channel strength is also a

weakness. Successful execution of HP's Big Data strategy to
broaden its portfolio will require that HP reinforce its position in
the enterprise space without alienating the channel that has been so
beneficial to the organization.

2012 IDC Energy Insights #EI236948 Page 11

 Focus on security. With the introduction of new viruses that can
invade process control systems, and with the addition of IP-
addressable devices on the grid, system vigilance has increased.
Regulators now require utilities to comply with security and
reliability standards for transmission networks and generation
assets, and this requirement may soon extend to protecting
distribution assets and networks as well. Concern with privacy of
customer data, especially around the implementation of smart
meters, requires more attention to the security area.

PARTING THOUGHTS
Utilities looking to improve their capabilities in Big Data and analytics
should consider the following:

Recognize the value of untapped data assets in supporting fact-

based decisions by generation, transmission, distribution and
customer operations, marketing, energy efficiency, and demand
response.

Recognize the implications of operating without critical

information and build use cases to address the challenges.

Conduct a gap analysis to determine what new technology and

staff investments are required.

Formulate a Big Data strategy that includes an evaluation of

decision makers' requirements, decision processes, existing and
new technology, and availability and quality of data.

Consider cloud services and a shared services model with other

utilities to reduce costs. This strategy will allow you to experiment
prior to investment. Utilities with limited resources may want to
consider this option in order to keep costs low.

Copyright Notice

Copyright 2012 IDC Energy Insights. Reproduction without written

permission is completely forbidden. External Publication of IDC
Energy Insights Information and Data: Any IDC Energy Insights
information that is to be used in advertising, press releases, or
promotional materials requires prior written approval from the
appropriate IDC Energy Insights Vice President. A draft of the
proposed document should accompany any such request. IDC Energy
Insights reserves the right to deny approval of external usage for any
reason.
This document was reprinted by HP with permission from IDC Energy
Insights.

Page 12 #EI236948 2012 IDC Energy Insights