Energy Efficiency
DOI 10.1007/s12053-010-9082-6
Measuring the savings from energy efficiency policies:
a step beyond program evaluation
Marvin J. Horowitz
Received: 17 March 2009 / Accepted: 19 March 2010
# Springer Science+Business Media B.V. 2010
Abstract Recent worldwide events are producing
growing agreement that rapid, large-scale energy effi-
ciency is a necessary, feasible, and economical way of
reducing the need for new energy supplies. Yet, many
obstacles must be overcome for commitments to energy
efficiency to be successful, perhaps none more impor-
tant than measuring policy-related energy savings. To
address this topic, this paper provides several examples
of how readily available MWH sales data can be
employed to monitor and verify energy efficiency
policies and detect changes in trends at the state and
utility levels. The analyses used for this kind of savings
verification are collectively referred to in this paper
as counterfactual simulation. The goal of this paper is
to demonstrate the practicality and value in using
standardized research designs and econometric models
to inform energy efficiency policymakers and the
public as to the benefits of energy efficiency policies.
Keywords Energy savings measurement .
Energy efficiency policy . Counterfactual simulation .
Econometric models
Introduction
It has long been recognized that it is difficult to
measure the energy savings that come from energy
efficiency measures. Energy efficiency is unlike any
M. J. Horowitz (*)
Demand Research LLC,
Fairfax, VA, USA
e-mail: mhorowitz@demandresearch.net
other energy resource. Wind, solar, coal, water, oil,
atoms, corn, or what have you, are physical objects,
as is the energy that comes from their conversion.
However, energy efficiency is not a physical object;
for any desired level of light, heat, or other service
that requires energy as an input, improved energy
efficiency means that less energy and fewer
resources are demanded. This creates a dilemma.
Simple counting by arithmetic cannot measure the
volume of fuel and energy that would have been
demanded but was not. Measurement can only be
done indirectly by statistical inference or what is
sometimes called counterfactual reasoning. Like every
“what if…” question, the answer to how much energy
an energy efficiency measure saves requires a leap of
reasoning.
Uncertainty notwithstanding, recent worldwide
events are producing growing agreement that rapid,
large-scale energy efficiency is a necessary, feasi-
ble, and economical way of reducing the need for
new energy supplies. With greater enthusiasm than
ever before, energy efficiency is viewed by many
as among the first options that ought to be chosen
for attempting to lessen greenhouse gas emissions
and mitigate global climate change. This is clearly
displayed by rising popular interest in national
energy efficiency policies and international climate
agreements. What is more, numerous efforts are
underway via proposed state/regional power and
emissions pacts to make energy efficiency a tradable
commodity equal to physical supplies of fuel or
electricity.
For energy efficiency to succeed as a major energy
resource many obstacles must be overcome, perhaps

none more difficult than producing credible measure-
ments of energy savings. A market for energy
efficiency is no different than a market for any
commodity or financial instrument. Mutually agreeing
to reductions in energy use or greenhouse gases, or
swapping emissions with a cap and trade system, can
only work well when all the engaged parties are
prudent and open about the amounts, terms, and risks
involved. Energy savings contracts cannot be priced
efficiently, and should not be bought or sold, without
such information. The same holds true for voluntary
national and global climate change agreements; pacts
cannot be meaningful unless trusted procedures are in
place for estimating and verifying the quantity of
energy efficiency resources that flow from large-scale
public efforts.
This paper contributes to the advancement of
energy efficiency program evaluation by providing
several examples of how evaluation focused on
aggregate impacts rather than individual program
impacts can overcome several of the most significant
challenges facing energy savings measurement. These
are the challenges of dealing with program overlap,
changes in consumer behavior, net savings, and lack
of rigorous evaluation standardization and reporting.
To avoid confusion, in the context of this paper an
aggregation of programs is referred to as an energy
efficiency policy. The term “portfolio” is eschewed
because it connotes a collection of programs imple-
mented by a single organization. The term energy
efficiency policy, on the other hand, is meant to
encompass the sum of energy efficiency measures,
projects, programs, codes and standards, and other
efforts implemented in a defined geographic territory
by all federal and local public agencies, regulated
utilities, and third parties.
The next section provides brief background infor-
mation on impact evaluation and the reasons for
employing counterfactual simulations for estimating
aggregate savings. This is followed by four related
analyses demonstrating the adaptability of the coun-
terfactual approach to different sampling frames,
different data availability, and different historical
conditions. As a proof of concept intended as the
first step towards large-scale deployment of this
evaluation approach, the analyses in this study focus
on residential MWH sales in many different states and
exploit empirical data and tested models rather than
fictional examples.
Energy Efficiency
Counterfactual simulation: a step beyond
Despite decades of experience with impact evaluation
methods that mimic the laboratory research techni-
ques found in the physical sciences (e.g., pre and post
period monitoring, treatment and control group
comparisons) this subject now requires a fresh look.
Widely acknowledged limitations in existing impact
evaluations include two fundamentally unrealistic
assumptions; first, that most programs can be reliably
evaluated as stand-alone efforts even though many
programs overlap each other, and second, that
consumer preferences and behaviors are fixed and
unresponsive both to market conditions and changes
in public policies. A third limitation is the inability of
these evaluations to produce “net” savings using
revealed, rather than stated intentions, and a fourth
is the lack of standardized technical definitions,
research designs, models, and reporting conventions.
This latter issue is of paramount importance to energy
system planners and policymakers wanting to com-
pare the value of different energy efficiency program
resources across sectors, years, and/or locations.
Initiatives in the United States and Europe address-
ing these shortcomings include the European Com-
mission’s project on evaluation and monitoring for the
end-use directive on energy end-use and energy
services (EMEEES 2008), the US Department of
Energy’s Office of Energy Efficiency and Renewable
Energy publication “International Performance Mea-
surement and Verification Protocol” (IPMVP 2007),
the National Action Plan for Energy Efficiency
Leadership Group’s, “The Model Energy Efficiency
Program Impact Evaluation Guide,” (NAPE 2007),
and the Northeast Energy Efficiency Partnership’s
Evaluation and Monitoring Forum, the mission of
which is described in “Regional EM&V Forum’s
Three Year Plan” (NEEP 2008).
Unfortunately, efforts to improve evaluation have
thus far focused only on enhancing well-known
microdata-based methods, such as those described in
the IPMVP handbook. They fail to embrace the
possibility that with the amount of funding for public
energy efficiency policies now being negotiated at the
state, regional, and national levels, permanent energy
savings in aggregate could very well be much larger
than that which appears from the summation of
savings derived from individual program evaluation
findings. Underestimation using individual program
Energy Efficiency
findings is likely to occur for several reasons. For one,
individual program evaluation research designs are
primarily intended to measure immediate, short-term
savings. In other words, they are intentionally myopic.
For another, these research designs focus narrowly on
program participants, not on society in general, which
means that they produce tunnel vision. As a result,
though they achieve some useful goals, such as aiding in
year-to-year program design and management and
resolving shareholder cost recovery and incentives
claims, they may inadvertently result in the undervalu-
ation of the collective, long-term benefits of concerted,
multi-faceted public policy efforts.
At present, the sum of individual energy programs in
many jurisdictions and countries is expected to yield
energy savings at rates of between 1% and 2% of energy
sales per year for the next decade. This is a far greater
amount of savings than ever before planned, with
cumulative impacts substantial enough to help forestall
the building of new electric generation and transmission
facilities and to slow down the rate of growth in
greenhouse gas emissions. Given the large expectations,
new methods need to be cultivated for measuring energy
savings. One method involves longitudinal statistical
analyses of the energy sales to entire populations of
consumers, not just short-term analyses for selected
consumer segments. If, after controlling for market and
other factors, public programs are found to meet or
exceed their long-term goals, reassurance would be
provided that energy efficiency programs produce
reliable, permanent results. If they are found to fall
short, this too would be of importance to know and
understand.
Time series and pooled cross-section time series or
“panel” econometric models are not only able to
overcome the short-sightedness and narrowness of
conventional program impact evaluations, but are also
able to eliminate the need for supplemental studies
that help transform program energy savings to net
impacts from gross impacts. Typically, a net-to-gross
factor for energy efficiency programs is composed of
a single estimate of free ridership derived via
consumer surveys of stated intentions. Econometric
models cancel the need for these surveys because free
ridership is a market effect driven by such variables as
energy prices, consumer incomes, and capital stock
trends, and properly-specified econometric models
control for these effect. As an added bonus, by their
very nature econometric models also incorporate the
effects of rebound, which reduces energy savings, and
spillover, which increases the energy savings that can
be attributed to public programs. Fundamentally, the
approach relies on the existence of variations in
governmental policies, and market factors, across
time and cross-sections to explain changes in energy
demand trends and for forecasting what would have
occurred in the absence of public policy interventions.
Numerous well-known studies use time series or
panel models to study energy demand, dating at least as
far back as Fisher and Kaysen (1962), Halvorsen
(1975), and Taylor et al. (1984). Recent studies include
those of Bernstein et al. (2003), Loughran and Kulick
(2004), Horowitz (2004), and Metcalf (2008). Of
particular relevance for the present study is Horowitz
(2007) which analyzes changes in national electricity
demand and estimates the impacts of electricity energy
efficiency policies in the commercial, industrial, and
residential sectors from 1992 to 2003 using the 48
mainland states. In this study, policy-related energy
savings were estimated after controlling for energy
prices, incomes, changes in equipment stock, and
weather. By the character of the research design and
the econometric models, the analysis netted out the
effects of market factors that both add to, and subtract
from, policy-related energy savings. The end result
was a comprehensive estimate of the long-term net
impacts of energy efficiency policy.
The four examples provided in this study follow in
this vein and are intended to demonstrate the
applicability of the econometric approach at various
levels of aggregation for which time series or panel
data are available. Generally speaking, for most
countries there exists at least three geographic levels
of publicly available data that are capable of being
used for energy efficiency policy evaluation. For
example, in the USA various kinds of public data are
available at the national level, the state level, and
utility service territory level, and even the county,
municipal, and zip-code levels. Time wise, various
kind of public data can be found at the decennial
level, the annual level, the quarterly level, and the
monthly level. Sector-wise, most of the data in the
USA can be divided into that which relates to
the residential, commercial, or industrial sector.
For clarity and focus, all four of these analyses take
electricity consumption in the residential sector as
their subjects and all four research designs result in
comparisons of actual energy sales data to simulated

or forecasted energy sales. Also, their econometric
models use similar explanatory variables, functional
forms, and estimators. However, standardization must
not be confused with rigidity. The optimal methodol-
ogy must be replicable from situation to situation, but
must also be adaptable. If they are to be true to reality,
energy demand analyses must sometimes differ in
their sampling frames, their time frames, their models,
and other specifics. The examples provided here
demonstrate these qualities; they take advantage of
information and resources that are unique to individ-
ual circumstances while remaining similar in key
features.
Single state, annual pre–post analysis
The simplest counterfactual simulation approach
uses a time series model for a single subject, such
as a state or a utility, to compare its actual energy
demand to that which would have been demanded
in the absence of an energy efficiency policy. There
are two features of this approach that determine its
success. The first is historical knowledge. Knowl-
edge of a subject’s energy efficiency programs and
policies is necessary for selecting a date, or dates,
when policy changes are likely to have occurred or
when their effects are likely to be detectable. In
addition, background knowledge of other major
events, or social and economic changes, that may
have also affected demand over the study period is
critical for developing and testing hypotheses and
for interpreting findings. The second ingredient for
success is knowing enough about a subject to be
able to specify a reasonable econometric model of
aggregate energy demand. Given these prerequi-
sites, a general function for a reduced-form time
series energy demand model, where the subscript
t represents a given year is:
Yt ¼ f ðPt ; Gt ; Wt ; Tt Þ:
In this function Y could be an absolute level of
energy or of energy demand, or an energy intensity
ratio of some kind. Determinants of energy demand or
intensity may include and of course are not limited to
electricity prices as represented by (P) and perhaps
also the price of the nearest substitutable fuel, such
natural gas prices; absolute or relative incomes as
Energy Efficiency
represented by (G); climatic conditions as represented
by (W); and time trend related to changes in
equipment stock as represented by (T). Estimating a
model based on this function for the years prior to
energy efficiency policy implementation, Y can then
be projected or simulated in the post-policy years.
This provides the necessary counterfactual by which
the impacts of the policies, i.e., energy savings, can be
empirically measured. Note that it is occasionally
suggested that economic variables, especially energy
prices, should be omitted from this function so that
impacts of non-economic policies can better be
detected (e.g., Sheppard et al. 2009). However, this
suggestion is not consistent with economic and
econometric theory. Models containing economic
variables and non-economic variables are readily
capable of being used to isolate the effects of each.
Given the basic function just described, the
strength of this approach is easily demonstrated.
Figure 1 depicts actual and simulated electricity use
(in BBTU) for four cross-section (state) subjects, i.e.,
New York, Oregon, Illinois, and Nevada. The
electricity sales data, taken from EIA’s SEDS database
covers 100% of all residential sector use in these
states. For convenience, each state is assumed to have
changed its residential sector electricity efficiency
policy after 1991 and to have an identical set of
determinants of aggregate demand. For the period
from 1970 to 1991, the model is specified such that
annual state average residential electricity use per
customer is regressed on average state electricity
prices, average state natural gas prices, average state
heating and cooling degree days, average state per
capita income, and a national equipment-related stock
time trend. These models, like all the models in this
paper, are estimated in log–log form. Model statistics
for this analysis are contained in the “Appendix”.
In Fig. 1, the shaded area represents the pre-regime
change period over which the behavioral relationships
are estimated. Each state’s model is then used to
retrospectively simulate the total annual residential
electricity use in the state from 1992 to 2005, taking
into account the actual number of residential custom-
ers in each year. In the figures, the solid line
represents actual use and the dashed line represents
simulated use through 2005. If the selected policy
cutoff year is accurate, and the model does not suffer
from misspecification, the simulation should produce
reasonably accurate trends showing what total elec-
Energy Efficiency

240,000 72,000
New York Oregon
220,000 68,000
64,000
200,000
60,000
180,000 56,000
160,000 52,000

140,000 48,000
44,000
120,000
40,000
100,000 36,000
80,000 32,000
1970 1975 1980 1985 1990 1995 2000 2005 2010 1970 1975 1980 1985 1990 1995 2000 2005 2010
180,000 40,000
Illinois Nevada
160,000 35,000

30,000
140,000
25,000
120,000
20,000
100,000
15,000

80,000 10,000

60,000 5,000
1970 1975 1980 1985 1990 1995 2000 2005 2010 1970 1975 1980 1985 1990 1995 2000 2005 2010

Fig. 1 Total electricity (BBTU) use by state, actual and predicted

tricity demand in each state might have been had
aggregate behavior from 1992 to 2005 been unaffect-
ed by energy efficiency policy. If the simulated
demand trend is well above the actual demand in the
later years of the policy, it can be concluded that the
energy savings from the policy change is observed
and verified.
It is essential to understand that the baseline for
these analyses (and unless otherwise noted for all the
other analyses using this econometric approach) is
approximately the mid-point of the pre-policy period,
in this case around the early 1980s, since the pre-
policy period stretches from 1970 to 1991. This must
be so because the coefficients of the models that are
used to forecast demand in the post-policy period are
Table 1 Four state energy
efficiency policy impacts for State Actual
2005
Use (BBTU)
the average, marginal effects of the independent
variables across all the pre-period years. Just as the
mean forecast for the pre-policy period will not be
equal to the actual value of the dependent variable for
the mid-year, so too the coefficients do not necessar-
ily represent the exact mid-point of the pre-policy
time period. This feature does not in any way dilute
the accuracy of findings—if anything, it provides a
more honest interpretation of the baseline, since in
reality the baseline is a moving target that should not
be viewed as a simple, single point in time.
From the individual state figures it can be seen that
policy savings appears to have occurred in New York
and Oregon. These findings seems reasonable given
that these two states had vigorous residential energy
Simulated Rel. SE (%) 90% error bound 2005 policy impact (%)
Use (BBTU) Lower Upper

NY 172,418 221,997 8.7 190,193 253,801 −28.8

OR 62,572 71,665 4.4 66,526 76,804 −14.5
IL 165,798 173,052 13.1 135,835 210,270 −4.4
NV 37,804 34,906 6.5 31,151 38,662 7.7

efficiency policies for at least the past 15 years. It is
also no surprise that in Illinois and Nevada, two states
without aggressive residential energy efficiency pol-
icies, actual and predicted total use do not differ by
much. Table 1 presents the data for these four states
for 2005, with Rel. SE, or relative standard error,
defined as the standard error of the simulated use
divided by the level of simulated use. In both New
York and Oregon, actual use is below the 90% lower
error bound of predicted use for 2005, whereas in
both Illinois and Nevada, actual use is well within the
expected range of predicted use. As such, the
estimated 29% policy impact in New York in 2005,
and the estimated 15% savings impact in Oregon are
statistically significant at the 90% level, whereas the
estimated impacts in Illinois and Nevada are not.
Again it should be emphasized that these findings do
not represent the incremental savings for a single
year, but rather the cumulative savings relative to the
approximate mid-point of the pre-policy period.
Introducing additional information to further ex-
plain the quantitative results is beyond the scope of
this discussion. However, it is worth noting that an
obvious objection to this approach might have to do
with the issue of attribution. Attribution hinges on the
degree of faith that is placed in the model specifica-
tion. If it is believed that the differences found in New
York and Oregon between simulated and actual use
are due to structural changes in the residential sector
rather than policy, this points to two likely modeling
problems; (a) either the model is misspecified,
meaning it has omitted key variables or suffers from
endogeneity bias, or (b) the true demand parameters
have changed over the study period for reasons other
than energy policy. The first of these problems is
easily correctable by revisiting the model. The second
problem is also easily correctable. Economic theory
argues that if demand parameters change at all, they
do so very slowly. The exceptions are when a
government policy targets demand or when major
events occur that change consumer preferences or
benefit and cost considerations. It so happens that in
these analyses what is being maintained is that policy
regime change has affected demand. To confirm that
it is not another policy or set of events that affected
demand in New York and Oregon, all that is needed is
a history of the period under inspection. In these four
state examples, notwithstanding that the policy cutoff
dates may not be as accurate as they could be, it is
Energy Efficiency
unlikely that any other non-energy policies or major
events affected their trends.
As stated above, this analysis of each state’s
policies is a starting point and should not be taken
as a definitive policy evaluation. Obviously, much
more time and effort must be put into studying the
impacts of energy efficiency policies in these and all
the other states, sectors, and fuels. The analysis is
offered to demonstrate the flexibility of this approach
and that it need not be overly costly or complex to
provide accurate, consistent, and unbiased estimates
of total, long-term policy-related energy savings.
Before going on to describe a second application of
counterfactual simulation it is worth emphasizing that
these measured savings, like those for all counterfac-
tual simulations using aggregate data, are net savings
in the broadest sense of the term. This cannot be
stressed enough given the current controversy in the
energy efficiency program evaluation field over the
inherent weaknesses in calculating of net-to-gross
factors for adjusting mean short-term savings. Using
the counterfactual approach, the confusion and con-
troversy surrounding this issue is avoided.
Single state, monthly pre–post analysis
A more detailed counterfactual analysis can be
conducted using monthly rather than annual aggregate
energy consumption data. To illustrate the adaption of
the econometric approach to monthly data and the
added value it provides, historical data for the
residential sector for all the Class A utilities in
Wisconsin are analyzed. These consumption levels
represent approximately 75% of the annual residential
sector electricity use in this state.
To initiate this analysis, the historical relationship
between average residential energy prices (in constant
dollars) and non-weather adjusted residential electric-
ity use are inspected from the earliest month avail-
able, January 1980, to the latest date available,
December 2008. Figure 2 vividly illustrates the
pattern of the two variables, with GWHR on the
left-hand axis representing aggregate residential con-
sumption and REALKWHP on the right-hand axis
representing real average cents per kWh. The indi-
vidual figures represent the shoulder months of May
and September. These similar patterns are typical for
all the months. They show rising levels of consump-
Energy Efficiency

13 13
12 12
11 11
10 10
1,200 1,400
9 9
1,100
8 1,200 8
1,000
900 1,000
800
800
700
600 600
86 05
88 05

96 05

92 09

08 09
04 05
06 05

84 09
86 09

00 09
02 09
82 05

98 05

08 05

88 09

94 09
8 4 05

90 05
92 05
9 4 05

00 05
02 05

05

82 09

9 0 09

96 09
98 09

04 09
0 6 09

09
M
M

M
M
M

M
M
M

M
M
M

M
M

M
M

M
M
M

M
M

M
M
M

M
M

M
M
80

80
GWHR REALKWHP GWHR REAKWHP

Fig. 2 WI class A utilities residential energy sales and prices, May and September (1980 to 2008)

tion over the three decades, coupled with declining are presented in Table 2 where GWHRF represents
prices until 1998, and rising prices thereafter. forecasted use and GWHRFSE represents the forecast
Clearly there was a dramatic shift in the relation- standard error.
ship of the two variables; this is confirmed with The findings show that savings varied by month.
diagnostic tests. The Quandt-Andrews breakpoint test However, caution should be used in overinterpreting
reveals what the naked eye can observe more quickly; the results and trying to look for specific meaning in
that a naive regression analysis that combines all the estimates for each month. Rather, the overall
years is likely to run into serious estimation and 2-year savings should be observed. It was about
interpretation problems. The most plausible explana- 2.8%, plus or minus approximately 0.5% at the 95%
tion for the structural break around 1998, backed by confidence level, relative to the baseline, roughly the
actual data, is that by the late 1990s the state of mid-year of the pre-policy period. Model statistics for
Wisconsin lacked its own generating capacity to meet this analysis are contained in the “Appendix”.
its increased demand and was forced to import higher-
priced electricity from Canada and other out-of-state
providers (Wisconsin Division of Energy 2006).
While a broad historical analysis may be of interest
2,000
for many other purposes, for the purposes of Actual Monthly GWH = Solid 2007m01
Predicted Monthly GWH = Dotted Line
estimating the recent impacts of energy efficiency 1,800
policies, this historical knowledge and empirical
evidence helps direct the focus of the demand analysis 1,600
to the post-1997 period.
To estimate the aggregate impact of energy 1,400
efficiency programs in 2007 and 2008, a monthly
model is estimated using the first month in 1998 up 1,200
through the last month in 2006. Then, using the actual
values of the independent variables in 2007 and 2008, 1,000
the monthly energy use for these years are forecasted.
Figure 3 provides a visualization of the analysis, with 800
the solid area representing actual monthly consump- 98 99 00 01 02 03 04 05 06 07 08

tion and the dotted line representing the monthly Fig. 3 WI energy demand model actual and predicted:
forecasts while the numerical results for the 24 months estimation period—1998 to 2006 (108 months)
 Energy Efficiency

Table 2 WI energy
efficiency policy impacts Obs. GWHR GWHRF GWHRFSE Obs. GWHR GWHRF GWHRFSE
for 2007 and 2008
2007M01 1,555 1,502 125 2008M01 1,688 1,527 129
2007M02 1,506 1,333 111 2008M02 1,458 1,400 117
2007M03 1,336 1,518 126 2008M03 1,375 1,511 127
2007M04 1,156 1,195 99 2008M04 1,183 1,205 101
2007M05 1,173 1,306 109 2008M05 1,157 1,267 107
2007M06 1,474 1,514 127 2008M06 1,290 1,547 132
2007M07 1,650 1,637 137 2008M07 1,617 1,716 146
2007M08 1,729 1,641 137 2008M08 1,567 1,676 142
2007M09 1,338 1,318 110 2008M09 1,243 1,335 113
2007M10 1,243 1,331 111 2008M10 1,172 1,268 106
2007M11 1,284 1,508 128 2008M11 1,350 1,486 126
2007M12 1,664 1,548 132 2008M12 1,651 1,530 130
Total 17,108 17,349 1,454 Total 16,751 17,469 1,475
Savings 242 718
% savings 1.4 4.3
Rel. SE 8.4 8.4

Multiple utilities, annual pre–post analysis
An extension of the single time series counterfactual
simulation involves using several cross-section sub-
jects, be they nations, states, utilities, etc., to estimate
jointly the impacts of energy efficiency policies. The
general function for a cross-section time series energy
demand model, where the subscript t represents a
given year and s represents a give cross-section and
the variables as defined above is:

Monthly heating and cooling degree days (HDD and
Yt;s ¼ f Pt;s ; Gt;s ; Wt;s ; Tt;s :
This version of the counterfactual approach is more
complicated than the former because it hinges on
finding compromises that work for all the cross-
sections, be they in establishing reasonable dates for
regimes changes, specifying a common model, or
identifying and controlling for variations in energy
efficiency policies. However, it can also yield higher
quality coefficients than a single time series because it
boosts the amount of information available for
analysis.
To demonstrate the feasibility and value of using a
time series cross-section approach at the utility level,
residential electricity sales are analyzed for the four
major utilities in the state of California, i.e., Pacific
Gas and Electric Company (PG&E), Southern Cali-
fornia Edison (SCE), San Diego Gas and Electric
Company (SDG&E), and Sacramento Municipal
Utility District (SMUD). Together, these utilities
represent 82% of 2007 residential electricity sales in
California.
To produce utility-specific data, the California
Energy Commission (CEC) map of electric utility
service territories was used to allocate each of the 58
counties in California to a single utility. Where two or
more utilities served the same county, the utility that
served the largest share of the county’s population
was designated the utility for that entire county.
CDD at base 65°F) for all the major weather stations
in California whose data covered the dates used for
this analysis (these are known as WBAN stations,
short for “Weather Bureau Army Navy”) were then
acquired from the National Climatic Data Center
(NCDC). Based on weather zones and proximities, a
total of 23 of these weather stations were assigned to
the 58 counties. To derive the average HDD and CDD
for each utility, the final step in this process was to
weight the weather data by the population in each
county, a procedure that is similar to the one used by
NCDC to derive state-level averages.
Annual residential electricity sales and prices data
for each utility was taken from the CEC’s Energy
Consumption Data Base (ECDMS). These data
extend as far back as 1990 and, for the residential
sector, are almost identical to the data available from
the US Energy Information Administration’s Form
Energy Efficiency

18
Inspection of California’s electricity prices are
instructive on many levels. They suggest that analyses
16 of California’s residential energy demand, at least
Real Cents per kWh

since the late 1990s, are fraught with danger and they
14 illustrate why some of the major features of the
econometric analyses used to derive counterfactuals
for estimating energy efficiency impacts cannot be
12
standardized but rather must take into consideration
the unique circumstances of the subject at hand. No
10 matter what the results of short-term evaluations,
none are capable of anticipating the additional
8 savings, or the reduction in savings, that can occur
1 - 90
1 - 94
1 - 98
1 - 02
1 - 06
2 - 91
2 - 95
2 - 99
2 - 03
2 - 07
3 - 92
3 - 96
3 - 00
3 - 04
3 - 08
4 - 93
4 - 97
4 - 01
4 - 05
from dramatic changes caused by economic, social,
technical, and political conditions.
1 = PG&E 2 = SCE 3 = SDG&E 4 = SMUD In implementing this multi-utility time series
analysis, a similar set of variables and a similar
Fig. 4 CA annual average residential electricity prices (1990–2007)
functional form is used as in the previous analyses for
the individual states. Only here, a weighted least-
EIA-861 database. The final data used for the cross- squares, fixed effects model is specified, and the pre-
section time series utility-level energy demand anal- policy period is the span of years from 1990 through
ysis was taken from the US Bureau of Economic 2005 with energy efficiency impacts estimated for
Analysis Regional Economic Information System 2006 and 2007. Note that since residential energy
which contains economic data for all the counties in efficiency programs have been widespread and
the USA. These county data were summed to arrive at growing in the California since the late 1970s, the
utility totals for variables, such as population and estimated impacts must be interpreted as the marginal
personal income. impact of the energy efficiency programs run in 2006
Prior to undertaking the energy demand analysis, and 2007 relative to the average impacts of the
average annual prices per utility were plotted. These programs run between 1990 and 2005. Table 3
prices are presented in Fig. 4, in which it can be seen presents the California utility results. Again, model
that California residents did not face the slowly statistics for this analysis are contained in the
changing administered prices usually found in most “Appendix”.
states and at most electric utilities. Rather, real In total, it appears that residential energy efficiency
average prices were erratic, most likely due to the programs in California led to a 2-year decrease in
upheaval caused by electric utility deregulation in the energy as of 4.8%. As noted previously, these results
mid-1990s that culminated in California’s disruptive are not interpretable as the incremental savings for
energy episodes in the early part of the present these 2 years combined. Rather, they are the cumu-
decade. lative savings of energy efficiency policies relative to

Table 3 CA energy effi-

ciency policy impacts for Util Obs. GWHR GWHRF GWHRFSE Av. Rel. SE (%) Total diff Total % diff
2006 and 2007 (sum of four
largest electric utilities) PG&E 1-06 30,823 32,981 1,731 5.13 −5,444 −8.8
PG&E 1-07 30,846 34,132 1,714
SCE 2-06 30,431 30,378 1,821 5.94 −847 −1.4
SCE 2-07 30,212 31,111 1,834
SDG&E 3-06 7,522 7,776 501 6.28 −416 −2.8
SDG&E 3-07 7,541 7,703 471
SMUD 4-06 4,747 4,816 277 5.57 −318 −3.4
SMUD 4-07 4,634 4,883 264
2-year total −7,024 −4.8

the approximate mid-point of the designated pre-
policy period, perhaps 8 or 9 years earlier. As a result,
roughly speaking the annual savings from residential
energy efficiency policies since the late 1990s seems
to average about one-half of a percent for the state as
a whole. Food for additional thought is whether or not
inter-utility comparison can be made to determine
why savings are higher at some utilities compared to
others. For this purpose, additional econometric work
and the introduction of individual program evaluation
findings could be quite instructive.
Two groups of states, annual pre–post analysis
To provide another example of the flexibility and power
of econometric analysis and counterfactual simulation,
the differential effect of utility energy efficiency policies
is analyzed for Oregon electric utilities. This example
also shows how the information generated from this kind
of analysis can be used for understanding and quantify-
ing the behavioral effects of energy efficiency policies.
For consistency, the model used for this analysis derives
from the same function described above, the difference
here being the inclusion of a subscript, R, to represent
entirely different energy efficiency policy regimes. The
function now appears as:

like the three utilities in the 1989 to 2002 period, their
Yt;s;R ¼ f Pt;s;R ; Gt;s;R ; Wt;s;R ; Tt;R
where policy regime change is assumed to occur for
some cross-sections and years and not others.
The residential energy demand model specified for
Oregon electric utilities consists of two groups of
utilities in Oregon, one containing three utilities with
large, comprehensive residential electricity efficiency
programs, the other containing 27 utilities with fewer
and less heavily promoted programs. The former
Table 4 OR energy efficiency policy impacts for 2003 through 2006 (sum of three largest electric utilities)
Year Actual use (GWH) Simulated use (GWH)
Pre-2003 (Avg.) 15,643 15,275
2003 16,477 15,925
2004 16,723 15,851
2005 17,011 16,242
2006 17,596 16,169
Energy Efficiency
group includes Portland General Electric and Pacificorp,
both of whom are investor-owned utilities and have
been involved in the Energy Trust of Oregon (ETO).
The third utility in the group is the Eugene Water and
Electric Board, a public utility that is not involved in
ETO, but that nevertheless maintains comprehensive
energy efficiency policies. Together, in 2006 these three
utilities were responsible for providing approximately
80% of the residential electricity in the state.
Using a similar set of variables to those employed
in the prior simulations, a weighted fixed effects
model is estimated for the small group of utilities with
significant residential electricity efficiency policies
and then the total energy demand of the group of 27
utilities is simulated. For the analysis, the estimation
period is 1989 through 2002, and the simulation
period is 2003 through 2006. Table 4 contains the
results of the counterfactual simulation in which the
behavioral coefficients of the small group of energy
policy intensive utilities were used to simulate the
energy demand of the other group of 27 utilities for
all the years from 2003 through 2006. Model statistics
for this analysis are also contained in the “Appendix”.
As can be seen in the column labeled “% diff.,” in
2006 the difference between actual and simulated
total residential electricity demand in the 27 utilities
was 8.8%. In other words, had the 27 utilities behaved
energy use would have been lower by almost 9%.
However, it would be a mistake to attribute all of this
difference to a lack of energy efficiency policies.
Comparison of actual and simulated use prior to 2003
for the 27 utilities reveals that simulated use is
consistently lower by an average of 2.4%. This
suggest underlying systemic differences between the
two groups of utilities that could be done prior to
energy efficiency policies or differences in the market
% diff. Rel. SE 90% error bound Implied policy impact (%)
Lower Upper
−2.4 2.5 14,642 15,907 –
−3.5 2.6 15,253 16,597 −1.1
−5.5 2.5 15,196 16,505 −3.1
−4.7 2.5 15,576 16,909 −2.3
−8.8 2.5 15,494 16,844 −6.4
Energy Efficiency
penetration of electric heating or air conditioning, or a
host of other factors. As such, to derive net policy
impacts the systemic difference is subtracted out,
leaving an implied policy impact for 2006 of 6.4%.
These findings suggest that if the three largest
utilities’ residential programs were adopted by the
group of 27 utilities, their combined annual energy
demand would show statistically significant decreases
at the 90% confidence level.
This counterfactual simulation is useful not only
for verifying the impacts of energy efficiency policies,
but for understanding energy demand behavior. For
example, the same energy demand model can be
estimated for both groups of utilities and their
coefficients compared (see Appendix). For these two
groups of Oregon utilities, the major difference in the
models is in the elasticities representing the impact of
changes in economic growth, a proxy for income. A
simple comparison indicates that increases in eco-
nomic growth in the three largest utilities are
associated with relatively less residential electricity
use per customer. This is consistent with the hypoth-
esis that energy efficiency policies have a beneficial
influence on the relationship between increasing
affluence and energy use by encouraging capital
purchases, and behaviors, that slow energy use.
Further, to get a quantitative sense of the importance
of the differential elasticities between the two policy
regimes it is possible, using the formula below, to
calculate how a change in the value of each variable,
holding all other variables constant, would affect Y,
energy use per residential customer:
% change in Y ¼ 1
ð1 þ % change in X ÞbX :
Using this formula and the estimated model
coefficients for economic growth, an assumed future
increase in economic growth statewide of 25% is
projected to lead to a decline in average residential
customer use of over 11% in the top three utility
service territories, whereas in the other utility service
territories, average residential customer use would
hardly decline at all.
These illustrations show that a wealth of informa-
tion can be derived from the counterfactual simulation
approach as it is applied to groups of subjects with
different policy regimes. It should also be obvious
that this approach is applicable not only to groups of
utilities, but to groups of states, or nations, or
communities. The only technical requirement is that
there are common sets of data collected over the same
time frequencies for all the subjects. Although an
inventory of such data is far from the scope of this
study, it is clear that much of these kinds of data
already exist at international and national agencies.
Also, it is not hard to imagine that with reasonable
effort, much of these local data can be collected from
state agencies such as state energy offices, public
utility commissions, and local electric and gas utilities.
Conclusion
The fundamental issue that this paper addresses is
how to measure or verify the total, long-term energy
savings that come from energy efficiency policies.
Towards this end four examples are provided that
demonstrate the feasibility, and the flexibility, of
achieving this goal using time series and panel data
econometric analyses. Not only has this paper shown
that counterfactual simulation is adaptable to many
different geographic circumstances and data constraints,
but it has also shown that these kinds of analyses yields
valuable information and insights that are unattainable
with surveys and pseudo-experimental evaluation
designs for individual programs.
For governments or public agencies wishing to
extend this approach further, the next step in the
development of energy efficiency policy evaluation is
to create a location-oriented database that matches
historical energy sales data with economic, demo-
graphic, and policy-related variables. In the USA this
database can consist of more than 2,300 electric
utilities and more than 1,500 natural gas utilities, most
of whom have at least 10 years of energy sales and
revenues information available for analysis. With this
degree of differentiation and data depth, many
different research designs are possible depending on
the nature of the counterfactual being investigated.
On the international scene, the levels of differentia-
tion between locations and policies may be quite
different than that found in the USA. Nevertheless, it is
likely that every country collects time series data that are
related to annual energy sales and other basic economic,
demographic, and policy variables. With even rudimen-
tary data, the counterfactual simulation approach to
estimating policy impacts can and should be tried.
Finally, it should be noted that the approach to
evaluating energy efficiency programs in aggregate that
 Energy Efficiency

is described and advocated in this study is not intended
to replace existing methods for evaluating individual
energy efficiency programs. Individual program impact
evaluations are essential to program management and
design, not to mention cost recovery and incentives.
They provide immediate feedback to program imple-
menters and policymakers, and permit programs to be
modified, improved, and optimized on a regular basis.
Counterfactual simulations for measuring policy savings
cannot provide the same information as do these
program evaluations. They should therefore be viewed
as complimentary and as providing new kinds of
program net energy savings estimates, ones that are not
satisfactorily provided by current methods.
Acknowledgements This research was funded by the U.S.
Environmental Protection Agency. The author would like to
thank Andy Henderson and Emanuel Klein for their sugges-
tions and advice, Anna Horowitz for her research assistance,
and Jim Mapp for providing the monthly MWh sales data for
Wisconsin. The author takes sole responsiblity for all errors and
opinions found within.

Appendix
Single state, annual pre–post analysis

Dependent variable: residential electricity consumption

(BBTU) per capita. Estimation time period: 1970–1991,
forecast time period 1992–2005

Illinois Actual values

Model variables Coeff. SE Prob. 1970 1991 2005
Constant −8.254 0.829 0.000
Electricity price −0.137 0.119 0.265 28.954 34.251 21.643
Natural gas price 0.058 0.055 0.306 3.706 5.750 10.135
Per capita income 0.849 0.311 0.016 16.596 25.123 32.294
Equip. stock trend −0.016 0.055 0.775 1.476 11.574 123.826
Heating degree days 0.036 0.089 0.689 6,290 5,836 5,929
Cooling degree days 0.150 0.040 0.002 973 1,153 1,137
Adj. R-squared 0.932 DV= 0.007 0.011 0.013

Nevada
Model variables Coeff. SE Prob. 1970 1991 2005
Constant −11.766 2.097 0.000
Electricity price 0.104 0.150 0.499 16.192 20.451 26.443
Natural gas price −0.227 0.095 0.030 5.059 6.410 10.452
Per capita income 0.805 0.365 0.043 17.927 24.586 33.161
Equip. stock trend −0.080 0.056 0.174 1.476 11.574 123.826
Heating degree days 0.462 0.177 0.019 4,321 4,133 3,501
Cooling degree days 0.199 0.170 0.260 1,536 1,675 2,088
Adj. R-squared 0.450 DV= 0.014 0.015 0.016

New York
Model variables Coeff. SE Prob. 1970 1991 2005
Constant −7.106 1.436 0.000
Electricity price 0.225 0.129 0.101 32.066 41.548 40.779
Natural gas price −0.038 0.074 0.612 4.973 8.469 12.821
Per capita income 0.042 0.177 0.817 17.700 28.380 36.309
Energy Efficiency

Equip. stock trend 0.181 0.049 0.002 1.476 11.574 123.826

Heating degree days 0.092 0.107 0.406 6,092 5,262 5,984
Cooling degree days 0.013 0.050 0.799 750 804 944
Adj. R-squared 0.960 DV= 0.005 0.007 0.009

Oregon
Model variables Coeff. SE Prob. 1970 1991 2005
Constant −8.085 0.824 0.000
Electricity price −0.165 0.091 0.089 13.268 16.704 18.817
Natural gas price −0.003 0.032 0.935 5.251 7.047 10.934
Per capita income 0.413 0.132 0.007 14.252 21.940 27.950
Equip. stock trend −0.006 0.021 0.795 1.476 11.574 123.826
Heating degree days 0.383 0.068 0.000 5,240 5,056 4,937
Cooling degree days 0.008 0.017 0.651 220 267 282
Adj. R-squared 0.734 DV= 0.016 0.019 0.017

Single state, monthly pre–post analysis

Dependent variable: monthly residential electricity

consumption (GWH) per state personal income.
Estimation time period: 1998–2006, forecast time
period 2007–2008

Wisconsin Actual values

Model variables Coeff. SE Prob. Jan. 1998 Jun. 2002 Dec. 2006
Constant 3.409 0.264 0.000
Summer (J,J,A) 0.088 0.030 0.004 0 1 0
Winter (N,D,J,F,M) 0.223 0.019 0.000 1 0 1
Electricity price −0.373 0.173 0.033 8.273 9.185 10.281
Equip. stock trend 0.132 0.046 0.005 66.882 90.352 172.866
Heating degree days −0.043 0.010 0.000 1,278 68 1,068
Cooling degree days 0.012 0.006 0.045 0.000 162 0.000
Adj. R-squared 0.634 DV= 25.548 23.615 23.887

Multiple utilities, annual pre–post analysis mation time period: 1990–2005, forecast time period
2006–2007
Dependent variable: utility residential electric con-
sumption (GWH) per utility personal income. Esti-

4 largest CA Utilities Actual mean values

Model variables Coeff. SE Prob. 1990 1998 2005
Electricity price −0.103 0.080 0.205 13.995 12.267 13.488
Shock2000–2003*Ele. price −0.024 0.007 0.001 0 0 0
Equip. stock trend −0.091 0.009 0.000 9.738 70.409 131.815
Heating degree days −0.063 0.070 0.373 2,194 2,339 1,900
Cooling degree days 0.035 0.044 0.421 1,309 1,037 1,151
Adj. R-squared 0.976 DV= 83.3 75.3 65.8
 Energy Efficiency

Two groups of states, annual pre–post analysis

Dependent variable: residential electricity consump-

tion (MWH) per customer. Estimation time period:
1989–2002, forecast time period 2003–2006

3 largest electric utilities Actual mean values

Model variables Coeff. SE Prob. 1989 2002 2006
Electricity price −0.119 0.029 0.000 0.057 0.069 0.067
Natural gas price −0.147 0.021 0.000 7.606 9.774 12.149
Economic growth −0.543 0.114 0.000 7.356 7.237 7.353
Equip. stock trend −0.029 0.003 0.000 8.373 100.000 143.356
Heating degree days 0.406 0.037 0.000 5,217 4,993 4,933
Adj. R-squared 0.981 DV= 13.841 11.926 11.817

27 smaller electric utilities

Model variables Coeff. SE Prob. 1989 2002 2006
Electricity price −0.220 0.021 0.000 0.059 0.063 0.061
Natural gas price −0.089 0.012 0.000 7.606 9.774 12.149
Economic growth −0.024 0.012 0.044 9.312 8.753 8.188
Equip. stock trend −0.003 0.002 0.116 8.373 100.000 143.356
Heating degree days 0.404 0.024 0.000 5,217 4,993 4,933
Adj. R-squared 0.971 DV= 15.098 14.435 14.162

References Metcalf, G. (2008). An empirical analysis of energy intensity

Bernstein, M., Fonkych, K., Loeb, S., & Loughran, D. (2003).
State-level changes in energy intensity and their national
implications. Santa Monica: Rand, MR-1616-DOE.
EMEEES (2008). Harmonised methods for evaluating energy
end-use efficiency and energy services—the European
Commission’s project on evaluation and monitoring for
the end-use directive on energy end use and energy
services, Final Conference, Brussels, BE.
Fisher, F. M., & Kaysen, C. (1962). A study in econometrics:
The demand for electricity in the United States. Amsterdam:
North Holland Publishing Co.
Halvorsen, R. (1975). Demand for electric energy in the United
States. Southern Economic Journal, 42(4), 610–625.
Horowitz, M. J. (2004). Electricity intensity in the commercial
sector: market and public program effects. The Energy
Journal, 25(2), 115–137.
Horowitz, M. J. (2007). Changes in electricity demand in the United States
from the 1970s to 2003. The Energy Journal, 28(3), 93–119.
Loughran, D. S., & Kulick, J. (2004). Demand side management
and energy efficiency in the United States. The Energy
Journal, 25(1), 19–43.
and its determinants as the state level. The Energy Journal,
29(3), 1–27.
National Action Plan for Energy Efficiency. (NAPE, 2007).
Model energy efficiency program impact evaluation guide.
Northeast Energy Efficiency Partnerships. (2008). Regional
EM&V forum’s three year plan.
Sheppard, Chamberlin, & Jacobson (2009). Exploring strate-
gies for implementing a performance based state efficiency
program: State energy consumption metrics—residential
sector analyses. Schatz Energy Research Center, Humbolt
State University.
Taylor, L. D., Blattenberger, G. R., & Rennhack, R. K.
(1984). Residential energy demand in the United States:
introduction and overview of alternative models. Advan-
ces in the Economics of Energy and Resources, 5, 85–
105.
U.S. Department of Energy’s Office of Energy Efficiency and
Renewable Energy (IPMVP, 2002 and updated in 2007).
International Performance Measurement & Verification
Protocol.
Wisconsin Division of Energy. (2006). Wisconsin energy
statistics (2006). Madison: Wisconsin Department of
Administration.