THE BEST OF

THE BEST O
MEASURIN F
&
EVALUA
LEAR G TIN
G
NING
SPONSORED BY:
 N T E N T S
TAB LE OF CO
R ’ S N OT E 3 A T+D Classic: How to Start an Objective 36 On the Level

EDITO
Evaluation of Your Training Program Don’t let good training fly out the window. Use
An excerpt from Kirkpatrick’s classic 1956 article. Level 3 to make sure it sticks. +“How to Apply
Kirkpatrick’s Four Levels of Evaluation”
By Donald L. Kirkpatrick
By Donald L. Kirkpatrick
By George Vellios
6 Why ROI Isn’t Enough
A shareholder focus may drive ROI evaluation—but 40 Tangible Return on Investment: Integrating
SHOWING ‘VALUE’ TRULY MATTERS

stakeholders in the Netherlands want more.
By Reinout van Brakel
Learning to Reach Desired Results
One organization’s experience illustrates how
This subject has been highly contested in the last 9 ROI of E-Learning: Closing In learning initiative design influences outcomes,
while making the case for continuing professional
50 years. Since Don Kirkpatrick’s Four Levels were

As companies look to e-learning to help meet
their strategic goals, they’re seeking ROI analysis.
development even in a tough economy.
By Rebecca Everett
published in T+D magazine in 1956, learning

Some of the most valuable measurements can
be hard to capture, but we’re closing in.
44 Measuring What Matters: How CEOs View
By Paul Harris
professionals have been debating the question, Learning Success
“Do metrics and measurement really matter?” 14 The Evaluation Heavyweight Match The mystery of the top executives’ view of
success in learning and development is brought
Donald Kirkpatrick and Jack Phillips, renowned to light in a recent survey of Fortune 500 CEOs.
As most learning professionals have seen over the last decade, return-on-investment matters to senior for their work and contributions in the
By Jack J. Phillips and Patti P. Phillips
executives, but only when learning professionals can prove how the initiative or training program links to the evaluation arena, sparred at the ASTD ROI
Network Conference in Amsterdam, organized
business goals of the organization. As Jim Kirkpatrick has written in T+D articles over the years, learning 50 Hitting the Suite Spot
by the ASTD ROI Network in cooperation with
professionals should know what kind of data they need to convince executives of a program’s worth, prepare
ProfitWise. The champions’ mutual respect was How learning leaders and executives can speak
the data effectively, and passionately present it. apparent, yet controversy wasn’t avoided. In the same language.
fact, an old score was settled. By Tom Kelly
The latest economic recession has put “value” at the forefront of a training program’s survival. With company Interview by Diederick Stoel
departments living on limited budgets and fighting for limited resources, top-level support for programs is vital
to the success of all endeavors. Gathering comprehensive data to create a chain of evidence is the best way to 54 Confronting CEO Expectations About the Value
17 Measurement Gets Strategic of Learning
demonstrate training value to stakeholders, but what data is the right data?
Abraham Maslow said, “If the only tool you
Top-level support is vital to maintaining any
have is a hammer, you tend to see every
Learning professionals can use the ADDIE (analysis, design, development, implementation, and evaluation) organizational learning function. Survey data
problem as a nail.” Has ROI become
model to uncover an initiative’s return on expectations. The only way to truly know what is important is to provide some clear directions for how, and how
that hammer?
document expectations of the program and then measure those expectations at the completion of the event. not, to provide executives with the appropriate
By Paul Bernthal evidence of learning success.
This compilation of T+D articles will not only introduce you to Donald Kirkpatrick’s famous “Four Levels” By Jack J. Phillips and Patti P. Phillips
article, but will also give you wisdom from Jack J. Phillips, Jim Kirkpatrick, and many others. Numbers and data 22 The C-Level and the Value of Learning
can be overwhelming, but the only way learning will remain relevant in today’s highly competitive business A groundbreaking study examines CXO-CLO 60 Overcoming Barriers and Valuing Evaluation
environment is to prove value in programs and initiatives. alignment and disagreement on the strategic
An ASTD-i4cp study underscores the progress
value of learning.
organizations are making in the area of learning
By Tony O’Driscoll, Brenda Sugrue, and Mary Kay Vona evaluation, as well as the gaps that need closure.
By Laleh Patel
30 Measuring ROI: Fact, Fad, or Fantasy
ROI has become a way of life in many training 62 ROE’s Rising Star
Paula Ketter departments. A recent meeting of learning
Why return on expectations is getting so much attention.
Editor, T+D executives provided some interesting insights
regarding its acceptance. By James D. Kirkpatrick and Wendy Kayser Kirkpatrick
pketter@astd.org
By Jack J. Phillips

SPONSORED BY:
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 1
 FUNDAMENTALS

THE MAJORITY OF THE RESPONDENTS (91.6

PERCENT) SAID THEIR COMPANIES EVALUATE AT
LEAST SOME OF THEIR LEARNING PROGRAMS AT
LEVEL 1, THE REACTION OF THE PARTICIPANTS.
THIS MAKES SENSE AS IT IS TYPICALLY THE
EASIEST MEASUREMENT TO TAKE.

From the 2010 ASTD Research Study The Value of Evaluation: Making Training
Evaluations More Effective. Visit www.astd.org/research for more info.

A T+D Classic: How to Start an Objective

Evaluation of Your Training Program
This excerpt is part of a larger article with
By Donald L. Kirkpatrick the same title that originally appeared in the
May-June 1956 issue of the Journal of the
American Society of Training Directors, a
predecessor to T+D. The article heralded
Kirkpatrick’s now classic four-level evalua-
tion model.
Most training men agree that it is
important to evaluate training programs.
They also feel that the evaluation should be
done by objective means. However, the typ-
ical training man uses evaluation sheets or
comment sheets as the sole measure of the
effectiveness of his programs. He realizes he
should do more, but he just doesn’t know
how to begin an objective evaluation.
According the Raymond Katzell, a
well-known authority in this field, the
evaluation of a training program falls into
a hierarchy of steps that can be briefly stat-
ed as follows:
Step One. To determine how the trainees
feel about the program.
Step Two. To determine how much the
trainees learn in the form of increased
knowledge and understanding.

2 | T+D | DECEMBER 2010 THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 3
Step Three. To measure the changes in the
on-the-job behavior of the trainees.
Step Four. To determine the effects of
these behavioral changes on objective cri-
teria such as production, turnover, absen-
teeism, and waste.
In climbing this ladder of evaluation,
most trainers have completed the first
step. Typically, the training director asks
the trainees to fill out evaluation sheets at
the end of the program. Questions that
are asked most frequently are
● How do you rate the program?
● What subject did you like best?
● What subject did you like least?
● What did you learn that you can use
on the job?
● What subjects would you like to have
discussed at future programs?
Usually the trainees are not
asked to sign their name for
fear they will not give an honest
reaction.
This kind of subjective evalua-
tion is important. It gives a good
indication of how the trainees
reacted to the program. If they
react favorably, the trainer can jus-
tifiably pat himself on the back and
say, “I gave them a program they
liked.” But he can’t rightfully claim
that the training program accom-
plished the objective, unless his
objective was to give them a pro-
gram they liked.
The immediate objective of any train-
ing course can be stated in terms of the
desired knowledge and understanding
that the program is trying to impart to
the trainees. It is this stage of evaluation
that should be undertaken as the second
step. It is much more difficult than step
one and, therefore, is not undertaken by
many trainers.
Among the possible methods for
determining whether increased knowl-
edge and understanding have taken place,
the best one seems to be the “before and
after” paper and pencil test. If the scores
4 | T+D | DECEMBER 2010
on the posttest are significantly higher
than on the pretest, the course can be
deemed effective.
In determining the effectiveness of the
training, it is important to note that the
paper and pencil test or inventory must
cover the principles and facts that are dis-
cussed in the course. If the trainer can find
a test that covers this material, he can use
it. If he cannot find a suitable one, he must
construct his own inventory. Some of the
inventories that are available are: How
Supervise? by File and Remmers; Supervi-
sory Inventory by Wesley Osterberg; and
the Supervisory Inventory on Human Rela-
tions constructed by this writer.
So far, then, it has been stated that a
before and after test can be used to deter-
Evaluation sheets give a valid measure
of trainee reaction to the program.
However, they do not give any evidence
of benefits derived.
mine whether or not increased knowledge
and understanding have taken place.
Also, that the inventory should cover the
course content. In order to determine
whether or not an available test is suitable,
a trainer must examine his course outline
and list the principles and facts he is try-
ing to teach. A comparison of test items
with these objectives will reveal whether
or not the test can be used. Because the
construction of a test involves such factors
as the choice of items, the wording of the
items, the number and type of possible
response, and the sequence of items, it is
far better to use an available inventory if it
covers most of the course content.
Having selected or constructed a test,
the trainer should consider some “Do’s”
for administering it:
● Give the pretest at the start of the first
class and the posttest at the close of
the last session. This will minimize the
influence of factors apart from the train-
ing course.
● Have the trainee sign both the pretest
and posttest. Then, the increased knowl-
edge and understanding can be computed
for each individual.
In instructing the trainees before they
take the pretest:
● Tell them it is a before and after
procedure.
● Explain the purpose of the test.
● Encourage them to answer truthfully
by assuring them that their scores will
have no effect on their pay or status in
the company.
● Tell them to answer every question
even if they have to guess. (This will be
taken into account in the statistical analy-
sis of scores.)
● Encourage them to take their time in
taking the test. This will help to motivate
them to read each item carefully.
In analyzing the test results, there are
two kinds of evaluations to be made:
● Was the entire course effective as
shown by gains from pretest to posttest
scores for all trainees?
● What specific facts and principles were
learned as shown by changes from pretest
to posttest for each item?
ALTHOUGH LEVEL 1 IS THE
Summary
MOST COMMONLY USED Training men agree that it is advisable to
TYPE OF EVALUATION, IT IS evaluate training courses as objectively as
possible. Typically, their evaluation con-
NOT THE MOST LIKELY TO sists of subjective comment sheets that are
completed by the trainees at the end of
BE SEEN AS HAVING HIGH the course. Providing that these are prop-
erly administered, these evaluation sheets
OR VERY HIGH VALUE. IN give a valid measure of trainee reaction to
the program. However, they do not give
FACT, ONLY 35.9 PERCENT any evidence of benefits derived.
The first step in objectively evaluating
OF RESPONDENTS WHOSE the effectiveness of a training course is to
COMPANIES USE LEVEL 1 determine whether or not the desired
EVALUATION SAID IT HAD
facts and principles were learned by the
trainees. This can be done by:
● Using a suitable paper and pencil test.
HIGH OR VERY HIGH VALUE. ● Testing the trainees before and after
the program.
BY COMPARISON, 54.9 ● Determining the overall effectiveness
of the course by comparing pretest and
PERCENT SAID LEVEL 2 HAD
HIGH OR VERY HIGH VALUE,
posttest scores for each trainee.
● Determining which specific facts and
principles were learned by analyzing the
AND THREE-QUARTERS SAID changes on each test item from pretest
and posttest.
THE SAME ABOUT LEVEL 3 The purpose of this article is to suggest
a specific technique for beginning an
AND LEVEL 4. objective evaluation of training program.
Further efforts should be undertaken by
every training man to follow up this kind
of evaluation by attempting to measure
From the 2010 ASTD Research Study The Value trainee change in behavior that occurs as
of Evaluation: Making Training Evaluations
a result of participation in the program.
More Effective. Visit www.astd.org/research
for more info.
Donald L. Kirkpatrick, renowned for his work
in the evaluation field, is the recipient of ASTD’s
2003 Lifetime Achievement Award in Work-
place Learning and Performance. He was an
assistant professor at the University of Wiscon-
sin’s Industrial Management Institute when he
wrote this article.
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 5
 PA S S P O R T
● Does the organization implement
evaluation policies and procedures?
Typically, the answer to those ques-
tions is no. PricewaterhouseCoopers in
the Netherlands is trying to change that.
For more than three years, PwC has con-
ducted evaluations using the methodolo-
gies of Kirkpatrick and Jack J. Phillips.
But here’s the rub: For projects
THE METHOD IN WHICH TRAINING IS DELIVERED CAN BE A STRONG in the Netherlands and Germany,
DETERMINANT FOR WHETHER AND HOW THAT TRAINING WILL BE PwC has found that cultural and insti-
tutional differences affect the feasibility
EVALUATED. WHILE MORE THAN 80 PERCENT OF LIVE CLASSROOM of measuring ROI.
Paul Boselie, Jaap Paauwe, and Paul
TRAINING PROGRAMS ARE EVALUATED BY EMPLOYEE REACTION, Jansen, in their International Journal of
Human Resource Management article,
ONLY 52 PERCENT OF E-LEARNING PROGRAMS ARE MEASURED AT “Human Resource Management
and Performance: Lessons From
THIS LEVEL. IN FACT, CLASSROOM PROGRAMS WERE MORE LIKELY the Netherlands” (November
TO BE EVALUATED AT EVERY LEVEL THAN E-LEARNING PROGRAMS. 2001), provide support for our
experience. They state that a Change happens when you link training to
focus on stakeholders—instead
From the 2010 ASTD Research Study The Value of Evaluation: Making Training of shareholders—encourages a business strategies, measure the impact of
training, and have stakeholder support.
Evaluations More Effective. Visit www.astd.org/research for more info. different system for measuring
HRM results. A focus on share-
holder value is a driver for mea-
suring ROI; however, Dutch
companies must also pay attention to
such stakeholders as customers, em-
ployees, and trade unions.
Knowledge of stakeholder importance
in training decisions has helped PwC tai-
lor evaluation methods. That’s a complex
To further complicate matters,
France, Germany, and the Netherlands
have tax systems that reward companies
when they invest in training. Those
countries offer a tax refund based either
on a percentage of payroll or on the
number of days each employee spends
in training—prescribed training that’s
based on a core curriculum rather than
on individual performance needs. That
reward leads companies to track training
input instead of output.
Performance data is another factor
that affects the feasibility of measuring
ROI. Whether you work in the Nether-
lands or the United States, to measure
the impact of training, you need data
that supports change in knowledge,
skills, performance, or business. While
consulting in the Netherlands and Ger-
many, PwC found that individual per-
formance data is often missing.
But that shouldn’t come as a surprise;
Based on our experiences, PwC
developed the following guidelines for
implementing evaluation:
Assess the organization’s readiness.
Before you start evaluating, use a readi-
ness assessment tool such as an interactive
dialogue. Send an email to organizational
stakeholders asking,
● Is performance transparent through-
out the organization?
● How do you measure individual and
team performance?
● What capabilities does the organiza-
tion have to measure those results?
● What data is available on individual
performance related to training or change?
● What decisions can you make based
on the evaluation outcomes?
● If there are no consequences of ineffec-
tive training, why bother measuring it?
● What barriers to measuring perfor-
mance do you expect?
Don’t just throw money at training.

Why ROI Isn’t Enough
A shareholder focus Ever since Donald Kirkpatrick intro-
duced the four levels of evaluation, acade-
may drive ROI mics and training professionals alike have
scurried to research articles, read books,
evaluation—but and attend seminars on evaluation and
stakeholders in the the return-on-investment of training.
Despite ROI’s popularity, the “ASTD
Netherlands want more. 2001 State of the Industry” report notes
that training and development profes-
sionals spend merely 6 percent of their
By Reinout van Brakel
time on evaluation. I’d hazard a guess that
the figure is lower in Europe.
In the Netherlands, for example,
university students do most of the
research on the effectiveness and impact
of training, and sometimes their efforts
result in training policy changes. The
real test comes after the students con-
clude their projects:
● Does the t&d department continue
the students’ efforts by measuring the
results of training regularly?
● Does the department attempt to
show ROI?
task, especially when you take into consid-
eration that in the Netherlands, and to a
lesser extent in Germany, the Rhineland
economic model has been predominant
for the past two decades. The Rhineland
model depicts a state governed by close
coordination of political partners, trade
unions, and industry associations. That
means that the relationship between busi-
ness strategies, HR policies, and perfor-
mance is moderated significantly by
institutions and stakeholders, both inside
and outside of the organization. Such col-
lective bargaining agreements and labor
laws prescribe, prohibit, and influence
HRM practices and policies.
you’ll find no “Employee of the Month”
in those countries because performance
doesn’t play a large role in their salary
and reward systems. In addition, setting
personal performance goals and making
individual contributions to a team effort
is often perceived as a threat. That
means less effort is spent on tracking
individual performance, which lessens
the evidence for impact of training.
Currently, t&d organizations in the
Netherlands don’t invest in systems that
could prove training’s impact because
even if you experience an increase in
sales due to a particular training initia-
tive, you don’t get paid better for it.
Change happens when you link training
to business strategies and measure the
impact of training—and when you have
stakeholder support. Because so many
stakeholders have a say in training mat-
ters, management is often prevented
from making decisions based solely on
ROI or shareholder value. A change
management approach that creates
awareness, knowledge, and acceptance is
essential when you want to make the
training function more results-focused.
Know the barriers and enablers. Richard
Swanson and Elwood Holton introduced
the concepts of performance drivers and
performance outcomes. They suggest

| T+DTDJune
6 72 | DECEMBER
2002 2010 TDJune 2002
THE BEST OF MEASURING & EVALUATING LEARNING | T+D73
| 7
 PA S S P O R T

By Paul Harris As companies look

The need to know the to e-learning to help

drivers and meet their strategic

goals, they’re seeking
HOW DO ORGANIZATIONSMOST outcomes of ROI analysis. Some
of the most valuable
COMMONLY MEASURE THE training only further measurements can be
RESULTS OF LEARNING, THAT IS,
endorses the need for hard to capture, but

a complete cycle
LEVEL 4 EVALUATION? THE MOST

ROI I
we’re closing in.
FREQUENTLY USED METRIC IS
CUSTOMER SERVICE, USED TO A of evaluation. n most departments within
HIGH OR VERY HIGH EXTENT BY a corporation, determining
that What enables performance? is a less
38.9 PERCENT OF RESPONDENTS’ threatening question to ask than What
the return on a given in-
vestment is a straightfor-
ORGANIZATIONS. MEASUR- was the performance outcome as a result of
training? Knowing the enablers provides ward accounting exercise
ING LEARNERS’ PERCEPTIONS managers with information on how to that produces a factual and
develop and support their people, which typically uncontested re-
OF PROGRAM IMPACT IS ALSO has the added benefit of increasing
sult. But when it comes to
stakeholder involvement.
COMMON, AS IS MEASURING Training evaluation in Europe usually e-learning, computing
consists of a lot of Level 1 (reaction) and ROI suddenly becomes a
PROFICIENCY AND COMPETENCY Level 2 (learning) measurement and complicated procedure re-
almost no measurement of Level 3 (job
LEVELS. performance), Level 4 (impact), or Level quiring thoughtful chin-

of E-Learning:
5 (ROI). Although Level 3 measurement stroking, serious seminar time,
is often perceived as a challenge, know- and earnest input from consul-
ing performance enablers and barriers is tants and vendors.
equally perceived as adding value.
The need to know the drivers and Why is that?

From the 2010 ASTD Research Study The Value of Evaluation:
Making Training Evaluations More Effective. Visit www.astd.
org/research for more info.
Closing In
outcomes of training only further There are reasons aplenty.
endorses the need for a complete cycle of Among them are that the
evaluation. That process includes con-
acquisition of an e-learning
ducting a needs assessment and setting
SMART objectives. Recognition of platform, courseware, or tech-
those forces will persuade t&d profes- nology produces an array of
sionals to make a shift from training bro- hard and soft cost-savings that
ker to performance consultant. often factor into the equation.
Reinout van Brakel is a training consultant
But not always. Every situation
at PricewaterhouseCoopers. He specializes in is slightly different.
Illustration by Bryan

measuring results of training; reinout.van.

brakel@nl.pwcglobal.com.

| T+DTDJune
8 74 | DECEMBER
2002 2010 THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 9
 ROI
H
ow much money can a company directly save by re-
placing its dependence on the classroom with a dis-
tance learning operation? It’s not rocket science. The
standard ROI equation calculates Return = Benefits
÷ Cost of the System.
Easily determined hard savings include reduction in
training budgets and materials, travel, instructors,
physical facilities, administrative time, and hours of lost
productivity when employees are off-site—among oth-
er savings. But what about improved productivity and
proficiency, learning curve, and employee retention,
satisfaction, and maybe even morale? They’re not al-
ways so easily measured. In some cases, they’re not even
relevant. Or they may be relevant to one level of the op-
eration but not to another. In addition, the increased
productivity of some departments, such as sales,
is more easily measured than that of others. And when
do you begin measuring those soft ROI factors? How
often do you do it? When do you stop, or do you?
How do you measure the impact of e-learning
when it’s part of a blended learning solution? How do
you get business units to buy into the concept so that
ROI calculations can be solidly based on their short-
and long-term strategic goals? (Wait. Where was that
seminar again?) And how do you ensure that ROI cal-
culations really determine the true value of e-learning
to an enterprise?
Fortunately for training professionals, there’s no
shortage of opinions or expertise. Indeed, an army
of e-learning vendors stands ready to assist any per-
plexed executive on the nuances of ROI and the
benefits to be gained by their particular products.
For example, SkillSoft dispatches representatives
twice a year on 30-city tours to provide workshops
on e-learning ROI. The free seminars focus on best
practices for analyzing the ROI of various types of
learning programs, business factors involved in ROI
analyses, tips on how ROI studies can help solve
business problems, and case studies. Attendees learn
how to design ROI studies in these typical areas of
cost justification: classroom expenses, employee
turnover, training duration, employee productivity,
and legal costs.
Visit any vendor’s Website and you’ll likely find a
white paper or two devoted to ROI issues. One such
document from LMS provider Plateau Systems
seeks to help the training manager sell the Plateau
enterprise system to senior managers in a language
they understand. It carefully explains the short- and
TDFebruary
1032| T+D | DECEMBER
20032010
long-term, as well as strategic and individual bene-
fits, and offers formulas to measure them.
And there is plenty to learn.
“Suddenly, the conversation on ROI is changing,”
says Brenda Benedet, director of e-learning strategies
for SkillSoft and a principal leader of its ROI work-
shops. “In the past, organizations merely looked at
cost-savings and benefits from an e-learning perspec-
tive. Today, as they position e-learning as a strategic
and majority element of blended learning, they’re also
looking for more tangible returns that relate to the
business—performance changes as well as the busi-
ness impact.” Adds Benedet, “Gone are days when at
the end of the year, organizations were content to re-
port x amount of e-learning usage. Today, they must
know if it made a difference. Hence, an interest in
ROI and multilevel measurement.”
Benedet says e-learning ROI is a vitally important
topic within companies and training departments face
challenges in making it relevant. “Historically, learn-
ing professionals were good at determining cost-effec-
tiveness and delivering that story based on utilization.
But as e-learning addresses business problems, they
need to look at actual impact. We see content now ad-
dressing specific kinds of business solutions, such as
call center training, sales training, IT certification, and
leadership. Those are strategic issues.”
Companies can measure performance and business
impact more aggressively by looking at the learning so-
lution in terms of the potential business impact and
potential ROI, says Benedet. She believes it’s important
that when conducting ROI of e-learning, corporate
trainers ensure they have appropriate stakeholders
within the organization on board. “It’s important to re-
member that learning professionals don’t own the busi-
ness problem or the data for measurement, so they
must be a partner with the unit. They must understand
what that unit’s business needs are, as well as the objec-
tives that the unit is being directed to deliver that year.”
Benedet points out that directors in training de-
partments tend to be close to the business but not a
true partner. “There’s the rub. Should an organization
want to implement a strategy for ROI, it must first re-
visit how its own learning organization is organized. It
should include ‘performance consultant’ as part of a
trainer’s job.”
Benedet suggests that organizations considering
implementing an ROI strategy start the process by
delivering an ROE (return-on-expectation) survey.
“Suddenly,
the conversation on
ROI is changing.”
Organizations position e-learning as a strategic and
majority element of blended learning; they’re also looking
for more tangible returns that relate to the business.
“It should be highly visible, and should pave the way
for future sponsorship and participation in real ROI
projects.” She advises companies to look at an oppor-
tunity and determine whether measures are in place.
“If not,” she advises, “do a return on expectations.”
Continual roadmap
Nucleus Research, a Wellesley, Massachusetts-based
research firm, keeps careful track of ROI issues
throughout e-commerce. It reported last fall that e-
learning solutions and e-business integration plat-
forms ranked among the ROI leaders; businesses such
as b2b marketing and stand-alone content manage-
ment ranked as ROI laggards.
Along with providing such analysis, Nucleus advis-
es companies on how to quantify their ROI. Its own
reports lean heavily on the hard costs and overhead
savings that are easier to obtain than productivity data.
Nucleus says that most companies gain significant re-
turns from even modest investments in e-learning
technology. “Most e-learning customers have quickly
recognized first-tier benefits, including reduced costs
for travel, customer support, human resources over-
head, and regulatory compliance—and eventually sec-
ond-tier benefits, such as increased employee
performance that directly impact profitability.”
Rebecca Wettemann, founding partner and senior
analyst at Nucleus, says that ROI analysis remains a
valuable roadmap for e-learning. “Calculating ROI
isn’t difficult; it’s merely structured,” she says.
Wettemann reminds that the typical CFO conducts
an ROI analysis on every other business component.
Similarly, the CIO must articulate the business benefits
of e-learning based on clear assumptions, she advises.
“The strategy for the CIO is to identify the benefits
and work to quantify them based on characteristics of
the organization, and to look at expected best-case and
worst-case scenarios.” She says companies can maxi-
mize the return on their e-learning investment by be-
ing as efficient as possible with the technology.
“Companies need to take a critical look at what
they’re trying to deliver. If they can use off-the-shelf
content, they should do that. If basic tracking is all
that’s needed, they shouldn’t buy an expensive LMS.
If they need high-fidelity learning, they’ll get the best
returns by developing the content. They should do
this across their learning goals.”
ROI analysis is a valuable roadmap to continually
measure the success of a company’s e-learning solu-
tion, notes Wettermann. “I do ROI analysis before I
deploy e-learning and once deployed, I use it to see if
I get the benefits I expected.” When analyzing bene-
fits, consider only those that have an impact on the
business and not those that offer only warm and fuzzy
feelings, she tells clients.
Kodak’s big KISS (keep it simple, stupid)
Sometimes an analysis of every conceivable ROI bene-
fit yields too much information, figures Catherine
Nowaski, director of e-learning, worldwide learning
TDFebruary
THE BEST OF MEASURING & EVALUATING LEARNING2003
| T+D 33
| 11
 ROI
and development at Eastman Kodak Company. In
fact, an ROI analysis should be as simple as possible,
she believes.
That’s the conclusion Nowaski and her colleagues
drew after pondering spreadsheets full of factors
deemed potentially relevant to the adoption of an en-
terprise-wide LMS at Eastman Kodak two years ago.
After examining the smorgasbord of hard and soft
measurements, including various models, a taskforce
opted for the squeaky-clean approach to a business
case for its proposed LMS candidate.
“We made a determination that the best way to
look at our investments was to ask what the cost for
classroom training is now, including vendor pay-
ments, as well as clerical and other overhead, and
what it would take for us to deliver the same or more
training in a self-study environment,” says Nowaski.
All other yardsticks were rejected.
“We didn’t even count things like time off the job,
even though that’s important,” she says. She adds,
“We looked at all of the other measurements and de-
cided they weren’t right for us.” Why? With 70,000
employees worldwide and countless training needs,
the size and scope of the project precluded the use of
most measurements, says Nowaski. “Vendor advice
became too complicated. We didn’t have the data.”
Soft ROI counts
Sometimes the adoption of e-learning is such an obvi-
ous choice and its benefits are so apparent that an
ROI analysis seems to write itself.
Such was the case at A.T. Kearney, a Plano, Texas-
based global management consulting firm owned
by EDS. With an army of consultants scattered
across 60 offices in 30 countries, it faces a constant
need to train new employees and keep veterans
abreast of best practices. It trained exclusively
in classroom settings until opting several years ago
for an even mix of classroom and online learning.
A.T. Kearney turned to Vuepoint, a Long Island-
based e-learning vendor that offers a single platform
that combines an LMS, an LCMS, and collabora-
tion technology.
From a training standpoint, A.T. Kearney had
three distinct problems, explains Al Morrison, vice
president of global professional development. For
starters, its consultants spend most of their time on
location with clients, making classroom attendance
difficult. Two, the rapid pace of technological
1234| T+D | DECEMBER
TDFebruary 20032010
change presents a challenge in keeping employees
current on the latest offerings, delivery mechanisms,
and other issues. Expensively produced learning ma-
terials were typically obsolete before everyone could
take the class. Last, the wrong people tended to
show up for the training. People who were properly
grounded and ready for training were always out-
numbered by colleagues not yet grounded and qual-
ified or others who already knew the material.
Online e-learning solved all three problems, says
Morrison. It solved the disbursement problem by
enabling the company’s consultants to access courses
anywhere for use on their laptops. It stayed abreast
of technological change with just-in-time courses
that are so inexpensively produced they’re consid-
ered a throw-away. And the e-learning courseware is
self-selecting, weeding out employees for whom a
course is inappropriate.
“We made our investment in Vuepoint without
considering ROI,” says Morrison. “We went back
and said, ‘This is a good idea and will pay for itself,
but let’s run some numbers and see what happens.’”
A.T. Kearney discovered that it could develop pro-
grams for half of what they used to cost, and that de-
velopment time dropped to 44 percent of the
traditional approach. The firm reported savings of
more than US$20 million with more than 4000
days worth of Web-based training within a 12-
month period.
Ara Ohanian, CEO of Vuepoint, agrees that im-
proving proficiency and the learning curve are key
ROI benefits of e-learning that companies shouldn’t
discount. Although such benefits may appear soft to
a training director, they most definitely are not re-
garded as such by the executive in charge of any
P&L area, he says.
Ohanian cites another example, Vuepoint client,
the homebuilder David Weekley Homes. A line
manager concluded one day that increased sales
would result if the company could improve the satis-
faction of its customers in the post-purchase phase
of the relationship. After all, that’s where most refer-
rals come from, he reasoned. J.D. Power & Associ-
ates was enlisted to measure client satisfaction, and
an e-learning program was adopted to address the
areas of concern with employee training. David
Weekley Homes is currently enjoying its most prof-
itable year, due in part to that effort, the company
claims. “It began by measuring the particular thing
Sometimes the adoption
of e-learning
is such an obvious choice
and its benefits are so apparent that an ROI analysis
seems to write itself.
that the business person knew would have an im-
pact,” says Ohanian.
Another soft but critical ROI component in-
volves speed. Not only is time-to-market crucial for
most sales organizations, but in this era of frequent
product changes, relaunches, and complex product
mixes, the rapid transfer of knowledge to the sales
channel is extremely vital.
The Hartford Company asked Vuepoint if it could
help shorten the six-month period needed to fully
train its sales channel about new products. With adop-
tion of the Vuepoint platform, that time was reduced
to two weeks. “Speed-to-knowledge may be a soft
measurement, but the impact is greater than any travel
reduction would ever be,” says Ohanian.
Improving ROI
If there’s anyone who still classifies performance im-
provement as a second-tier benefit of e-learning,
don’t tell that to Brian Carlin. He’s president and
CEO of Maritz Learning, a Fenton, Missouri-based
provider of custom training, e-learning, and perfor-
mance management for sales and service organiza-
tions. Maritz Learning helps companies ensure that
e-learning is the means to a profitable end.
Maritz Learning, a unit of Martiz, a US$1.3 bil-
lion provider of performance improvement, re-
search, and travel services, views performance
improvement as a three-legged stool: 1) training and
professional development, 2) motivation through
rewards and recognition, and 3) performance mea-
surement. The ultimate goal of the integrated ap-
proach is to get “energized competence” out of
employees as measured by the approach’s impact on
a client’s top line, not its expense line.
By marrying performance measurement with
professional development, Maritz goes well beyond
the average e-learning vendor. A case in point is tele-
com client Sprint, which sought increased perfor-
mance from its outbound call center. Maritz
Learning began by measuring the performance of
Sprint’s sales reps in a dozen core competencies, fol-
lowed by a 360-degree skills assessment to deter-
mine how they ranked. Following a gap analysis, it
prescribed training interventions and developmen-
tal activities, using its own LMS and courseware.
Part 2 involved customer satisfaction. Calling on
Maritiz’s research business unit, Maritz Learning
conducted follow-up surveys to measure satisfaction
ratings against specific competencies. The resulting
interactive report detailed each call center rep’s skills
assessment rating and customer satisfaction outlook.
Maritz Learning also helped office furniture maker
the Hon Company double its sales with an e-learn-
ing program for its network of 500 nondedicated
dealers and 8000 representatives, developing the
Hon Masters Program to reward reps for selling
Hon products and successfully completing specially
designed Hon product training modules.
Carlin believes that the growing ability of Maritz
and other vendors to help their customers meet their
strategic goals—and to effectively measure e-learning’s
impact—will increase the corporate world’s reliance
on e-learning to solve training needs. That message is
resonating with customers, he says. And the value of
ROI analysis is only beginning to be felt. TD
Paul Harris is a freelance writer based in Alexandria,
Virginia, and frequent contributor to Learning Circuits
GO WWW.LEARNINGCIRCUITS.COM; pharris307@aol.com.
TO
THE BEST OF MEASURING & EVALUATING LEARNING2003
TDFebruary | T+D 35
| 13

Donald Kirkpatrick
Interview by Diederick Stoel
Donald Kirkpatrick
and Jack Phillips,
renowned for
their work and
contributions in
the evaluation arena,
sparred at the ASTD
ROI Network
Conference in
Amsterdam,
organized by the
ASTD ROI Network
in cooperation with
ProfitWise.
The champions’
mutual respect was
apparent, yet
controversy
wasn’t avoided.
In fact, an old score
was settled.
1446| T+D | DECEMBER
TDJanuary 2004 2010
The Evaluation
Heavyweight
This online article has been modified since
the print version.
To get straight to the point: People all over
the world have used the Kirkpatrick four-
level framework. Why has that framework
been so successful?
Kirkpatrick Before I came up with the
four levels in 1959, evaluation was a gen-
eral thing, an all-encompassing thing.
People talked about evaluation in an elu-
sive way. The main thing I contributed
was breaking down the subject of evalua-
tion into meaningful terms. So now
when we talk about evaluation, every-
body’s talking about the same thing: reac-
tion, learning, behavior, and results.
Phillips In the late fifties, evaluation was
like rocket science. You knew you had to
be accountable for training, but no one
really understood how you did that. Don
brought structure to it. That’s why he’s
highly recognized. He’s in the HRD Hall
of Fame, and he has been recognized by
ASTD for his work, which is a major con-
tribution to our field. He gave us all a clear
roadmap to evaluation.
Yet, Kirkpatrick’s four-level framework has
been widely criticized. The fiercest criti-
cism came from Ed Holton in 1996,
which stated that the four levels are
flawed—that it’s not a model because
there’s no causal relationship between the
various levels.
See page 4 for an Executive Summary of this article
Phillips I see no value-added from research
that shows correlations between the four
levels. It adds little insight into learning and
development. Don has been criticized by
many academics, and I think it’s jealousy.
Kirkpatrick When I read [Holton’s] arti-
cle, I thought: I don’t think I ever called it
a model. I just called it the four levels. So
someone criticized me and said, “It’s not
a model; it’s a taxonomy.” I thought to
myself: So what? I don’t care if it’s a tax-
onomy or a model. It’s four levels, and
people find it helpful.
The four-level framework set the standard
almost 40 years ago and inspired many
people. The framework of Jack Phillips’s
ROI model is based on the original frame-
work. We ask Phillips when he used the
model for the first time.
Phillips Our first ROI study wasn’t as
thorough as our current studies. It was
pretty rough. But, we started. I’ve been
using Kirkpatrick’s framework right from
the start, and it has been very helpful to
me all these years. We added a fifth level.
More importantly, we created a process
model to collect and analyze data. We
brought standards to the process and
tackled the tough implementation issues.
Why did you add a fifth level?
I know Don doesn’t agree with
Phillips
me on this. He puts ROI in Level 4. I
Match
did too for a while. But improvement in
a Level 4 measure may have a negative
ROI. For example, absenteeism could
be reduced after a solution. But if you
spend too much money on the solution,
you can end up with a negative ROI.
We found that confusing, because we
have success and lack of success on the
same level.
So, we said, “Maybe we can look at
this as a different level.” The fifth level
requires you to always bring in the costs.
Kirkpatrick Yes, I think a good example is
a sales training program. You can have a
sales increase, but what was the cost of
increasing the sales? So, ROI is certainly
an addition to my model.
Many people see evaluation as something
to be done after training. But the art is to
arrange your training in such a way that
success is guaranteed. We launch a series of
questions to see whether we can build a
chain of impact up front: How can we
make sure people learn? How do we ensure
people use their newly acquired skills
on the job? How do we know that the
application of these skills returns the
desired effect?
Kirkpatrick That’s very simple. Let me
give an example. At Intel, they said,
“Let’s take your four levels, start with
the last one, and work backwards. What
results are we looking for? What behav-
iors are needed to accomplish those
results? What knowledge, skills, and
attitudes do people need in order
to behave in that way? And how can
we do it in such a way that they’ll react
favorably?” Answering those questions
in that order is at least one less
headache.
Phillips We see more people shifting to a
proactive posture regarding evaluation,
instead of evaluating because someone’s
asking for it. Try to understand what
your training program might add to the
business by starting to think about
results and ROI up front.
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 15
Jack Phillips
Kirkpatrick Now there’s one thing I’m
concerned about, Jack. It’s the constant
emphasis on ROI. People start looking
for a simple answer and kind of bypass
the other ones. Let me tell you a story:
I did a program for a client that pub-
lishes canned programs. The client asked,
Don, tell us, how do you measure results?
We want anybody that buys our package
programs to be able to measure the
results. I said, “It isn’t that simple. Let me
go through the different processes.”
I was scheduled there from 9 a.m. to
12 and from 1 till 2 p.m. At about 11:30,
I’d gone through the knowledge needed
to do the evaluation. So, now let’s talk
about how we can apply this knowledge,
I said. The client stopped me right there
and said, “You can go now. We’ll have a
debriefing meeting.” So, I went out into
the hall and waited for her to come out
and tell me how to proceed. But she
never did. So I just took off.
Phillips(laughing) I recognize that quest
for a magic answer. People are attracted to
TDJanuary 2004 47
E va l u a t i o n

Measurement
ROI because they think it’s Kirkpatrick Phillips most information in the short-
the answer to all account- est period of time. That’s diffi-
ability issues. Let’s put it Level 1: Reaction Level 1: Reaction, cult enough as it is. If you look
into perspective: When you Satisfaction, at the reaction sheet of the ho-
think of an impact study for Planned Action tel here, see how detailed and

a program, with 75 pages,
there may be three pages on
ROI. ROI is only one mea-
sure of six—the four levels,
ROI, and the intangibles.
So, we have to keep remind-
ing people that ROI is only
one measure that demon-
strates the value of training.
Don’t overreact to it or
overuse it.
Kirkpatrick has been known
to announce in his presenta-
tion that he’s going to sue
Phillips because he altered his
model. Phillips not only
added a fifth level, but he also
changed behavior into appli-
cation (Level 3) and results
into business impact (Level 4).
What’s the story?
Kirkpatrick (turning to Phillips) Why
didn’t you like behavior?
Phillips (smiling) I’m glad you brought
that up.
Kirkpatrick and Phillips have known
each other for many years, but this partic-
ular matter has never been brought up
before. Their appearance together in Ams-
terdam was only the second time on stage
together. Phillips takes up the gauntlet.
Phillips It’s simple. The concept of the
four levels doesn’t only work in training;
it works in many other settings. We
used it in technology groups, and they
love the four-level framework. But they
relate better to application than to
behavior, for Level 3.
Level 2: Learning Level 2: Learning
Level 3: Behavior Level 3: Application
Level 4: Results Level 4: Business Impact
Level 5: ROI
Kirkpatrick I understand. In their cul-
ture, the word application is more mean-
ingful than behavior.
Phillips The same is true for results.
Business impact means more to them
than results. So, I just stepped back and
said, “We need some words here that
can fit quality, technology, Six Sigma,
marketing, even policy changes. I tried
to be more generic.
We ask Kirkpatrick if he’s still thinking of
suing Jack.
Kirkpatrick (grinning) No, I guess I’d
probably lose.
What does it take to become an effective
evaluator?
Kirkpatrick Start to develop a simple
reaction sheet on Level 1. Try to get the
how much time it would take?
You’d need to have a PhD to
answer those questions.
Phillips I agree. You don’t have
to be a statistician to be a good
evaluator. Preferably not. If we
make evaluation too instru-
mental and technical, we
become isolated. Be involved
as much as you can, and make
sure you know the practical
ends. Statistics aren’t that
important; communication is.
We need good communica-
tion skills—both spoken and
written. Because whatever we
do, we’re going to present it to
different stakeholders. If we’re
capable of doing that, we really
demonstrate our value-add.
How do the two men see their futures?
Phillips will continue to educate through
his workshops, conferences, and publica-
tions. “Publish, consult, teach,” he says.
Kirkpatrick just formed a partnership
with his daughter Sue and son Jim, called
Dr. Donald Kirkpatrick & Associates.
He’ll continue communicating the four
levels, with the help of his overhead
sheets. He doesn’t use PowerPoint or
attachments, even though he admits his
son Ted has tried to show him how to 100
times. Kirkpatrick prefers to type what he
has to say directly in his email. “I always
say that if it ain’t broke, don’t fix it.” TD
Diederick Stoel is CEO and president of
ProfitWise, Amsterdam, the Netherlands. Its
mission is to make training and development
human resource management programs a
rewarding enterprise.
Gets Strategic
Abraham Maslow said, “If the only tool you have is a
By Paul Bernthal hammer, you tend to see every problem as a nail.”
Has ROI become that hammer?
FOR MANY YEARS, measurement and evaluation have been
treated as afterthoughts in the workplace learning and performance
community. We measured basic needs, but most of the subsequent
activity focused on ensuring a good implementation. If the subject of
evaluation came up, we discussed it as a method for selling the pro-
gram, ensuring future funding, or trying to make a link to the business.
We knew we influenced the bottom line, but, because we often didn’t
control strategy, attempting to document results was a daunting and
sometimes risky task.
Today, the profession has made significant advances in redefining the
role of the WLP professional, most notably with ASTD’s 2004 competency
study, “Mapping the Future: Shaping New Workplace Learning and Per-
formance Competencies.”
We are becoming more strategic and are defining clear value proposi-
tions for organizational success.The roles of learning strategist and busi-
ness partner are supplementing the important roles of professional
specialist and project manager. The effects of those changes are felt
everywhere, including the area of measurement and evaluation. ASTD’s
new model requires professionals in most areas of expertise—or AOEs—to
conduct needs analyses, monitor progress, evaluate outcomes, and pro-
vide feedback for change.
WLP professionals have learned new methods for measurement and
evaluation and are applying them in different contexts. Examples detail-
ing how a project manager or professional specialist can tactically apply
measurement methods are plentiful. But what about strategic approaches
to measurement?
Measurement is far more than a set of methods. It’s an integrated part
of how WLP professionals partner with stakeholders and implement
their

Copyright ASTD, May 2005 TDMay 2005 53

48 TDJanuary 2004
16 | T+D | DECEMBER 2010 THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 17
MEASUREMENT

strategies. Without the strategic perspec- strategies for improving the business. sors about why their engagement scores
tive, measurement tools tend to Organizational leaders determined that were high or low and address any out-
define rather than facilitate our solutions.one way to promote growth was through standing issues. Divisional HR managers
building engagement in its team mem- met with regional managers to discuss ac-
Lead indicators bers. Director of Training and Develop- tivities that would further engage team
Most organizations use some form of an ment Tom Labadie knew that employee members and build a winning culture,
employee satisfaction or culture survey to engagement was significantly related to and created action plans. In a year, Com-
gather valuable feedback for change and increased customer satisfaction. Another pUSA plans to re-administer the engage-
serve as an indicator of the WLP program’s benefit was long-term retention of team ment measure to track its progress.
influence. Recent studies have refocused members, who gain experience and build Everyone at CompUSA, from the
many of those surveys on the concept of stronger relationships with customers. president and CEO down, is linked to the
employee engagement, which is a proven Labadie’s philosophy was that engage- idea of creating a winning culture. Labadie
driver of employee loyalty and discre- ment drives customer satisfaction, and describes how, when you walk into stores
tionary effort. Those outcomes lead to the tool that CompUSA used allowed with high engagement scores, you can
higher performance and retention. En- them to track progress and target action sense the positive tone. Employees whistle
gagement surveys, in their current form, plans for WLP programs. and smile; they approach customers; and
represent the next generation of macro- After field-testing an employee the store gives off that elusive approachable
level measures of WLP success. Through engagement survey with a small group of feeling that customers appreciate.
proven statistical relationships to out- stores, CompUSA administered the survey From a measurement standpoint,
comes—such as turnover, productivity, nationwide to more than 200 stores. That Labadie feels that the engagement mea-
and profitability—engagement surveys are measure—Development Dimension Inter- sure is an excellent way of tracking the in-
an obvious lead measure of organizational national’s E3 survey—provided a baseline fluence of WLP programs. Engagement
for engagement and gave links to company strategies (developing
Tom Labadie, director of training and managers and employees a “winning culture” and growth), tracks
a common reference progress over time, and has a direct link to
development for CompUSA, has a theory: point. WLP programs. In addition, the common
Essentially, the engage- reference point builds accountability for
Engagement drives customer satisfaction. ment survey built a sense managers and offers a uni- versally val-
of shared accountability ued measure of progress. Everyone, not
in the workforce and just HR, uses the same measure to focus
made obvious links to effort.
potential WLP solutions. Return-on-expectations
At the HR level, WLP Author Dan Tobin writes, “If you start and
outcomes. While it might be difficult professionals used the engagement re- end all of your learning efforts by focusing
to link WLP programs directly to an org- sults to build on your organization’s goals, you will nev-
anizational outcome, it’s much easier a strategic training plan. The training er be asked to do an ROI analysis to justify
to link them to the level of employee function met internally to review the your budget.”
engagement. numbers and determine priorities relative The topic of return-on-investment
The content of most employee engage- to engagement scores and other issues. is popular, and there are many books,
ment surveys addresses issues such as Regional HR managers also set goals con- articles, and resources that give specific
goal alignment, collaboration, support and cerning the changes they would pursue. step-by-step instructions on how to ad-
recognition, and development. All of those Regional HR directors then partnered with minister the appropriate analyses. While
work environment qualities are direct cor- regional managers to set improvement ROI has its place, some would argue that
relates and outcomes of most WLP pro- goals for engagement. The hope was that ROI studies for specific programs aren’t
grams. In effect, measures of employee WLP-related improvements would affect necessary if the program being evaluated
engagement serve as proxy measures for team success and that scores would im- is appropriately selected, positioned, and
organizational success that have a direct prove. implemented. Intangible benefits can be
link to WLP programs. Most traditional Managers took feedback from the quantified, but is such a calculation neces-
measures of employee satisfaction or cul- survey and developed action plans at the sary if the program plays a clear role in the
ture don’t include that value proposition individual level. The engagement scores realization of a company strategy? Has ROI
link to the bottom line. were examined during performance become a hammer in search of
reviews. To avoid a “rank and spank” men- a nail?
CompUSA study tality associated with store performance Most organizations conduct ROI stud-
At CompUSA, the goal was to focus its HR in some retail organizations, managers ies to demonstrate the value of a program
strategy by aligning it with companywide were prompted to talk with their supervi- to internal customers or to choose be-
tween two or more potential paths
of action. However, little research has
been conducted to demonstrate that
ROI results create compelling cases or
shift opinions. In fact, one controlled lab-
oratory study conducted by University of
Toronto professors Gary Latham and
Glen Whyte found that ROI studies—or
utility analyses—actually reduce the
support of a manager for implementing
a valid selection procedure.
Those findings don’t mean that we
shouldn’t conduct ROI studies. Rather,
we should be more careful about how we
use them and understand when the situ-
ation warrants such an approach. WLP
professionals need to work with internal
customers, such as management and
senior leaders, to create measurement
contracts that clearly outline program
expectations or objectives and how
they’ll be tracked. Program success
should be evaluated in terms of the
agreed-upon expectations and how well
the program accomplished those objec-
tives. That return-on-expectations—or
ROE—approach is a more appropriate
and broadly applied method than ROI.
WLP professionals need to play the
role of business partner to manage
expectations and develop a realistic ap-
proach for measurement. Every perfor-
mance improvement solution should be
targeted toward a specific, measurable
change. When discussing possible solu-
tions with internal clients, WLP profes-
sionals will need to ask three key
questions:
● What is the solution intended to
change?
● Is it realistic to assume that the solu-
tion can produce the change?
● Can we objectively isolate and mea-
sure the impact of the solution?
If an expectation can pass those three
questions, it’s added to the measure-
ment contract. Essentially, a measure-
ment
contract outlines what will be measured
and how. Using that process ensures that
measures are realistic and accountabili-
ties for impact are clear.
Wyeth Consumer Healthcare study
During the past few years, Wyeth evolved
from being a holding company (Ameri-
can Home Products) into an integrated
health-care company that’s focused on
original research and development. The
organization developed a new mission,
vision, and values, and created a focus on
the people, not just the numbers. For
Wyeth Consumer Healthcare, a division
of Wyeth, attracting, developing, and re-
taining talent has been a “top 5” priority
for the past two years. Today, senior lead-
ers buy into the idea of human capital
and focus explicitly on driving a talent
management strategy. Recent changes
include a direct reporting relationship
between the HR executive and the presi-
dent. HR is building strong relationships
with strategic leadership and moving
away from a transactional role.
This year, WCH’s competency devel-
opment and interviewing training team
won an internal Global Excellence Award
for their work evaluating changes in
the interviewing process. That award is
given to up to 10 teams each year that
take action to produce a significant busi-
ness impact. The project didn’t include
an ROI study, but it did an excellent job of
managing customer expectations and
providing measurement outcomes in
line with program expectations.
The selection process at WCH had
been working well, but the team recog-
nized an opportunity to improve the sys-
tem: They used Development Dimension
International’s Target selection system to
help interviewers become better pre-
pared to conduct valid candidate in-
terviews. The team worked to establish
expectations for changing the system
and held meetings with senior leaders to
provide examples of successful inter-
viewing processes at other organizations
to gain their buy-in (influencing stake-
holders). They also made links between
the overall talent management strategy
and showed how better preparing inter-
viewers would promote the strategy
(thinking strategically).
Fortunately, the organization had al-
ready created the business case for the
value of a competency-based HR system.
Senior HR leaders were receptive and
saw competency-based interviewing as a
natural step in the process. The team
wasn’t pressured to document ROI or
link to outcomes (such as turnover) be-
cause it was clear that the program sup-
ported the business strategy. Their
agreement for measurement was ap-
proved in advance and included achiev-
able and realistic goals. Outcomes of the
study included documented improve-
ments in interviewing skills, confidence,
and quality of the interview process.
The measurement process at WCH
included the application of both strategic
and tactical measurement tools. Rather
than assuming that ROI or other analyses
would be appropriate for the situation,
WCH took the time to manage stake-
holder expectations and identify links to
the corporate strategy. In that way, the in-
dividuals involved create realistic and
valid measures that accurately reflect the
desired outcomes of the program.
HR scorecards
In recent years, several important mod-
els—such as the HR Scorecard created
by Brian Becker, Mark Huselid, and David
Ulrich and the Service Profit Chain
developed by James Heskett, Thomas
Jones, Gary Loveman, Earl Sasser Jr., and
Leonard Schlesinger—have shown a
strong relationship between people,
HR practices, and company performance.
Those models created a clear value
proposition that allowed WLP profes-
sionals to link their solutions to strategic
outcomes.
Effective implementation of WLP so-
lutions act as lead indicators for subse-
quent customer and financial outcomes.
HR scorecards typically track multiple
lead and lag measures, such as employee
engagement, turnover, implementation
successes, program costs, ROI, diversity,
and cycle time. The list of possible mea-
sures is endless and varies greatly de-
pending on the organization’s strategic
goals. The advantage of the scorecard is
its ability to position WLP professionals
in the learning strategist role by outlin-
ing links to the business strategy.
As in all good research studies, multi-
ple methods and measures offer a more
complete picture of overall impact. It’s
difficult to relate the effect of a single
solution to the bottom line; it’s easier to

Copyright ASTD, May 2005 Copyright ASTD, May 2005 TDMay 2005 55
54 TDMay 2005
18 | T+D | DECEMBER 2010 THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 19
MEASUREMENT
show how a solution leads to long-term
performance. Essentially, WLP profession-
als who use a scorecard minimize the
need to prove bottom-line impact for
every solution they offer. In fact, many so-
lutions were never intended to have a di-
rect impact on company performance,
and HR scorecards can show how those
programs fit into the big picture.
HCA study
Operating nearly 200 hospitals and a range
of surgery and ambulatory care centers in
24 states, as well as the United Kingdom
and Switzerland, Hospital Corporation of
America is a large and dispersed organiza-
tion. It’s important for the company to co-
ordinate WLP professionals at individual
locations and unify their efforts toward a
common set of metrics. HCA has been us-
ing an HR scorecard—or “dashboard”—for
the past four years, after it was initiated in
response to a recruitment and retention
task force.
The scorecard has been through
several iterations over the years. Current-
ly, the scorecard focuses on a mix of lead
and lag measures, such as labor costs,
THE MOST COMMON WAY TO MEASURE CHANGES IN JOB BEHAVIOR
IS WITH FOLLOW-UP SURVEYS OF PARTICIPANTS, USED TO A HIGH OR
VERY HIGH EXTENT BY 31.0 PERCENT OF RESPONDENT COMPANIES.
THESE SURVEYS ARE ALSO SIGNIFICANTLY CORRELATED WITH THE
EVALUATION SUCCESS INDEX (ESI).
From the 2010 ASTD Research Study The Value of Evaluation: Making Training
Evaluations More Effective. Visit www.astd.org/research for more info.
56 TDMay 2005
20 | T+D | DECEMBER 2010
productivity, employee survey scores, and
labor supply. Eventually, it will be en-
hanced to include more detailed mea-
sures of financials, customer satisfaction,
quality, and operations.
Donna Yurdin, assistant vice pres-
ident of organizational effectiveness,
points out that the HR scorecard has
been instrumental in increasing the
visibility of HR and its impact on the
business. Most important, the scorecard
helps to focus the organization on
opportunities for improvement and
prioritizing the levers for change.
In the health-care industry, market
changes and governmental regulations
are always affecting business priorities.
The scorecard keeps HCA focused
on what’s important for supporting
human capital effectiveness. Yurdin’s
work with the HCA scorecard illustrates
how WLP professionals can focus on dri-
ving results and applying business acu-
men when creating a value proposition
for HR.
For example, the organization was ex-
periencing higher levels of turnover in the
six to 12 month new-hire period. HCA
Copyright ASTD, May 2005
added a tenure-based breakout of
turnover to the scorecard and introduced
improvements in selection and onboard-
ing to address the need. The dashboard
served as a lag measure for onboarding. In
the future, HCA could evaluate the imple-
mentation of selection and onboarding
changes to predict future changes in NINE DIFFERENT FACTORS WERE SIGNIFICANTLY
short-term turnover.
Although HCA hopes to improve and CORRELATED WITH THE EVALUATION SUCCESS
expand its scorecard, Yurdin has a lot of
advice for organizations considering their INDEX. WE RECOGNIZE, HOWEVER, THAT MOST
own HR scorecard: “There needs to be a
collaborative effort between HR, business
ORGANIZATIONS CAN UTILIZE ONLY A FEW
operations, and department heads to find
external measures allowing you to gauge
MEASUREMENT TECHNIQUES AT ANY ONE TIME.
your progress. It all comes down to the SO, WHICH APPROACHES, IF AN ORGANIZATION
scorecard needing to speak to the organi-
zation. You can’t just take an example MUST CHOOSE, LEAD TO THE BEST EVALUATION
from another organization and make it
work. Your scorecard needs to be reflec- RESULTS? BASED ON A FURTHER ANALYSIS OF
tive of the business culture and serve as a
living document that changes over time THE DATA USING MULTIPLE REGRESSIONS, THE
as your business changes.” TD
ASTD/I4CP TEAM SUGGESTS THAT
Paul Bernthal is the manager for DDI’s Center for
Applied Behavioral Research; paul.bernthal
ORGANIZATIONS FOCUS ON THE FOLLOWING
@ddiworld.com. THREE FACTORS:
FOLLOW-UP FOCUS GROUPS
PERFORMANCE RECORDS MONITORING
FOLLOW-UP SURVEYS OF PARTICIPANTS
From the 2010 ASTD Research Study The Value of Evaluation: Making Training
Evaluations More Effective. Visit www.astd.org/research for more info.
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 21
 O R G A N I Z AT I O N A L C H A N G E
The C-Level
and the Value
of Learning
A groundbreaking study LEARNING IS BECOMING an increasingly im-
portant function for all companies to develop
examines CXO-CLO and maintain the individual and organiza-
alignment and tional skills needed to create a competitive
advantage, increase efficiencies, and improve
disagreement bottom-line results. IBM’s 2004 Global CEO
on the strategic Study revealed that CEOs are focused on accel-
erating growth, enabling transformation, and
value of learning. increasing productivity. They also recognize
that employees’ development is essential to
achieving these goals. In fact, 75 percent be-
lieve that employee education is critical for fu-
ture enterprise success.
As a result, chief learning officers and other
learning executives are increasingly being
called upon to play a more strategic role in or-
ganizations. But the question remains: Do their
strategies for maximizing the impact of learn-
ing align with what CXOs, the C-level business
executives, perceive as valuable? How should
the corporate learning function respond to the
strategic business needs of growth, transfor-
mation, and productivity? How should it be
held accountable for adding value to the busi-
ness? And, most important, are CXOs and
CLOs in sync with the role that learning plays
in responding to changing business needs and
how that response is measured?
To address these questions, IBM and ASTD
Research recently partnered on a study to in-
vestigate the CXO mindset and how it com-
pares to that of CLOs. It was the first known
study to specifically examine the alignment of
C-level and CLO perceptions around the strate-
gic value of learning.
The study included in-depth interviews
with C-level executives and CLOs at 26 organi-
zations across 11 industry sectors. At each or-
ganization, a CXO and CLO were interviewed
By Tony O’Driscoll, separately and asked the same six questions
Brenda Sugrue,
and Mary
22 | Kay
T+D | Vona
DECEMBER 2010
Copyright ASTD, October 2005
strategic thought process and data flow of
executives.”
By contrast, CLOs emphasized chang-
approaches, and infrastructures that
simultaneously respond to enterprise-lev-
el strategic needs as well as business unit
Translating strategic activity
into business results
Learning is viewed by both CXOs and CLOs
ing the learning function to serve the com- operational needs and individual develop- as having significant impact on business
pany at the business unit productivity ment needs. results, including revenue, productivity,
level. One CLO explained, “Ninety-five per- turnover, and innovation. These business
cent of our learning is technical and fo-
cused on skills and performance. At a
basic level, training allows our organiza- Strategic Learning Learning
tion to operate.” CLOs also indicated that Business Needs Function Response Function Performance
they are moving beyond learning to focus
on talent and performance issues related
Accelerate Alignment with Strategic Strategic
to the current needs of business units. Ac-
Growth Business Needs Alignment Indicators
cording to one participant, “Learning will
engage early to analyze and improve the
design of jobs, processes, and technology-
focusing on the learning function’s response to
based performance support tools.” Enable Articulation of Business Business Value
business needs and its performance relative to
that response. The questions were based on
The study’s findings suggest that Transformation Value Contribution Contribution Measures
to more closely align with CXOs on
the assumptions that learning needs to accel-
the strategic value of learning, CLOs
erate growth, enable transformation, and in-
crease productivity (chart 1).
should articulate learning’s value at the
enterprise, business unit, and individual Increase Learning Learning Function
Then, the answers were compared to deter-
capability levels. Productivity Investment Process Productivity Metrics
mine where perceptions aligned and where
they did not.The findings suggest that there are
CLOs should also ensure that their
senior executives are aware that they
significant opportunities for CLOs to align more CHART 1
are developing learning strategies,
closely with C-level expectations and aspira-
tions, to make learning more strategic and cen-
tral to the ongoing success of the enterprise.
Aligned Q1. How does the learning function provide strategic
Learning as a provider of Varying
strategic value to organizations value to your organization?
According to the study’s results, both CXOs
and CLOs think learning provides strategic val- Strategy 14
12
ue in many ways. These value propositions fall Implementation
into three categories: enterprise strategy, busi- Transformation 13 8
ness unit productivity, and individual capabili-
Enterprise
ty. (Chart 2 indicates the number of times that Globalization 8 7 Strategy
CXOs and CLOs mentioned particular aspects
Innovation 6 2
of learning’s strategic value. The green dia-
monds show areas where there was close
alignment, and the red diamonds represent
variance in responses.) Talent Management 6 10
CXOs emphasized learning’s increasingly Business Unit
BU Enablement 4 10
strategic role in changing the business at the Productivity
enterprise and individual capability levels. Performance 2 9
They see learning’s strategic value as building Improvement
the capability necessary to address the future
challenges of the enterprise. For example, ac- 9
Leadership 9
cording to one CXO participant, “Learning is CXO Individual
the major investment to build the capability to 13 4 CLO Capability
Capability
drive the business forward.” Some CXOs also
expressed a desire for the learning function to
be more proactive. One participant said, “The
learning function should lead, not respond. It
0 5 10 15 20 25 30
should be proactive and be included in the CHART 2 Frequency
TDOctober 2005 71
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 23
Copyright ASTD, October 2005
 O R G A N I Z AT I O N A L C H A N G E
Aligned Q2. How does the learning function’s strategic activity
Varying
translate into business results?
evidence of learning’s strategic value. For location seek to optimize the learning in- models that seem to incorporate the best
Business Outcomes 12 22
Business Productivity example, one CXO said, “I am not con- vestment relative to the needs of the of both worlds.
15 18
5 8 Organization cerned with separating out the business business. To more closely align with CXOs
CSAT 6
Outcomes value attributed to learning.” Another ● Integrated learning investment plan- around learning investment processes,
Culture 5 5
Transformation 7 2 7 said, “We don’t have any hard metrics ning ensures that enterprise, business CLOs should engage senior leadership in
Turnover 5 3 around effectiveness and ROI. I would unit, and individual learning require- the development and financial support of
2 rather have an ROI measure, but for now I ments are captured and addressed. future-focused enterprise level learning
Capability 5 10 take it on faith.” Yet another advocated a ● Learning investment allocation lever- strategies. Companies should recalibrate
Human Capital
Time to Performance 2 5 compromise: “You can’t measure every- ages councils to ensure balanced invest- their learning function’s organizational
3 thing. Some programs should have ROI, ments across enterprise, business unit, structure to include a balance of central
Learning Innovation 5 14 but others you just have to invest in on and individual learning needs. and local decision making and control as
Learning Productivity 3 11 Learning faith.” ● Learning investment management appropriate. There are many variations of
Usage 5 5 Operations The results also indicated that CLOs ac- seeks to optimize control of the invest- centralized, decentralized, and hybrid
Performance Improvement 2 5 9
14 knowledged that CXOs are not as con- ment via centralized, localized, or hybrid structures. CLOs need to find the most
cerned about ROI and isolating learning’s management. cost-effective and efficient structure for
Challenge 11 5
9 contribution to business results. For exam- Chart 4 shows the number of times their particular business context.
Learning Value Perception 5 9 CXO Learning Value ple, according to one CLO participant, “Our that each group mentioned various as-
ROI 1 5
CLO
Comparison 1 5 13 4 Measurement efforts in tracking ROI for learning are pects of learning investment planning, al- Alignment of learning with
greater than the demands of our senior ex- location, and management. strategic business needs
ecutives.”Another CLO said,“We’re striving According to the study, CXOs empha- CXOs and CLOs agree on a number of key
to come up with better ROI metrics that sized integration of learning investment indicators that demonstrate learning’s
0 5 10 15 20 25 30 35
truly connect to business value. But senior planning with the annual business plan- ongoing alignment with strategic busi-
CHART 3 Frequency executives are not demanding this from ning process, tradeoffs between invest- ness needs. Learning alignment indica-
us. They are comfortable with anecdotal ment in learning and other business tors mentioned during the interviews fall
observation.” Also, some CLOs are provid- functions, and how to balance enterprise into three categories: integration, learn-
Aligned
Varying Q3. What is your involvement in the learning investment process? ing correlational evidence to CXOs. For ex- and business unit learning needs. As one ing operations, and perceptions.
ample, one CLO participant said, “Our CXO explained, “I look to the governance ● Integration indicators monitor how
competence index can be tied to business process to match my requirements for well the learning function is integrated
outcomes so I can show correlation be- learning to the reality of what can be ac- with the business goals and work
Business Requirements 14 18
tween role competence and business im- complished given the business climate.” processes.
Enterprise Requirements 15 10
Another said, “We’ll scale back in some ● Operation indicators monitor how
pact.” Others focus on program-
Councils 4 14 Planning level ROI. For example, one participant areas and emphasize others based on the proactive and responsive the learning
BP Integration 11 4 said, “We’re pretty organized around mea- needs of the business.” function is in meeting business needs,
Individual Requirements 3 5 suring learning’s value at the program lev- CLOs emphasized the bottom-up the level of learning access and usage,
2 el. However, we are aware that ROI analysis business planning process to identify and the talent within the function itself.
Business Priority 15 16 is subjective and can be open to judgment.” and prioritize business unit learning re- ● Perception indicators monitor execu-
To more closely align with CXOs on quirements. For example, one CLO par- tive, business unit leader, and employee
Technology 4 10
how learning translates into business re- ticipant said, “I see myself as the product and learner satisfaction.
Foundational 3 5 Allocation
15 sults, the survey results suggest that CLOs manager for learning for the enterprise. Chart 5 shows the number of times
Benchmark Priority 2 6 should balance metrics and ROI data with I ensure that the money spent per em- that each group mentioned aspects of in-
Tradeoffs 6 1 anecdotal evidence. CLOs should also fo- ployee is appropriate and that the distri- tegration, learning operations, and per-
9
cus on measurement that correlates the bution of that money spent is aligned ceptions as indicators of alignment.
Hybrid Management 2 16 CXO learning function’s activity with its strate- strategically.” CLOs also emphasized the The results indicate that both CXOs
Local Management 3 5 9 CLO Management gic value at all levels. management of learning investment, and CLOs see integration of learning
Central Management 1 5 particularly obtaining the right central- goals with business goals as a critical in-
4 Investing in learning: ized/decentralized balance. However, dicator of alignment. For example, one
0 5 10 15 20 25 30 assessing CXO and CLO roles there was disagreement among CLOs re- CXO said, “How the learning investment
Learning governance is seen by both garding the best mix of centralized and aligns with the CEO’s goals should be ob-
CHART 4 Frequency CXOs and CLOs to be the primary mecha- decentralized management. One CLO vious to any bystander. The dashboard
nism for aligning learning activity with summarized this dilemma: “Centralized for global learning should start with the
results fall into three categories: organi- contribution and the data they currently ment. strategic business needs. Overseen by the budgets will get cut during expense chal- CEO priorities and then map all of the
zation outcomes, human capital, and use for evaluation. Chart 3 illustrates the CXOs emphasized the challenge of CLO, learning investment governance lenges. But if you decentralize, you will learning activities to those.” Both CXOs
learning operations. In addition, both CX- number of times that each group men- linking learning to business results and falls into three primary categories: plan- end up with a cornucopia of solutions and CLOs indicated a desire to increase
Os and CLOs discussed the challenges in- tioned aspects of the business results appear comfortable with correlations ning, allocation, and management. and technologies.” As a result, many the transparency of the learning func-
volved in measuring learning’s value categories and learning value measure- and stakeholder perceptions over direct ● Learning investment planning and al- CLOs are using hybrid management tion. One CLO said, “Success for learning

72
24 | TD
T+D | DECEMBER
October 2005 2010 TDOctober
THE BEST OF MEASURING & EVALUATING LEARNING 73 | 25
2005| T+D
Copyright ASTD, October 2005 Copyright ASTD, October 2005
 Aligned Q4. How do you know the learning function is maintaining
Varying Aligned Q6. How do you know the learning function is performing
ongoing alignment with your strategic business needs? Varying
as efficiently as possible?
Goal Integration 21 14

Transparency 8 15 Usage 11 10
Unplanned Activity 4 7 Integration Technology 6 6 8
Work Integration 3 6 4
Process Improvement 2 8
Biz Unit Integration 5 2
Learning
2
Alignment 5 4 Operations
Proactivity 9 15

6 Learning Standardization 2 7
Usage 8
Responsiveness 5 4 Operations Outsourcing 3 5 10
15
Learning Talent 5 4

13 Year-to-Year Reduction 4 8
Business Performance 8 CXO
ESAT 8 9
Benchmarking 6 5 CLO Financial
CXO Perceptions
Executive Perception 3 8
9 CLO 4 6
LSAT 5 4 Cost per Unit
4 9

0 5 10 15 20 25 30 35 40 0 5 10 4 15 20 25
Frequency CHART 7 Frequency
CHART 5

Aligned Q5. How do you measure the learning function’s value capital, learning operations, and percep- CXOs indicated that they don’t need to learning on innovation, and time to mar-
Varying contribution to your organization? tions. Again, the challenge of measure- know learning’s direct business contribu- ket is reduced and new product sales go
ment was identified by both CXOs and tion where up-front strategic alignment up, that is good. I don’t need to know how
CLOs. Chart 6 shows the number of times has been established. One said, “I don’t much was attributed to learning.”
Business Outcomes 12 20
that CXOs and CLOs mentioned specific have a dashboard for learning. When we Many CXOs indicated that they rely on
ESAT 9 7
Organization measures of organization outcomes, hu- look at things like time to market and the perceptions of their business unit or
CSAT 6 5
Outcomes man capital, and learning operations, as customer service, it is clear that many at- line leaders. One CXO said, “My col-
Turnover 3 6
4 well as mentions of measurement ap- tributes contribute to these measures. If leagues in the line do not invest in any-
Capability 12 17 Human proaches. we increase the investment and focus of thing unless they get a return. They
Time to Performance 3 6 Capital justify their own investments in learn-
15
Usage 11 15 Learning ing.” Some CXOs distinguished between
Cost Ratios 6

7
6
10
Operations
Elevating Learning Across the Company measurement of learning in different
skill areas. For example, one respondent

and Into the Future

Business Perceptions said, “For hard skills I can measure re-
LSAT 15 6 8 Perceptions sults. For softer skills like leadership, it is
Executive Perceptions 2 7
Understanding the important role spond to enterprise, business, and indi- more challenging. Yet it is the soft skills
13
Challenge 12 8 learning plays in a company’s success vidual development needs that move the needle of business perfor-
ROI 4 9
CXO Measurement is just the first hurdle. And, as IBM’s ● balance metrics and ROI data with mance to a greater extent.”
Benchmarks 3 5
9 Global CEO study reveals, many anecdotal evidence, particularly stake- Some CLOs described their efforts to
CLO
Comparison 2 5 CEOs already acknowledge that holder perceptions identify complex causal relationships
4 employee education is a critical suc- ● combine top-down and bottom-up between learning activity and business
cess factor for future enterprise suc- investment planning processes and outcomes. However, other CLOs were
0 5 10 15 20 25 30 35 40 cess. So while this is true for many emphasize the alignment of learning beginning to sound more like CXOs. For
Frequency companies, a huge window of oppor- goals with business goals example, one said, “We got wrapped
CHART 6
tunity remains. ● be proactive in facilitating the formu- around an axle with ROI.Those we play to
By taking learning to the next level, lation of strategy for the business and don’t care as much as we do about it. Less
is that we are so tightly integrated with tors, perception is key. The best indicator Measuring learning’s value CLOs can play a proactive role in dri- identifying opportunities for perfor- is more with metrics.” Some CXOs wished
the business that we are almost invisi- of learning alignment is to ask people.” contribution to the organization ving organizational productivity, trans- mance improvement they had harder metrics; for example, one
ble, but we contribute to such an extent To demonstrate alignment to CXOs, Similar to the participants’ responses on formation, and growth. Our analysis of ● continue to focus on process im- said, “Right now I don’t know how we
that our efforts are lauded.” CLOs should emphasize the alignment of how learning is maintaining ongoing CXO and CLO perceptions, particularly provement, service-level agreements, measure the value contribution of learn-
Also, the data revealed that both learning goals with business goals and alignment with their organization’s where they vary, suggests that to in- standardization, and leveraging tech- ing. It is not about hours per employee per
groups look to stakeholder and customer leverage perception data to validate that strategic business needs, both CXOs and crease their role as strategic business nology and outsourcing to maintain or year. It is about productivity and results.
perceptions for indicators of alignment alignment. CLOs should also highlight how CLOs placed greater emphasis on percep- partners, CLOs should reduce learning function costs while in- We need to link learning to these results.”
and misalignment. For example, one CXO the learning function is proactive and re- tions as measures. This fell into the cate- ● simultaneously anticipate and re- creasing operational efficiencies. Some CLOs also acknowledged the
said, “When it comes to leading indica- sponsive to the changing business needs. gories of organization outcomes, human growing measurement challenge as the

76
26 | TD
T+D | DECEMBER
October 2005 2010 Copyright ASTD, October 2005 THE BEST OF MEASURING & EVALUATING LEARNING
TDOctober | T+D
2005 77 | 27
Copyright ASTD, October 2005
O R G A N I Z AT I O N A L C H A N G E

definition of learning gets broader. One categories: financial and operations. tivity of learning’s operations. For exam-
CLO said, “We are moving away from dis- Learning access/usage was the most fre- ple, according to one CLO participant, “It
crete learning events, so measurement
becomes harder. As learning becomes
quently mentioned indicator of learning
operations efficiency.
is about doing more with less each year
either by having better development
RECOMMENDED ACTIONS
more invisible, it is harder to claim re- Chart 7 shows the frequency with standards or design practices that are COLLECT DATA THAT IS MEANINGFUL TO
sponsibility for success.” which CXOs and CLOs mentioned specif- well honed.” Another said, “We outsource
To align more closely with CXOs, ic financial and operational measures to create a variable staffing model.” LEADERS. OTHERWISE, THEY WILL NEVER SEE
CLOs should place more emphasis on of efficiency. To satisfy the efficiency requirements
up-front strategic alignment with orga- CXOs view learning like any other of CXOs, CLOs should continue to focus THE VALUE OF EVALUATIONS.
nizational outcomes rather than mea-
surement of value contribution. There
function, requiring that it deliver prod-
uctivity gains on a year-to-year basis.
on process improvement, standardiza-
tion, and leveraging technology and out-
WHERE POSSIBLE, STANDARDIZE EVALUATION
should be an increased awareness of ser- One CXO said, “Learning, like any other sourcing to maintain or reduce costs DATA ACROSS DIFFERENT FUNCTIONS WITHIN
vice-level agreements to the business, function in the business, needs to while increasing alignment with business
and CXOs, CLOs, and business unit lead- improve their productivity on an needs. TD THE ORGANIZATION TO MAKE IT EASIER TO USE
ers should agree up front on what those ongoing basis.” Another said, “Our CLO
service-level agreements include and has a lot of experience assessing our THE DATA EFFECTIVELY.
how they will be measured. efficiency against external benchmarks.
With respect to craft skills, we have over- Brenda Sugrue is senior director of research for SPEND MORE TIME AND MONEY EVALUATING
Learning’s efficiency in performing spent. We have a Rolls Royce where a ASTD, directing all of its research and bench-
Both CXOs and CLOs emphasize similar Ford will do.” marking activities; bsugrue@astd.org. Tony
BEHAVIORS AND RESULTS AND LESS ON PARTICI-
indicators of efficiency: learning access
and usage, technology/automation, cost
These findings also indicate that CLOs
are focused on applying process im-
O’Driscoll is a researcher with IBM’s Almaden
Services Research. Mary Kay Vona is the Ameri-
PANT REACTIONS. AT THE VERY LEAST, DON’T
per unit, strategic alignment, and bench- provement, technology, standardization, cas learning and development partner for IBM’s RELY ON REACTIONS SO COMPLETELY.
marking. These indicators fall into two and outsourcing to improve the produc- Human Capital Management Practice.
GIVE SUPERVISORS MORE RESPONSIBILITY
WHEN IT COMES TO LEARNING EVALUATION
TRAINING. THEY SHOULD GIVE EMPLOYEES OP-
PORTUNITIES TO USE THEIR TRAINING, AND THEY
SHOULD HELP TRACK PERFORMANCE BOTH PRIOR
IT SEEMS COMPANIES ARE SATISFIED WITH THEIR USE OF PARTICIPANT TO AND AFTER THE TRAINING.
REACTION MEASUREMENT (LEVEL 1). THERE WAS ONLY AN 8.6 PER- WHEN EVALUATING CHANGES IN BEHAV-
CENTAGE-POINT DIFFERENCE BETWEEN RESPONDENTS WHO SAID THEY IOR, USE STRATEGIES SUCH AS FOLLOW-UP
USE EVALUATIONS TO MAKE SURE EMPLOYEES LIKE THE PROGRAMS TO SESSIONS, FOCUS GROUPS, AND PARTICIPANT
A HIGH OR VERY HIGH EXTENT AND THOSE THAT SAID THEY SHOULD SURVEYS. USED WITH ACTION PLANNING AND
DO SO. PERFORMANCE MONITORING, THESE STRATE-
GIES ARE THE MOST HIGHLY CORRELATED WITH
EVALUATION SUCCESS.
From the 2010 ASTD Research Study The Value of Evaluation: Making Training From the 2010 ASTD Research Study The Value of Evaluation: Making Training
Evaluations More Effective. Visit www.astd.org/research for more info. Evaluations More Effective. Visit www.astd.org/research for more info.

28 | TD
78 T+D | DECEMBER
October 2005 2010 THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 29
Copyright ASTD, October 2005
ROI
M e a s u r i n g
Fa c t, Fa d , o r Fa n ta s y ?
ROI has become a way of life in many training departments.
Editor’s Note: The calculation of
return-on-investment (ROI) in the
workplace learning and development
field started more than a decade ago.
In 1994, ASTD published Measuring
the Return on Investment, the first
book of ROI case studies, edited by
Jack J. Phillips. Since then, Phillips
has written many articles and
books making the case for ROI and
explaining how it works. Here, he
charts its growth and acceptance and
offers tips for making sure it works.
A recent meeting of learning
executives provided some interesting
insights regarding the acceptance
of ROI. The group determined that
ROI was no longer a hot topic,
42
30 || T+D
T+D || april 2007 2010
DECEMBER
The journey
ROI’s journey took many unusual
twists and turns. At conferences
around the globe, ROI was debated,
placed on trial, and dissected by many
organizations and individuals.
At first there was resistance, which
is natural when this level of account-
ability is expected, or demanded, by
key stakeholders, including executives.
As one learning executive commented,
“If my programs are not working, why
should I publish a study to let my cli-
ents know that?”
Still, progress was made as more
organizations embraced it and began
to use it.
To some, ROI became key to both
personal and organizational success.
One individual told us that his promo-
tion to vice president was the direct
result of his work with ROI. Another re-
counted how ROI helped her keep her
job as her company eliminated many
positions in a move from Puerto Rico
to Malaysia.
Many people have been able to
secure jobs and acquire meeting
By Jack J. Phillips
invitations thanks to their expertise,
knowledge, and implementation of the
ROI methodology. The learning man-
offering up this explanation: “If ager for a large brewery indicated that
organizations want to show this level his ROI presentation gave him his first
of accountability, they are doing it opportunity to meet face-to-face with
already. ROI has become a routine the company CEO.
activity in the workplace.” In addition, many organizations
ROI has become a mainstream give high praise to the ROI methodol-
process like Six Sigma and project ogy for its help in relationship build-
management. To those of us who work ing and budget enhancement. Some
full-time with ROI, this is good news. say that it has helped them earn a
It has become a way of life for many seat at the table, change the image of
training departments. their company’s function, and build
The concept of ROI began as an key partnerships. Others indicate that
idea for my master’s thesis. It was their training department budgets in-
developed in the 1970s and refined creased, even as other company bud-
through application and use in the gets were decreasing. One organiza-
1980s. During this time, the ROI pro- tion revealed that its budget increased
cess slowly began to catch on. In the four-fold in two years with the use of
1990s it achieved global recognition, this methodology. Another doubled
implementation, and prominence. its budget.
photo by iStock
ROI around the world
My wife, Patti Phillips, and I have
devoted our careers to this issue, as
have almost 100 other full-time ROI
consultants around the world. We have
conducted ROI certification classes in
Indonesia, at a golf club in Australia,
and at a cricket club in South Africa.
Studies of ROI have ranged from
training for drug-sniffing dogs in Can-
ada to beekeepers in Australia. In Ja-
maica, the members of the local ASTD
chapter had a bake sale to pay for part
of the ROI certification. The Prime
Minister of New Zealand attended an
ROI presentation and then wanted to
learn more. The ROI methodology has
been the subject of dozens of televi-
sion interviews, radio shows, and
at last count, at least 100 magazine
articles.
Current status
ROI was still a fairly new term in 1996
as many professionals were just begin-
ning to embrace the concept. A decade
later, the ROI methodology has gradu-
ated from a hot topic to a recognized
process within many organizations.
Since the first impact study, Mea-
suring the ROI in a Cooperative Educa-
tion Program, for Lockheed Martin, in
1973, more than 200 case studies have
been published in books, journals, and
industry publications. More than two
dozen books have been published on
the ROI methodology and its appli-
cation. In addition, more than 3,000
organizations use the ROI methodol-
ogy through planned implementation,
and at least 200 government agencies
are using it. Approximately 24,000 in-
dividuals have attended workshops in
almost every major international city,
with 4,000 attending the week-long
ROI certification course.
Lingering questions
Although interest in ROI methodol-
ogy and its use have increased, some
THE BEST OF MEASURING & EVALUATINGapril 2007 || T+D
LEARNING T+D || 31
43
 The ROI Model
Planning Data Collection
Develop
Evaluation
Plans and
Baseline
Data
Inputs/Indicators
people still argue that using ROI
is difficult.
WLP professionals must show ac-
countability for the investment in
learning programs. Measuring ROI in
a few significant, high-profile pro-
grams is an excellent way to show fis-
cal responsibility. Between 30 and 40
percent of learning and development
functions in the United States use ROI
as a tool. The gap between actual use
and desired use is huge and under-
scores the misunderstandings and
misconceptions of ROI as a legitimate
part of measurement.
For almost two decades, we have
assisted organizations with this impor-
tant issue. In the past decade, we have
logged many questions about ROI that
are asked in conferences, workshops,
and consulting assignments. Here are
some of the most-debated questions:
Is ROI just one single number?
How can you communicate a pro-
gram’s value with a number? The ROI
methodology captures six types of
data, with the actual ROI calculation
being only one of them. The six types
of data include reaction, satisfaction,
and planned action; learning; application
and implementation; business impact
and consequences; ROI; and intangible
benefits. The intangible benefits in-
clude measures that cannot be cred-
ibly converted to monetary values.
44 | T+D | april 2007
32 | T+D | DECEMBER 2010
Collect
Data During
Program
Collect
Data After
Program
1. Reaction and Planned Action
2. Learning
3. Application and
Implementation
4. Business Impact
and Consequences to understand issues such as revenue,
Isn’t ROI based on nothing but
estimates that can be too subjective?
No. Estimates are only used when
isolating the effects of a program’s busi-
ness impact, when converting data to
monetary values, and when tabulating
program costs. Estimates are used only
when other methods are not available
or become too time-consuming or
expensive to use. Also, estimates are
adjusted to improve credibility.
How does ROI in learning differ
from the ROI used by the finance and
accounting staff? The classic defini-
tion of ROI is earnings divided by the
investment—no matter what the ap-
plication. In the context of calculating
the ROI of learning and development,
the earnings become the net benefits
from the program (monetary benefits
minus the costs) and the investment is
the actual program cost. The difficulty
lies in developing the actual monetary
benefits in a credible way.
Do I have to be a whiz at finance
and statistics to understand the ROI
methodology? No. Most of the basic
principles of finance and account-
ing do not relate to what is needed
to develop the ROI in learning and
development. However, it is important
Isolate the
Effects of
Program
profit, and cost. Only basic statistical
processes are required to develop most
ROI impact studies.
Doesn’t ROI cost too much? No.
When external resources are used, the
cost for an ROI study may be as little as
5 percent of the entire project. A large
banking group and a large telecom-
munications company report that the
average costs for ROI studies range
from $3,000 to $5,000 per study. The
total cost of all evaluation, includ-
ing selected ROI studies, is usually in
the range of 3 to 5 percent of the total
learning and development budget.
Does the ROI process reveal pro-
gram weaknesses? Strengths? Recom-
mendations? Yes. At low levels, data
always capture deficiencies or weak-
nesses in a process. At the application
level, the process requires collecting
data about the barriers (which inhibit
success) and enablers (which help suc-
cess). Each study contains a section for
recommendations for improvement.
Is it appropriate to calculate ROI
for every program? No. Only a few
select programs should be subjected
to evaluation at the ROI level. Ideal
targets include programs that are very ex-
pensive, strategic, operationally focused,
highly visible, involve large target audi-
ences, and have management’s atten-
tion in terms of their accountability.
photos by iStock
Data Analysis
Convert
Data to
Monetary
Values
What types of applications are typi-
cal for ROI analysis? The applications
vary, but usually they include a range of
programs such as sales training, super-
visory training, team building, execu-
tive development, communications,
competency systems, software use, and
leadership development.
The best way to dispel some of the
myths and misunderstandings sur-
rounding the ROI methodology is to
analyze the payoffs for the organization
once the ROI is calculated. Whatever
the barriers, they are completely out-
weighed by the benefits, which include
• showing the contribution of specific
programs
• showing the value of the program on
a pilot basis
• justifying or maintaining budgets
• eliminating programs that are not
adding value
• redesigning or improving programs
• building relationships with key
executives
• building support from the manage-
ment team
• improving the satisfaction of the
learning and development team.
ROI implementation
We have seen tremendous success in
some organizations, and miserable
failures in others. Ten critical success
factors have evolved that, when ad-
Capture Costs
of Program
Calculate the
Return-on-
Investment
5. ROI
Identify Intangible Measures
Intangible Benefits
dressed appropriately, can make ROI
work in amazing ways. But when they
are ignored, at least for the majority,
failures are imminent. Overall, these
10 critical success factors represent
the lessons learned by the thousands
of people who have embraced the ROI
methodology.
Fix the front-end analysis. Unfortu-
nately, within too many organizations
the needs assessment process is inad-
equate. Sometimes, its inadequacy is
unknown until the ROI work begins.
To develop the ROI and capture the six
levels of data, objectives must include
application and impact. This means
that programs must be implemented
with the business need in mind. Un-
fortunately, few learning programs
have this level of performance analysis
up front. Without it, programs often
do not deliver value, which creates
negative ROI. This creates frustration
and makes practitioners realize that
perhaps none of their programs are
adding value. This must be addressed
early in the process.
Develop the skills. While this is a giv-
en, the skill sets necessary for effective
ROI are not the same skill sets used in
other processes, such as leadership
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 33
Reporting
Reach
Conclusions
and Generate
Report
Communicate
Information to
Target Groups
development. ROI makes us think and
learn things we have not learned be-
fore if our preparation did not include
measurement and evaluation.
Follow the guiding principles. The
standards developed for the ROI meth-
odology are the “12 guiding principles.”
They provide rules for collecting and an-
alyzing data and reporting it to various
target audiences. The rules are needed
for consistency, stability, and standard-
ization. More important, the rules add
credibility. The principles are based on a
conservative philosophy, and when data
are presented to a management team,
credibility is a key issue. These standards
can help secure acceptance for the data.
They must be followed.
Educate management. The manage-
ment team is our greatest ally, and
must understand ROI. Initially, manag-
ers will say they understand ROI. Many
studied it for their MBAs. They see ROI
calculations for their divisions, depart-
ments, or companies. They know it is a
financial measure. But they can’t fully
appreciate ROI for learning and devel-
opment unless they understand that
it is a process that not only generates
the financial ROI, but also collects five
other types of data. All these data sets
are important for management teams
to know. This will require repeated ex-
planations and ROI briefings at every
opportunity.
april 2007 | T+D | 45

Plan, plan, plan. Planning is one of the
most important steps of the process,
but it is often either short-circuited or
omitted altogether. Thorough plan-
ning of a study is absolutely essential
to keep cost and time at a minimum.
Collect the right quality and quan-
tity of data. This is easier said than
done. Quantity of data reflects the vast
amount of information needed for an
ROI analysis. If data collection is left to
questionnaires, which only 50 percent
of studies use, the challenge is to have
an adequate response. Good return
rates can be obtained, and it is not
unusual for our clients to achieve a 60
to 90 percent return rate on a three-to-
five page questionnaire.
Isolate the effects of the program
on the data. A step that is omitted by
some, and ignored or hated by others,
is isolating the effects of the program
on the data. Whenever a learning and
development program is implemented
and a business measure is influenced,
the key challenge is to determine how
much of the change in the measure
is actually connected to the program.
Perhaps the most important contri-
bution that we have made in the last
decade is to refine ways to make this
determination.
Plan a face-to-face meeting with key
executives for the first study. This
is a difficult, but necessary step. The
first ROI study in an organization often
brings interest and anticipation. Exec-
utives will usually attend a face-to-face
meeting. They may be curious, sup-
portive, or cynical, but they will come.
The challenge is to make sure this
meeting is planned properly and ex-
ecuted perfectly. The meeting should
educate the team about the methodol-
ogy as well as the data that has been
collected, analyzed, and reported.
Take steps to make the cost of the
process minimal. Many shortcuts can
be used to keep the time and direct
monetary costs to a minimum. Too
often they are not used. Take short-
cuts whenever necessary, and when
alternatives are available, always take
the one that costs the least or saves the
most time.
34
46 | T+D | DECEMBER
april 2007 2010
ROI Methodology
The ROI methodology is a step-by-step process tool for evaluating any
program, project, or initiative within any organization. This process is
simplified in the ROI process model, which provides a systematic, step-
by-step approach to ROI evaluations that helps users address one issue
at a time. The methodology is based on 12 guiding principles, which are
necessary for a credible, conservative approach to evaluation through
the different levels.
1| When conducting a high-level evaluation, collect data at
lower levels.
2| When planning a high-level evaluation, the previous level
of evaluation is not required to be comprehensive.
3| When collecting and analyzing data, use only the most
credible sources.
4| When analyzing data, select the most conservative alternative
for calculations. WHAT ARE THE BARRIERS TO TRAINING EVALUATION?
5| Use at least one method to isolate the effects of a project. THE BARRIER THAT LOOMS LARGEST IS THE DIFFICULTY
6| If no improvement data are available for a population or from a
specific source, assume that little or no improvement has occurred. COMPANIES HAVE ISOLATING LEARNING AS A
7| Adjust estimates of improvement for potential errors of estimation. FACTOR THAT HAS AN IMPACT ON RESULTS. OF
8| Avoid use of extreme data items and unsupported claims
when calculating ROI.
COURSE, THIS TENDS TO COME INTO PLAY MOSTLY
9| Use only the first year of annual benefits in ROI analysis IN LEVEL 4 AND LEVEL 5 OF THE KIRKPATRICK/PHIL-
of short-term solutions.
10| When analyzing ROI, fully load all costs of a solution, project, or
LIPS MODEL. IT IS NOT A BARRIER FOR LEVEL 1, SINCE
program. THERE ARE NO OTHER FACTORS INVOLVED IN THE
11| Intangible measures are defined as measures that are purposely
not converted to monetary values. PARTICIPANTS’ REACTIONS.
12| Communicate the results of ROI methodology to all key stakeholders.
Make plans to sustain the process. agement, meetings and events, higher
With any change comes a need to education, and many other fields.
sustain it. Early in the implementa- While the journey has been difficult, From the 2010 ASTD Research Study The Value of Evaluation: Making Training
tion, processes must be put into place challenging, and certainly stressful at Evaluations More Effective. Visit www.astd.org/research for more info.
to make ROI evaluation routine and times, it has been extremely reward-
important. Otherwise, it will become ing. As I reflect upon the difficulties
a nuisance and an add-on activity that experienced, as well as the successes
may quickly cease if the ROI champion enjoyed, I hope ROI will continue to
leaves the organization. make a difference in this field.T+D
Other applications
One of the most unexpected develop-
ments is the use of the ROI method-
ology in other fields. Originating in
learning and development, it quickly Jack J. Phillips is founder of the ROI
moved to HR, consulting, information methodology and chairman of the ROI
technology, marketing, project man- Institute; jack@roiinstitute.net.
photo by iStock THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 35
Fundamentals //

Don’t let good

training fly out
the window. Use
Level 3 to make
sure it sticks.

At 4:59 p.m. you are As a trainer, you briefed, asked ques-

tions, and checked for understand-
What is staggering
making closing statements ing. Heck, you may have even had a is that research
to a group of managers
enrolled in a management
post-training assessment completed to
validate the training and measure their
indicates that only
development class by learning—a Level 2 requirement. Now between 10 and 20
their supervisors. After you really can’t wait to be asked, “How
was the training?” You are eager to show
percent of managers
eight hours of role play, off post assessment learning scores with successfully transfer
simulations, and case a fancy spreadsheet and graphs.
behaviors into the
studies, you distribute Then, a funny thing happens. Two

smile sheets to gauge what
the participants thought
or felt about the training.
You receive a round of
applause as you conclude
with, “Thank you all for
coming.”
The next day, your peers and a senior
manager ask, “How was the training?”
In response, you glance at the smile
sheets, notice that most participants
circled the highest score choice, and
say, “It went great. They loved it.”
This is where training results enter
a gray area instead of what should be a
black and white picture. The question
asked was, “How was the training?” That
question is quite ambiguous, inviting
a variety of answers. The keyword in
that question is, “was,” as if to state that
training is a terminal event, that we now
months pass, and a significant number
of those participants who scored high
on reaction and learning are now ask-
ing for copies of the handouts, when
the next refresher course is offered, or
about a principle or method that was
discussed during the training. This
is the point where the real answer to
the question, “How was the training?”
should be answered.
As the most challenging of evalua-
tions, Level 3 seeks to evaluate whether
the participants successfully trans-
ferred their skills and behaviors into
the workplace. Level 3 evaluations
help identify impediments to success-
ful knowledge and skills transfer. What
is staggering is that research indicates
that only between 10 and 20 percent of
managers successfully transfer behav-
iors into the workplace. Not exactly the
type of numbers one would want to
produce during budget time.
workplace.
recently hired. In what can be described
as a very volatile, dynamic organization,
making time to attend the training was
one challenge; making sure the training
was successful was another.
In this example, 20 managers were
trained throughout a three-day period
consisting of eight-hour days. Coach-
ing and counseling, interpersonal skills,
and conflict resolution were among
the training modules offered. Level 1
showed, to no one’s surprise, that every-
one was quite happy and eager to put
their training to use. Level 2 was also
encouraging since participants demon-
strated a significant level of learning. So
far so good, right? Well, not exactly.
There seemed to be a gap from what
was learned and displayed during the
training, and what was actually trans-
ferred and used in the workplace. The

On the Level
have improved the skill, behavior, or
knowledge of each participant. After all,
many participants wrote in the com-
ments section how excited they are to
apply this new learning.
By George Vellios
False assurance
When a New York-based healthcare or-
ganization implemented a management
development training program, the par-
ticipants were either newly promoted or
participants, who represented various
branch locations across the tri-state
area, were reaching out to the train-
ing facilitator and requesting refresher
training, for one. Issues such as man-
aging poor-performing employees,

36 | T+D | DeCember
26 DECEMBER2008
2010 Photo by IPNSTOCK.com Photo by iStockphoto.com THE BEST OF MEASURING & EVALUATING
DeCember
LEARNING
2008 | T+D | 37
27
Fundamentals //
How to Apply Kirkpatrick’s conflict resolution, and communication
were not decreasing as a result of train-
Four Levels of Evaluation
ing. The HR department’s data showed
an increase of 22 percent in terms of
By Donald L. Kirkpatrick
internal complaints regarding manage-
ment processes. What went wrong?
While most workplace learning and performance professionals are familiar with
In the preceding example, the train-
my four levels of evaluation, some still don’t know how to implement them into
ing facilitator conducted a detailed
their training programs.
interview and survey with all of the
I have put together some guidelines for each of the four levels and suggestions
participants who attended the training.
for how to apply them.
Data was analyzed using content analy-
sis to depict key themes and variables
Level 1: Reaction Guidelines
which acted as barriers for successful
1. List all the items from which you want to get reaction, including items dealing
transfer of behaviors to the workplace.
with the content such as, “Did it meet the needs of the participants and the
The use of Level 3 evaluations was an
instructor?” and items dealing with audio-visual aids, case studies, hand-
eye-opener for the organization. The
outs, meals, the facilities, and the schedule.
data told the real story.
2. Design a form that will quantify reactions.
Seventy percent of the participants
3. Encourage written comments and suggestions.
claimed that they were overwhelmed
4. Get 100 percent immediate reaction.
with day-to-day work and didn’t have
5. Get honest responses by making the form anonymous.
the opportunity to use their newly
6. Communicate the responses to the appropriate people.
acquired skills. This caused them to
To implement these guidelines, have a comment (smile) sheet available for
resort back to old behaviors. This also
participants to complete before they leave and convene a focus group of six or
resulted in the participants forgetting
eight participants after a month to openly discuss the reaction results.
what was learned.
Sixty percent of the participants
Level 2: Learning Guidelines
stated that their own managers didn’t
1. Evaluate knowledge, skills, and attitudes before and after the program.
consider training to be of primary impor-
2. Use a “paper-and-pencil” test to measure changes in knowledge and attitudes.
tance. Management didn’t follow up on
3. Use a performance test in the classroom to measure changes in skills.
the progress or assess their use of the skill.
4. Get 100 percent response.
Of this group, only 15 percent stated that
5. Communicate the amount of learning to the appropriate people. their supervisors reinforced their training
If changes in knowledge and attitudes are being measured, a pretest should be
through developmental goals.
made for the program content. The post-test should be identical to the pretest—
why make two different tests to measure the same information
Changing course
If you are measuring changes in skills, a pretest of performance should be Based on the survey findings, the fol-
made and compared with a performance post-test. If the course is teaching
lowing corrective action steps were
presentation skills, an expert should measure each participant on performance
implemented by the training function
against certain standards. The post-test should measure the same type of
at the healthcare organization in an ef-
presentation and again be evaluated by the expert. This needs to be done for
fort to remedy the situation.
each person.
First, establish partnerships with
If you are teaching a new subject, there is no need for a pretest. To measure
executives to incorporate professional
if the participants have learned a new technique, create a role play exercise with
development learning and use as part
participants playing the parts of coach, coachee, and observer. The participants
of their work goals. The goals would
will change roles until each has performed all three roles.
cascade down to their subordinates,
creating an overall buy-in and personal
investment to training.
Next, promote a culture within the
organization advocating training. To
create this culture, managers should
partake in train-the-trainers, and serve
as training facilitators.
38 | T+D | DeCember
28 DECEMBER2008
2010
How to Apply ... cont.
Finally, train subject matter experts
on how to conduct Level 3 evaluation
Level 3: Behavior Guidelines
1. If practical, measure on a before-and-after basis to determine what change
surveys. This process enabled the orga-
has taken place.
nization to create a training committee,
2. Allow the proper amount of time for the change to occur.
where key themes were explored for
3. Get information from the learner as well as from one or more of the following
management and technical training.
people:
Level 3 was critical to the success of
a) The learner’s supervisor
training at this organization. Everyone
b) A peer who works closely with the learner
became an advocate for the training.
c) One or more subordinates of the learner—be careful because many people
The corrective action steps were
do not want their subordinates to get involved. Be sure to check with the
incorporated and significant changes
learner, who might select certain people to interview or survey.
were made. This was best illustrated
4. Get a sample of the learners to evaluate. Obviously, the larger the sample,
12 months after the corrective action
the more weight the results would carry.
implementation took place.
Learners who attended a program on “How to orient and train employees”
A group of 38 managers were evalu-
can be evaluated the first time they hire a new employee, but, if you are evaluat-
ated based on training they had par-
ing a participant of a course on leadership, you should wait three months or more
ticipated in within the last year. Data in
before evaluating behaviors.
this particular study showed that only
To evaluate behavior, use a patterned interview that asks the same questions
19 percent claimed time constraints
of each learner so the responses can be quantified. Make sure you ask how much
as a barrier, and only 11 percent stated
the participant has changed since he attended the program. If little or no change
that support from management was
has occurred, ask what prevented the behavior change and include reasons such
not present. HR data also reported a
as the things I learned did not apply to my job, I had higher priorities, I tried the
66 percent decrease in internal com-
techniques but it didn’t work out, my boss discouraged or prevented me from
plaints regarding management pro-
changing, or other reason.
cesses. A paradigm shift had occurred
When interviewing the learner, persons might not want to say “little” or “not
in terms of gathering and evaluating
at all” because it may make them look bad. So, be sure to ask for honest answers
data to target key barriers for successful
and emphasize that the purpose of the interview is not to evaluate the person, but
implementation.
the program.
So when someone now asks you,
Another technique is to use a survey instead of the interview, a focus group of
“How was the training?” feel free to
selected learners, or storytelling with a group of learners.
answer that very complicated question
with a simple response: “What you can’t
Level 4: Results Guidelines
see, Level 3.”
1. Measure before and after the training.
2. Allow time for results to take place.
George Vellios is director of organiza-
3. Repeat the evaluation at appropriate times.
tional development/HR for TheraCare Inc.;
4. Be satisfied with evidence if proof is not possible.
GVellios@aol.com.
There is no magic answer to these statements. Use your judgment. If you
want to measure the results of a leadership or coaching program, the first
measurement may be completed in six months and the second in a year. But if you
want to measure increase in sales from a new selling approach, you may want to
start measuring after two or three months and again after six months and a year.
To evaluate results of a training program, ask a focus group of selected
learners what results they have achieved and how much money was saved.
This would include not only what they did when they returned to the job, but
also any new department initiatives because of the ideas and suggestions they
brought back.
I hope I have given you some guidelines and techniques to help you evaluate.
The next thing you can do is adapt forms, procedures, ideas, and techniques from
What do You think? other organizations.
T+D welcomes your comments. If you would
like to respond to this article, or any article
that appears in T+D, please send your feedback Donald L. Kirkpatrick is professor emeritus at the University of Wisconsin. He is past
to mailbox@astd.org. responses sent to the president of ASTD; dleekirk1@aol.com.
mailbox are considered available for publication
and may be edited for length and clarity.
THE BEST OF MEASURING & EVALUATING
DeCember
LEARNING
2008 | T+D | 39
29
 tangIbLe return on
Investment
IntegratIng LearnIng to reach DesIreD resuLts
By Rebecca Everett
One organization’s experience illustrates how learning
initiative design influences outcomes, while making
the case for continuing professional development even
in a tough economy.
Bachrach & Associates Inc. (BAI),
a professional development firm for
financial professionals, recently took a
close look at the business results of their
advisor-clients. Those results revealed
statistical confirmation of tangible ROI
attributable to the design of its blended
learning and reinforcement approach.
BAI’s professional development pro-
grams train financial professionals in the
distinctive philosophy of “values-based
financial planning” (VBFP).
Recognizing the importance of
reliable, actual return on training
investment, BAI decided these results
should be shared with professionals
who continue to strive for reportable,
tangible results for their training and
professional development dollars so
the design and implementation could
be replicated
BAI was founded 20 years ago by
Bill Bachrach with the intent of devel-
oping a learning system for financial
planners, CPAs, insurance agents, and
other finance professionals seeking to
dependably and ethically acquire new
clients. In the last two years, BAI has
transformed its foundational learning
materials into true training instru-
ments, with more of an adult learn-
ing and blended learning approach.
40
50 || T+D
T+D || FEBRUARY
DECEMBER2009
2010
The company has also continued to
improve its programs and services and
has refined measurement tools for its
own and its advisor-clients’ results.
The outcome of the latter was the
finding that BAI’s program design
helps create a genuine return-on-
investment—results large enough to
surprise BAI and its VBFP advisor-
clients. The program design consists
of the following resources:
• individualized learning and
implementation plan that
integrates values, goals, and
objectives
• learning materials including
audio, a DVD, and assessments,
plus tools for implementation
• on-site three-day learning envi-
ronments with other implement-
ers of VBFP, each offered three
times per year
• a study group
• accountability coaching in the
form of 30-45 minute calls held
at least two times per month
• 60-minute interactive telecon-
ferences facilitated by Bill Bach-
rach, held twice per month
• full-day workshops on busi-
ness practices and interpersonal
listEn to this fEatuRE
skills. at www.astd.org/TD/TDpodcasts.htm
Photo by Media Bakery THE BEST OF MEASURING & EVALUATING
FEBRUARY
LEARNING
2009 | T+D | 51
41
 Research has Results from the real world
BAI studied the results of advisor-
shown that the clients who participate in BAI’s “Being
more integrated the Done” study group. This is a group
of advisors who enroll in a year-long
learning approach reinforcement program consisting of
and the more robust bi-weekly conference calls, daily activ-
ity reporting, and bi-weekly, individual
the reinforcement of accountability coaching calls.
implementation of These participants are voluntarily
held to a very high level of account-
the desired behaviors, ability. The group is labeled with the
the more likely the phrase “being done” to illustrate an
important component whereby par-
sought-after results ticipants choose a specific target date
will be achieved. when they no longer have to acquire
new clients; the annual, predictable,
recurring revenue meets all their fi-
nancial needs; and they have the time
to deliver the quality of service that
will establish intense client loyalty.
Participants only serve clients who
fit their ideal client profile (those who
have the economic and personality
characteristics that a participant most
appreciates and enjoys serving) and
only once they are “done” with promo-
tional activities. The date at which this
is achieved is known in BAI’s vernacu-
lar as the being done date.
To discern the effects of imple-
menting VBFP and to learn how else
BAI could continue to improve their
offerings, the organization reviewed
one full year of data from group par-
ticipants. Each member of the study
group independently entered multiple
data points in a daily report submitted
tabLe 1| results of four-year college investment
compared with VbFP.
Payback
initial Period
investment Roi (in months) iRR
College $107,300 937% 95 19.4%
VBfP $108,000 3,381% 7 499.7%
42 | | T+D
52 T+D | |FEBRUARY
DECEMBER 2010
2009
to BAI Monday thru Friday, via email
for weekly and annual compilation.
In these reports, participants share
the number of client appointments (of
several types) that were scheduled and
kept, the number of phone calls made
and appointments scheduled and kept
with referrals, and the number of sup-
portive activities completed to engage
referrals. They also report their current
ideal client counts.
BAI selected these data points
because a previous analysis with a
very small sample indicated they are
the best predictors of progress and
success. The information is compiled
each day, and from it, BAI extrapolated
the data prompting this report. The
extreme high and low results were ex-
cluded to get a more realistic picture.
Demographic information (both
current and historical) and progress
measures, including participants’ de-
sired annual income and being done
date, were also added. BAI also asked
them to reveal their desired predict-
able, minimum annually recurring
revenue (PMARR) per client—an indi-
cator of the type of clients they choose
to serve. A higher PMARR indicates
a higher annual fee for service and,
likely, more affluent clients. The aver-
ages of those data came out as follows:
• desired annual income: $403,000
• being done date: November 15, During those 9.5 years, the advisor • sale of the business five years after development programs were directly
2012
• PMARR: $3,860 per client.
BAI then calculated the number of
clients that participants would need
to serve to achieve the desired income.
Dividing the income desired by the
PMARR per client, an advisor needs
105 clients to achieve that annual in-
come. (This equation varies, of course,
depending upon the desired annual
income, how many clients an advisor
wishes to serve, the expenses the advi-
sor incurs, and the desired predictable
annual revenue. BAI saw client commu-
nities as small as 50 and as large as 225,
with most advisors choosing 100 to 125.)
Measurement
To calculate the ROI throughout the
life of the business for this example,
tabLe 2| results of the 14-year timetable.
their practices. They invest in few,
if any, other training programs
for client acquisition and client
Years Payback
services. The design of BAI’s pro-
initial fessional development resources,
implementing
VBfP investment Roi Period
(in months)
iRR
however, is replicable, and the
data indicates its efficacy.
• The group participates in nearly
every program, invests in the rec-
9 $108,000 2,477% 12 214% ommended resources, and consis-
tently uses most of BAI’s services.
• For the most part, the best results
4.5 $108,000 3,381% 7 499.7% are achieved by participants who
take advantage of accountability
coaching every week, either in a
group or individually, and they
we completed a very conservative bear in mind some additional pieces follow the guidance and advice of-
projection, with a sale date of the of data: fered by their accountability coach
business in November 2017, in 9.5 • the payback period is 7 months (professional consultants within
years (five years after reaching the • internal rate of return (IRR) is BAI who are assigned to assist
BDD). Note that the revenue listed 499.7 percent advisors in making constant prog-
is only what was generated from • positive cash flow begins after 4 ress toward achieving their goals).
implementing VBFP—nothing from months. These accountability coaches are
the current business revenues at the BAI reworked the numbers for a 14- an integral and substantial part of
time of engaging in VBFP. year period of staying in the business, the entire set of resources.
Multiplying 105 clients by $3,860 doubling the length to the being done Research has shown that the more
over 9.5 years (114 months) reveals date (from 4.5 years to 9 years), not the integrated the learning approach and
that the business would generate $3.7 length of staying in business once that the more robust the reinforcement
million, or $2.9 million plus $808,000 date was achieved (5 additional years). of implementation of the desired
from the sale of the business. (That The following factors did not change in behaviors, the more likely the sought-
$808,000 comes from multiplying the this second example: after results will be achieved, as BAI’s
annual income [roughly $404,000] • desired income data has verified. Based on the results,
times 2, a conservative way to estimate • PMARR BAI has concluded that the design and
the value of a business for sale.) • number of ideal clients needed implementation of its professional
would have invested a total of a little reaching the being done date. responsible for monetary results,
over $108,000 in BAI’s programs, Both the total investments and the meeting Level 5 on the Kirkpatrick
resources, and services, plus incidental revenue increase in accordance with standard of measure.
costs (such as travel to programs this longer timetable. Table 2 indicates In an economy where every dollar
and referral resources) as part of the results of the 14-year timetable, spent is scrutinized for earning poten-
implementing VBFP. with additional data as follows: tial, the learning professional is well
Based on research conducted in • payback period is 12 months served to refer to BAI’s results to justify
2006 on the ROI of investing just over • 2,477 percent ROI this type of comprehensive design and
$107,000 in a four-year college educa- • IRR is 214 percent deployment in her own proposals and
tion, one can expect to earn approxi- • cash flow turns positive in month 7. budgets. T+D
mately $1.1 million more than a high
school graduate can throughout the Back to the bottom line Rebecca Everett is CEO of Connecting Point
course of a 40-year career. This is a compelling case for an advisor Consulting; rebecca@rebeccaeverett.com.
Compare that result to the VBFP (or a company that employs financial
results. An investment of virtually that advisors) to consider implementing
same amount—a little more than VBFP as a means to succeed in this What Do You think?
$108,000—and a payoff of almost $3.8 profession. Here are the caveats: T+D welcomes your comments. If you would
like to respond to this article, or any article
million in 9.5 years versus 40 years. • The advisors in BAI’s sample that appears in T+D, please send your feedback
Calculated in percent, the ROI is more group are committed to imple- to mailbox@astd.org. Responses sent to the
than 3,000 percent. See Table 1, and menting VBFP and only VBFP in
mailbox are considered available for publication
and may be edited for length and clarity.
THE BEST OF MEASURING & EVALUATING LEARNING
FEBRUARY 2009 || T+D
T+D || 43
53
 Measuring Me
What Matters: Wh
HOW CEOs ViEW HO
LEarning SuCCESS LE
By Jack J. Phillips and Patti P. Phillips
Some mysteries are meant to be solved. The mystery
of the top executives' view of success in learning
and development is brought to light in a recent
survey of Fortune 500 CEOs.
easuring the success of
learning and development
has earned a place among
the critical issues in the learning
and development field. For decades,
this topic has been on conference
agendas, and journals and
newsletters regularly dedicate space
to it. Professional organizations
have been created to exchange
information on measurement and
evaluation, and more than 50 books
have been written on the topic. More
importantly, top executives have an
increased appetite for data about the
business contribution.
Although interest in measuring train-
ing success has been heightened and
much progress has been made, it is still
an issue that challenges even the most
sophisticated and progressive learning
and development functions. The top
44 | T+D | AuguSt 2009 Photo by iStockphoto.com
By Jack J.
Some my
of the
and dev
survey
executive group, the most important
stakeholder, is the key.
While some learning and develop-
ment leaders argue that developing the
a successful evaluation process is and
too difficult, others are quietly and thi
deliberately implementing effective age
evaluation systems and reporting new
results to executives. The latter group to i
has gained tremendous support from hav
the senior management team and has inf
made much progress. eva
Regardless of the position taken on hav
the issue, the reasons for measurement imp
and evaluation are clear. Almost all inc
learning and development professionals bus
share a concern that they must show the
results of learning investments to senior ing
executives. Otherwise, funds may be mu
reduced or the function may not be able an
to maintain or enhance its status and sop
influence within the organization. and
listen to this feature
at www.astd.org/tD/tDpodcasts.htm
THE BEST OF MEASURING & EVALUATING LEARNING
AuguSt 2009 | T+D | 45
 Figure 1 | CEO-reported tions in learning and development
investment range team members in this recession.
$640 Million Executives intuitively feel that provid-
ing learning opportunities is valuable, and
they logically anticipate a payoff in im-
portant, bottom-line measures, such
as productivity improvements, quality
enhancements, cost reductions, time
savings, and improved customer ser-
vice. Yet the frustration comes from the
lack of evidence to show that programs
really work.
While results are assumed to exist
$138 Million and learning programs appear to be
necessary, more evidence is needed,
or executives may feel forced to adjust
$10 Million
future funding. A comprehensive mea-
Low Average High surement and evaluation process, de-
Source: ROI Institute signed with top management in mind,
represents the most promising, logical,
Executive dilemma and rational approach to accounting
The dilemma surrounding the success for the learning investment.
of learning is a source of frustration
for many senior executives. Most ex- CEO survey
ecutives realize that learning is a basic We conducted a CEO survey aimed at
necessity when organizations experi- obtaining direct feedback about the
ence significant growth or increased success of learning and development
competition. from a significant number of CEOs in
Formal learning is also impor- large organizations. To our knowledge,
tant during business restructuring no significant data has come from
and rapid change, where employees these elusive groups. And although
must learn new skills and often find many one-on-one interviews are pre-
themselves with heavier workloads. sented in profiles in magazines, rarely
However, during an economic decline, do they discuss specifics on results.
some executives are not so sure that Surveys of this nature are often
learning and development is needed, directed toward heads of learning and
as evidenced by some of the reduc- development, where they are asked to
Figure 2 | CEO input on investment
Which of the following best describes your approach to investing in learning and
development? (Respondents checked a single best answer):
We try to avoid the investment if possible, using contract
workers, hiring fully competent employees who do not need 4%
training, and using temps when necessary.
We invest only the minimum—what is absolutely
necessary for job-skill training.
We invest at levels consistent with our benchmarking studies,
using measures such as the training cost per employee.
We invest heavily in learning and development, essentially
meeting all needs that are identified in the organization.
10%
We invest when we can see some type of benefit for investing,
essentially investing when there is a payoff.
Don’t know/Did not respond 9%
46 | 46
T+D| T+D
| AuguSt 2009 2010
| DECEMBER
provide their impression of the results
their executives want, instead of ob-
taining this information directly from
the CEOs.
To obtain the executives’ views on
learning and development, we sent a
survey and a letter with instructions
that asked CEOs not to forward our
survey to the learning and develop-
ment department in hopes that we
could hear directly from them. The
survey was formatted and designed
for the most optimal response rate
possible, which for us, was at least 10
percent, and an optimistic 30 percent.
Techniques to achieve a higher
response rate. We collected data
between October 2008 and February
2009 using the most accurate CEO da-
tabase directly from Fortune Magazine.
We discarded any firms on the list that
were currently facing economic turmoil,
such as AIG, Lehman Brothers, General
Motors, Ford, Merrill Lynch, Morgan
Stanley, and Chrysler, or any company
that had reported a significant loss.
This trimmed 99 companies out
of Fortune 500, leaving 401 for this sur-
vey. We selected 50 large, private-sector
employers, using Hoover’s website as
a guide. Essentially, these companies
would be listed in The Fortune 500 if
they were publicly held companies.
Together these provided a total of 451
firms in the large-company sector to
receive this survey.
20%
39%
18%
N=96 (large public and private firms)
Source: ROI Institute
In a well-executed return-on-in-
vestment study, for example, it is not
unusual to have a 70 to 90 percent
return rate. We applied the same disci-
pline, determination, and techniques
to this project because this group was
particularly difficult to reach.
Their exclusivity is in part a result
of their “gatekeepers” (assistants, vice
presidents, and others), who protect
them from tasks that may be deemed
time-consuming or not essential to
their role. In many large organizations,
dozens of gatekeepers may be assigned
to filter just one executive’s requests
and demands. Consequently, we knew
that our approach for this survey had to
be creative. We used 10 powerful tech-
niques to achieve a higher response
rate. Here are a few of them:
• Unless CEOs elected to provide
contact data, survey responses
were anonymous. (No one chose
this option.)
• We wrote personal notes on almost
all of the letters, based on our rela-
tionship with that organization. For
many of them, we are sharehold-
ers, and wrote a plea for results as
an act of accountability. In other
cases, we are customers, and wrote
that we were interested from a
customer viewpoint. Sometimes
we would mention our current
relationship, with phrases such as
“we have projects ongoing,” “we
serve as a regular consultant,” or,
as in the case of IBM Services, “we
act as an official subcontractor.”
This personal approach may have
Photo by iStockphoto.com
Figure 3 | Satisfaction With Measures
of Success of Learning and Development
Satisfied
(score of 3)
41%
45%
Dissatisfied
(score of 2)
helped the survey stand out among
numerous other requests.
• We tried to work with someone
else in the company, who was not
in the learning and development
area. In approximately 20 percent
of these firms, we knew someone,
usually a middle manager. We
asked them to deliver the survey
directly to the CEO.
Response rate. Ninety-six individu-
als responded, representing 21.3
percent of the total. Executives chose
to remain anonymous, and some
did not answer particular points or
provide comments. A few executives,
however, gave us extensive com-
ments and seemed to take a great
interest in doing so.
This response is especially sig-
nificant when considering the diffi-
cult economic circumstances during
the time the survey was conducted.
Spending a few minutes on a survey
addressing learning and development
is not a priority for most executives
during this time. Additionally, to our
knowledge, all of the returned surveys
were actually completed by the CEO,
which was our request. We suspect
that if a CLO had completed it instead,
she would have acknowledged it.
Results
Investments of these companies
ranged from $10 million to $640 mil-
lion. Sometimes, executives scribbled
notes about how they were not sure of
the exact amount. The average number
THE BEST OF MEASURING & EVALUATING LEARNING
Very satisfied
(score of 4)
6%
Very dissatisfied
8%
(score of 1)
Average score = 2.5 out of 4
Source: ROI Institute
was $138 million. Figure 1 provides
additional details.
Regarding the rationale for setting
the investment level, CEOs selected
a strategy from a list. (Figure 2 shows
the responses.) Although these results
confirmed what we expected to a cer-
tain extent, a few surprises surfaced.
Only 4 percent acknowledged that they
try to avoid these investments, but we
suspect that this number may be a
little higher.
Twenty percent of CEOs said they
invest only the minimum. In part, this
may be due to the current economic
times, whereby executives have had to
trim activities that are perceived to be
unnecessary. As expected, benchmark-
ing was the most-reduced (39 percent of
respondents), but because we asked that
only one strategy be checked, there may
be a combination of benchmarking and
other possibilities. We limited the choice
to only one in the survey to determine
the dominant approach for setting the
investment level.
Surprisingly, a significant num-
ber of CEOs (10 percent) mentioned
that they invested in all learning and
development needs. Although we
worded this option so that they would
feel comfortable with the choice, this
is probably over-investing. From our
own experience, we see this routinely,
and some executives are proud that
they can invest in practically any
learning request.
Finally, it’s quite refreshing that a
significant number invest when they see
value (18 percent). We assume, however,
AuguSt 2009 | T+D | 47
 Figure 4 | results on Specific Measures
We should
We currently measure this
Measure measure this in the future
inputs: “Last year, 78,000 employees received
(90) 94% (82) 86%
formal learning.”
efficiency: “Formal learning costs $2.15 per hour of
(75) 78% (79) 82%
learning consumed.”
reaction: “Employees rated our training very high,
(51) 53% (21) 22%
averaging 4.2 out of 5.”
learning: “Our programs reflect growth in knowl-
(31) 32% (27) 28%
edge and skills of our employees.”
application: “Our studies show that at least 78% of
(11) 11% (59) 61%
employees are using the skills on the job.”
impact: “Our programs are driving our top 5 busi-
(8) 8% (92) 96%
ness measures in the organization.”
roi: “Five ROI studies were conducted on major
(4) 4% (71) 74%
programs, yielding an average of 68% ROI.”
awards: “Our learning and development program
won an award from the American Society of training (38) 40% (42) 44%
and Development.”
Note: Number in parentheses indicates number of respondents.
that there are many different definitions spends with other direct reports.
of the term “value,” and that this does Senior executives have limited time,
not always mean ROI calculations. and they spend it in those areas they
perceive to be critical, important, and
Reporting relationships. An impor- central to the organization’s success.
tant factor in this study is to deter- Unfortunately, many executives do not
mine how close the CLO, the head of see learning and development rising to
learning and development, is to the this level of criticality, and they there-
CEO. In this study, a “1” indicates that fore allocate little time to engage in it.
the CLOs report directly to the CEO. The problem is compounded when
A “2” means that there are two levels the learning and development execu-
between them, and a “3” means that tive reports through one or more execu-
there are three levels between them. tives and only interacts with the CEO
The average was 3.2, which means on special projects or during periodical
that the CEO is at least three levels reviews of the learning and develop-
above the CLO. This distance is a little ment budget. Regrettably, these budget
disturbing, considering the great reviews are when senior executives
amount of effort that has been focused hope to see a connection between
on pushing this function to a higher learning outcomes and the business to
level within the company. justify increasing or sustaining budgets.
One challenge that has com- It is no surprise, then, that there is con-
pounded accountability issues in fusion and misunderstanding of expec-
learning and development is the lack tations with regard to requirements for
of interaction between the learning measures of learning success.
and development professional and
the senior executive team. In most Satisfaction with the current mea-
organizations, this interaction is lim- sures of success. Limited interac-
ited. In only a few organizations does tion with executives often forces the
the top learning executive report to learning and development leader to
the CEO. Even in those organiza- “guess” what the top executives want
tions, the time spent with the CEO is in terms of measures of success. This
first two categories were inputs and effi-
ciencies, which are process measures or
My ranking of the inputs to the process, including volume,
importance of this measure costs, and speed.
The next two categories, reaction
average rank and learning, are typical learning
measures. Application is the extent of
6.73 6 the use of knowledge and skills. ROI, as
well as impact, which is the business
6.92 7 measure, created much interest. We in-
cluded ROI because of the abundance
of information about its use. Finally,
7.15 8
we included awards, which many
CLOs are currently pursuing and re-
4.79 5
porting to the executives, particularly
in large organizations.
3.42 4 Obviously, these map into the levels
of evaluation described in the literature.
1.45 1 The first two categories (inputs and ef-
ficiencies) are Level 0, reaction is Level 1,
learning is Level 2, application is Level 3,
2.31 2
impact and awards are Level 4, and ROI
is Level 5. Although other specific mea-
3.23 3 sures may be identified, they all should
fit into one of these eight categories.
Given this list, we wanted to know
Source: ROI Institute three things:
when filtered through multiple layers •“What metrics are being reported
of interpretation. to you now?”
Asking top executives outright • “What should be reported that
what specific measures they want to isn’t being reported now?”
improve often yields ineffective or •“How would you rank these in
misguided dialogue. After all, top ex- terms of value?”
ecutives do not see their responsibility Figure 4 shows the responses. The
as defining the measures of success. first percentage column is the percent
Essentially, they want the learning of CEOs who checked this item as a
and development leaders to report metric being reported, the second is the
improvement in measures of success percentage indicating that it should be
that are meaningful to them in terms reported, and finally, the last column
of business contribution. is the average ranking number for the
We asked the executives a very gen- group, recognizing that the lower the
eral question regarding their level of number, the higher the ranking.
satisfaction with the measures of suc- A score of 8 would indicate an 8th
cess for learning and development. We place ranking on the list, and a “1”
created a four-point scale to force the would indicate first on the list. Inputs
executives to take a stand. On a four- and efficiencies were ranked 6th and
point scale, a “1” would be very dissat- 7th, respectfully.
isfied, and a “4” would be very satis- Input indicates the scope and vol-
fied. We were hoping to see at least a ume—something executives need to
“3,” but unfortunately the results were know. These types of data are always
2.52, indicating some dissatisfaction. being reported. While most CEOs
(Figure 3 shows the results.) receive this kind of information now,
they quickly recognize its limited
Metrics coverages. We knew it was easi- value. Reaction is ranked the lowest,
est to have executives respond to par- which may not be a surprise, though
ticular checklists, and deciding which it is the number 1 outcome measure
metrics to use was very critical. We reported to executives. This particular
provided eight categories and mapped measure could be improved with more
Awards was rated higher than we
expected. Both pessimists and op-
timists wrote comments. The opti-
mists were proud of their awards and
thought they reflected the quality and
significance of the learning and de-
velopment team. Others said that the
awards mean very little and are often
based on how much they’re willing to
spend on the award application.
The two highest-ranked areas were
impact and ROI, which should come
with little surprise because CEOs always
want to see this kind of data, especially
when the economy is struggling. This
reporting represents a very important
challenge and opportunity for learning
and development: These are the least-
reported data sets, but at the same time,
they are the most valuable to executives.
Learning scorecard. We asked about
the learning and development score-
card, and we discovered that only 21
percent of the CEOs surveyed said that
they had a learning and development
scorecard. This is surprising given the
work to develop balanced scorecards in
large organizations. Of course, a score-
card very well could be in place, but it
does not make its way up to the CEO.
On the positive side, this result
indicates that some executives are
reviewing scorecards on a routine ba-
sis. For the most part, the comments
in the scorecard were either negative
or constructive. Only one indicated
being pleased with the scorecard in
use. The other comments indicated a
belief that the scorecard was “inad-
equate,” “incomplete,” “doesn’t have
all the data,” or “doesn’t really con-
nect to the business.”
Executive involvement. A critical issue
for learning and development depart-
ments is the extent of the executive
involvement. Most would argue that ex-
ecutives are taking a more active role or
are more involved with the investment,
and thus, more results will be achieved.
As expected, the top area is that
the CEO personally approves the
learning and development budget,
which was indicated by 78 percent.
Second on the list, indicated by 73
percent, was that they review re-
quests for major programs, while 61
programs. Twenty-four percent use
a scorecard to monitor the progress
and make adjustments.
Next, 29 percent opened and closed
major programs, while 21 percent host
or conduct periodic review meetings,
and only 18 percent actually teach seg-
ments of major programs. Disappoint-
ingly, the two weakest levels of involve-
ment, holding periodic review meetings
and being involved by teaching seg-
ments, can have the most impact on
learning and development success.
Periodic review meetings represent
an opportunity to review progress,
make adjustments, and check results.
This is a great way to stay connected
and provide feedback to see the results
in order to boost funding in the future.
Getting involved in teaching is a
powerful way to connect learning and
development to the organization and
deliver value. Jack Welch (GE) and
Andy Grove (Intel) are two historical
examples where this type of involve-
ment was extremely effective.
Summary
While the results are based only on 96
executives, the amount of information
is significant. To our knowledge, the
results may represent the highest level
of CEO involvement in research on
measuring the success of learning and
development ever assimilated. The
results present some challenges for
the CLO and the learning and develop-
ment team. T+D
Jack J. Phillips is chairman of the ROI
Institute and an expert on accountability,
measurement, and evaluation; jack@
roiinstitute. Patti P. Phillips is president
and CEO of the ROI Institute and an au-
thority on measurement and evaluation;
patti@roiinstitute.net.
interesteD in orDerinG e-Prints?
Would a digital version of this article be a
great fit for your next course, presentation,
or event? Are you interested in e-prints
of several T+D articles on a specific topic?
Email us the specifics at eprints@astd.org to
learn more.

not proportional to the time the CEO guesswork becomes more inaccurate them into the levels of evaluation. The focus on content. percent review the results of those

48 | 48
T+D | AuguSt
| T+D 2009 2010
| DECEMBER AuguSt
THE BEST OF MEASURING & EVALUATING 2009| |T+D
LEARNING T+D| |49
49
G THE
HITTING THE HITTING THE

ITE
SUITE SUITE
SPO
By Tom Kelly By Tom Kelly

SPOT:
SPOT:
rningHow
Leaders
Learning Leaders How Learning Leaders
Now is a crucial time to secure the
utivesand
CanExecutives
Speak Can Speak and Executives Can Sp
kind of C-level buy-in and support
your learning initiatives deserve.
Language
the Same Language the Same Language
Getting this done is all about
conquering the language barrier.

R R
scores Respondents indicated
ecent research that training
underscores Respondents indicated that training ecent research underscores Respondents indicate
ves leaders often whatstruggle
learning toexecutives
use business leaders often struggle to use business what learning executives leaders often struggle to
des— outcome-based have knownmetrics fortodecades—
communi- outcome-based metrics to communi- have known for decades— outcome-based metrics t
ween cate with senior
effective management
communication about
between cate with senior management about effective communication between cate with senior managem
es in the the impactleaders
learning and efficiency of their oper-
and executives in the the impact and efficiency of their oper- learning leaders and executives in the the impact and efficiency
Even in ations.
C-suite Only 23 percent
is difficult of the respon-
to achieve. Even in ations. Only 23 percent of the respon- C-suite is difficult to achieve. Even in ations. Only 23 percent o
ons that dents indicated
the most theyorganizations
visionary used these types that dents indicated they used these types the most visionary organizations that dents indicated they used
-in, en- ofhave
business metrics,
complete while the
executive majority
buy-in, en- of business metrics, while the majority have complete executive buy-in, en- of business metrics, whil
e budget (62 percent) and
gagement, usedalearning-focused
considerable budget (62 percent) used learning-focused gagement, and a considerable budget (62 percent) used learnin
, lan- metrics such and
for training as Kirkpatrick’s
development, Levels
lan-1 metrics such as Kirkpatrick’s Levels 1 for training and development, lan- metrics such as Kirkpatri
t always through
guage about3 andlearning
volume-baseddoes not metrics
always through 3 and volume-based metrics guage about learning does not always through 3 and volume-ba
usiness on the amount
translate of training
to language aboutdelivered
businessor on the amount of training delivered or translate to language about business on the amount of trainin
learners
and theparticipating.
greater enterprise. learners participating. and the greater enterprise. learners participating.
Expertus Speaking about
The research, training volume
conducted by Expertus Speaking about training volume The research, conducted by Expertus Speaking about trainin
ndustry. and learning metrics
in collaboration with to the C-suite is
TrainingIndustry. and learning metrics to the C-suite is in collaboration with TrainingIndustry. and learning metrics to t
s often similar to ordering
com, found your dinner
that training leadersin Eng-
often similar to ordering your dinner in Eng- com, found that training leaders often similar to ordering your d
and lish whilewith
struggle sitting
how intoa determine
café in Paris.
and lish while sitting in a café in Paris. struggle with how to determine and lish while sitting in a café
metrics Bridging thecritical
communicate gap is possible,
business butmetrics Bridging the gap is possible, but communicate critical business metrics Bridging the gap is pos
tiveness the
thatburden of translation
demonstrate trainingrests on
effectiveness the burden of translation rests on that demonstrate training effectiveness the burden of translation
ment. the
and shoulders
success to ofsenior
learning leaders.
management. the shoulders of learning leaders. and success to senior management. the shoulders of learning

LISTEN TO THIS FEATURE

at www.astd.org/TD/TDpodcasts.htm

50
48 | T+D | DECEMBER
SEPTEMBER2010
2009 Photos by Shutterstock.com Photos by Shutterstock.com 48 | T+D THE
| SEPTEMBER 2009
BEST OF MEASURING & EVALUATING LEARNING
SEPTEMBER 2009 | T+D | 51
49

WHAT’S NEXT?
Here are a few questions that
can help gauge the success
of communications between
learning leaders and the C-suite:
How does your learning
organization define its number
one mission for the enterprise?
(If it’s generating customer
training revenue, you are probably
on the wrong path. That’s easy,
but almost always financially
irrelevant in most companies.)
Do the C-Suite executives know
about that primary mission?
Do they publicly agree with or
support it?
How will success of that mission
be recognized if it happens? What
are the metrics of success?
Is your learning organization
tracking, measuring, and
reporting on formal training and
informal learning?
Does your learning organization
need to improve its business
language skills? Its metrics?
Unfortunately, “fill rates” (how full a
class is) and “hours of training” are
almost never the terms that bring
management to their feet. These are
“busy-ness” metrics that have no
bearing on the success of the enterprise.
Instead, key phrases such as “increased
sales,” “reduced costs,” “improved
productivity,” or “increased customer
satisfaction” are terms that will hit home
with C-level executives concerned about
the performance of the enterprise.
The secret to learning leaders com-
municating with the C-suite resides in
identifying metrics that align with the
business goals and succinctly deliv-
ering them in a format that reflects
the C-level view of the enterprise and
concern about the future.
50 | 52
T+D| T+D
| SEPTEMBER 2009
| DECEMBER 2010
PURPOSEFUL
LEARNING PURSUITS
Training and learning has come a
long way over the years, evolving from
a “nice-to-have” to “a must-have.”
Today, learning organizations are be-
coming cornerstones of successful
enterprises that recognize that their
most significant asset is their people.
Senior management has come to
accept that if managed properly,
learning organizations can accelerate
enterprise growth, reduce risk, and
advance organizations in the knowl-
edge-based economy. These are facts
that learning leaders have believed
for a long time, but have been unable
to prove.
In my opinion, a training or learn-
ing department should exist for only
three reasons:
• to increase top-line revenues
• to heighten productivity and
therefore reduce costs
• to improve customer satisfaction
and loyalty.
Here are some guidelines to keep in
mind when looking to translate learn-
ing activity into enterprise results,
whether you are a C-level executive or
the leader of a training organization.
METRICS, METRICS, METRICS
Learning departments are often
blamed when a group isn’t measuring
up to expectations, because it appears
as though the group were not trained
properly. Conversely, learning
departments often complain about
metrics around what knowledge gaps
need to be filled. Without baseline
data, it’s difficult to fix this training
problem. Let’s take an example:
In Company XYZ, the call center
has been underperforming. Em-
ployees have been unable to take
calls quickly enough, and a large
percentage of calls have necessi-
tated a supervisor’s involvement.
The director of the call center went
to the executive team to complain
about the poor training that the
call center’s employees received.
After the executive team briefs
the training leader about the
complaints, it seems evident that
the learning program needs to be
enhanced. As a result, the training
organization immediately begins to
revamp the whole program: bolster-
ing courses, adding instructor-led
courses, and reallocating the budget.
However, the call center issues are still
the same 14 months later. Now what?
There were no metrics determined
to measure the issue. Therefore, there
was no strong basis for the solution.
It became an expensive endeavor on
blind knowledge that “something”
wasn’t working.
The key here is to have solid metrics
defining the problems to have mean-
ingful metrics for solutions.
Learning leaders need to get to the
root of the problem and build a busi-
ness case for a well thought-out strate-
gic solution to a well-defined, docu-
mented, and understood problem.
The C-suite, business head, and
learning leaders need to agree on the
problems and build metrics in which the
solutions can be measured. This requires
strategic alignment and communication
about metrics so the problem and solu-
tion can be accurately assessed.
Once problems are resolved and
results (such as retained customers,
increased sales, or shorter time to pro-
ductivity for new hires) can be docu-
mented and shared, only then have
learning leaders demonstrated value
to the enterprise. The difficulty in this
approach is that learning professionals
want to measure the effects on indi-
viduals (microresults), and the C-suite
wants to measure the bearing on the
enterprise (macroresults).
SAVE MONEY AND MAKE MONEY
The most successful learning leaders
operate more like a profit center than a
cost center, whether or not they actu-
ally bring in any direct revenue. This
means approaching each decision as
an investment—a business decision
with a measured, quantifiable return
in top-line revenue; productivity; or
satisfaction and loyalty.
It’s a fact that effective training
programs with sales teams, resellers,
and channel partners can drive in-
creased top-line revenue. Be creative
with ways to educate these critical
constituencies and turn the learning
department into a meaningful, effec-
tive part of the business.
If every dollar you invest in train-
ing the internal salesforce and the
channel partners, for example, can be
linked to an incremental $3 of product
or service revenue, then the enterprise
will notice and care about the learning
organization. Often, it probably won’t
be a direct cause-and-effect link, but a
correlation between training and top-
line revenue growth can be demon-
strated with minimal effort.
Scalability is equally important be-
cause it extends the reach and reduces
the costs associated with learning.
Learning leaders need to select ap-
propriate deployment methods proven
to be an effective way to manage costs.
By critically selecting from instructor-
led courses, webinars, e-learning, and
other channels, the “investment deci-
sions” will pay off.
FORMALIZE THE INFORMAL
The key to the learning business is in-
vesting dollars and manpower wisely.
Learning leaders need to take the time
to look at the organization’s business
objectives and decide which metrics
are effective.
While learning management sys-
tems can be excellent at tracking some
course (formal) metrics, approximately
80 percent of what an employee does
to learn about his job is informal learn-
ing. Employees execute quick research
(for example, via a Google search or an
internal learning database), find the
appropriate information, and go back
to work. For the first time in decades,
informal learning is taking a central
role in organizations, and both the
C-suite and learning leaders need to
embrace new ways of sharing informa-
tion and knowledge and measure them
more effectively.
In August 2007, Forrester Research
analyst Claire Schooley (with Kyle
McNabb and Shelby Semmes) released
research indicating that “informal
learning continues to gain ground in
the corporate training environment as
Photo by Shutterstock.com
instant messaging (IM), blogs, commu-
nities of practice, and expertise location
The key to the learning
find homes in lines of business. Today’s business is investing
workers enter the workforce expecting dollars and manpower
access to the same kind of tools they
use in their personal life to help them
wisely. Learning leaders
do work. But there’s a disconnect: most need to take the time to
organizations still spend the majority of look at the organization’s
their training dollars on formal learning
even though most employees now learn
business objectives and
primarily through informal activities.” decide which metrics
There are a variety of new tools that are effective.
have the capacity to measure informal
learning—on-demand video deploy-
ment tools, “audio over slides” presenta-
tions, on-demand webinars, searchable
white papers, wikis, blogs, communi-
ties of practice infrastructures, and
other implements. Thanks to these new
measurement tools, we have access to
metrics such as number of downloads,
page views, and click-throughs.
The added benefit of informal
learning techniques is that companies
have lower costs and a scalable busi-
ness solution that yields tremendous
returns. T+D
Tom Kelly is an advisor to small- to mid- INTERESTED IN ORDERING E-PRINTS?
Would a digital version of this article be a
sized companies focusing on learning orga- great fit for your next course, presentation, or
nizations, learning systems, and enterprise event? Are you interested in e-prints of several
T+D articles on a specific topic? Email us the
learning strategies; tomkelly58@gmail.com.
specifics at eprints@astd.org to learn more.
SEPTEMBER
THE BEST OF MEASURING & EVALUATING 2009 | T+D | 53
LEARNING 51
 There is no stakeholder in the learning and development
process more important than the CEO of an organization.
In the August 2009 issue of T+D, we examined the results
of a significant survey of CEOs. This study revealed what
CEOs want and do not want for measures of success of
learning and development. In this article, we’ll combine
those survey results with other executive input to develop
prescriptions for what is needed to ensure that learning
professionals meet the expectations of this critical group.

Executive concerns
All of the data points to several critical concerns, and consequently, opportunities for
learning and development leaders. As expected, CEOs want to see value in terms that
they can appreciate. They view the value of learning and development in terms of busi-
ness impact, business alignment, and return-on-investment. They do not see value in
the inputs (how many people attended training) or reaction (participants rated overall
satisfaction 4.2). There is very little data at the business contribution level presented to
them, and yet, that is their most important data set. Ninety-six percent of the CEOs said
that they wanted to see learning and development connected to business impact data,

ONFRONTING CEO CEO

CONFRONTING
but only 8 percent see this now. Seventy-four percent wanted to see ROI data, yet only 4
percent see it now.
They see little value in lower levels. As anticipated, top executives see very little
value in having a tremendous amount of input data. Although it is needed (they want

EXPECTATIONS
to know how many people are involved, how many hours, and the cost), this informa-

EXPECTATIONS
tion does not show value in terms of contribution; it’s only input. Also, the least-ranked
data item is reaction data. Executives view this as people enjoying their experiences. A
few suggested that it is important, because satisfied customers are necessary to make
the process work; however, they do not see the connection with actual business results.

BOUTABOUT
THE VALUE
THE VALUE
Additionally, there is not much appetite for learning data. As a few executives com-
mented, that’s an operational measure.
Executives want to see the higher levels of evaluation. Interest picks up with applica-
tion data, but there seems to be a disconnect here. Sixty-four percent of executives said

OF LEARNING
that they would like to see application data (change in behavior, use of skills, use of tech-

OF LEARNING
nology), yet only 11 percent actually have this data. More revealing was the tremendous
disconnect with this business impact and the ROI, as discussed earlier. In comments
on these surveys, executives routinely mention terms such as “business contribution,”
“business alignment,” “business value,” and “connection” to the business. Eighteen per-
By Jack J. Phillips and Patti P. Phillips cent of CEOs said that they determined the funding for learning and development based
By Jack J. Phillips and Patti P. Phillips
on the payoff of the investment.
vel support is vital to maintaining any organizational learning
Top-level support is vital to maintaining any organizational learning
Survey data provide some clear directions for how, and how not,
function. Survey data provide some clear directions for how, and how not,
e executives with the appropriate evidence of learning success.
to provide executives with the appropriate evidence of learning success.
LISTEN TO THIS FEATURE
at www.astd.org/TD/TDpodcasts.htm
Photo by Corbis
54
52 | T+D | DECEMBER 2010
JANUARY 2010 Photo by Corbis THE BEST OF MEASURING & EVALUATING
JANUARY
LEARNING
2010 | T+D | 55
53

Most CEOs do not see a learning and
development scorecard. If learn-
ing and development organizations
are using a scorecard, it’s not making
its way to the top executives. Only 22
percent of executives said that they
have a learning and development
scorecard. Executives noted that if
there were a scorecard, they would
probably use it and use it properly be-
cause they’re applying it across many
parts of their organization. The score-
card, however, must not be dominat-
ed by inputs (Level 0), reaction (Level
1), and learning (Level 2). It must have
information that includes application
and that connects to the business in
terms of both tangibles and intangi-
bles. And yes, awards and ROI would
be appropriate, too.
Other concerns. CEOs are not as ac-
tively involved in learning and devel-
opment as they need to be, and like
many other functions in an organiza-
tion, learning and development needs
executive support. Direct involvement
by top executives is an excellent way to
stimulate the interest and ownership
of others and helps drive results for the
various programs. More effort is need-
ed to increase executive commitment.
This often creates a dilemma because
executive commitment is needed to
drive results, yet one of the best ways
to secure additional commitment is
54 | T+D | DECEMBER
56 JANUARY 2010
2010
to show results. This will represent a
challenge for the learning and devel-
opment team.
Less than half of the CEOs refer to
the learning and development func-
tion as a corporate university, at least
by the traditional definition of corpo-
rate university. Some are pleased with
that concept, while others believe that
it can go astray and create an entity
that is not relevant, and perhaps even
unattached, to the organization. Most
of the executives were concerned that
learning must show its value in terms
that can be clearly seen.
Most CEOs consider their compa-
nies to be “learning organizations,”
and they discuss the possibilities of
learning from all perspectives. But as
companies use more learning vehicles
and approaches, there is a need for
those vehicles to be relevant, connect-
ed, and driving results in some way.
What caused this situation?
Many professionals in this field know
how to connect learning and develop-
ment to the business. Practitioners
have been exploring and discussing
this issue for more than half a century.
Since then, practitioners, researchers,
and professors have improved evalua-
tion practices and have written dozens
of books on the evaluation of learning
and development.
However, most observers in this
field conclude that there has been little
actual progress when it comes to con-
necting learning and development to
the business. Our primary stakehold-
ers—top executives—who must fund,
support, and make a commitment to
learning and development, are not
necessarily happy about it. Unfortu-
nately, there are real barriers for not
addressing this. The real barriers often
are not identified directly and explic-
itly in benchmarking and survey data.
Fear of results. Perhaps the stron-
gest barrier is fear of disappointing
results, whether that means a nega-
tive ROI study or results that are far
less impressive than executives ex-
pect. It’s a fear of the consequences.
We often encounter this comment,
“If my key program is not delivering
business value, why should I conduct
a study to show my top executives
that it’s not working?”
The fear is that the results may be
used for a performance review of the
chief learning officer, lead to a cut in
the budgets, or be used to hold specific
team members accountable. If a pro-
gram is not delivering results, the cli-
ent probably already knows it. He just
doesn’t yet have the study to show the
details. After all, business results are
generated in the business units, and if
the business measures are not improv-
ing, the executives are aware of it. So a
negative or disappointing study may
not surprise them.
A much better approach is to
tackle the issue on a proactive basis
with process improvement in mind.
Take a high-profile strategic pro-
gram, and conduct an analysis of
its success with the goal of process
improvement; so if it’s not working,
plan to make changes to make it suc-
cessful in the future. This proactive
approach wins points with senior
executives and helps to improve the
partner relationship.
Waiting for the request. Closely
linked to the previous barrier is
waiting for the request to show the
business contribution. Often, we see
CLOs interviewed on the issue of ROI
and business connection. Some CLOs
state that they’ve never been asked
for ROI, and thus, there is no need
to pursue it. There is faulty logic at
work here because if you wait for the
request, it will often be too late.
When a request for ROI (or busi-
ness impact) on a particular program
is made, results are often expected as
soon as possible. Without prior plan-
ning for a study, an executive waiting
for results will become impatient. It
takes time to develop processes, build
capabilities, change practices, and col-
lect and analyze data. When you wait
for the request, you are on the execu-
tive’s timeline and agenda, which is an
uncomfortable place to be.
Smart CLOs are taking the initia-
tive to develop this capability before it
is requested. They are controlling the
agenda and the timeline, providing the
Photos by iStockphoto.com
executives with a healthy dose of ac-
countability, routinely and consistently.
Lack of investment. Let’s face it: We
have not invested enough funds for
measurement and evaluation processes.
To determine an appropriate level of
investment, estimate the cost for mea-
surement and evaluation as a percent of
the learning and development budget.
This includes any expenditure for staff
and resources for collecting data (mea-
surement) and using the data to make
adjustments (evaluation). For most or-
ganizations, expenditure is at 1 percent
or less. Annually, we benchmark with
organizations using a comprehensive
measurement and evaluation system,
and these best-practice organizations
spend between 3 and 5 percent on mea-
surement and evaluation.
To make the case for additional
funds, show executives the bench-
marking data of best practices, and
show that your investment in mea-
surement and evaluation for learning
falls far short of other processes in the
organization. Then, fund measure-
ment and evaluation gradually with
the success generated by the process.
For example, the first impact or ROI
study will show how a program can
be improved or how it can be done
at lower cost or in less time. In either
case, there is added value because
you’ve actually evaluated the pro-
gram. With results in hand, you have
a great opportunity to make the case
for additional measurement and
evaluation funds.
Think about evaluation early. The
time to think about evaluating a partic-
ular program at the business level is at
the time of conception. Unfortunately,
by habit, practice, or teaching, we don’t
think about evaluation early enough.
For years, learning and development
professionals used the ADDIE model—
analyze the need, design the solution,
develop the solution, implement the
solution, and evaluate the solution.
Unfortunately, this model causes us to
think about evaluation after the pro-
gram has been implemented. For most
projects, this is too late.
By thinking about evaluation much
earlier, we ensure that the solution is
connected to clearly defined business
measures. Objectives are developed at
multiple levels, including application
and impact, to provide the proper focus
throughout the program. Expectations
are created for participants and oth-
ers to clearly see why the program is
being offered and what their role is in
its success. Data collection is built into
the program to make it more palatable
to the participants. Planning for data
collection is accomplished early, mak-
ing the process much more efficient
while assigning more responsibilities
to others.
More effort is needed
to increase executive
commitment. This often
creates a dilemma because
executive commitment is
needed to drive results,
yet one of the best ways
to secure additional
commitment is to show
results. This will represent
a challenge for the learning
and development team.
They asked for it. The learning and
development team is a support group,
supporting the needs of the organiza-
tion by delivering learning and devel-
opment to satisfy those needs. When
a top executive requests a leadership
development program or an operating
executive requires technical training,
the programs usually are provided with
the anticipation of results. After all, they
are requested by someone who should
understand the requirements and
needs, and they are designed to meet
those needs. So why should we evaluate
these programs? Isn’t it assumed that the
value exists? And why waste resources
trying to prove value when the requestor
already understands that value?
This seems logical; unfortunately
however, the logic breaks down. Execu-
tives often request a learning solution
when they see a problem. If something
is not working in the organization, those
THE BEST OF MEASURING & EVALUATING
executives assume that employees don’t
have the knowledge or skills that they
need. Research continues to show that
when there’s a dysfunctional or ineffec-
tive process, the most appropriate solu-
tion is often a nonlearning solution.
Unfortunately, managers are not
experienced in the analysis techniques
needed to determine the cause of the
problem and the appropriate solu-
tion. The managers are not the bad
guys, though they appear to be. We
force them into this process, and we’ll
have to change their behavior. But we
have to do it diplomatically, subtly,
and gradually, teaching them to en-
sure that the request is appropriate.
Lack of preparation. Unfortunately,
learning and development profession-
als often don’t possess the skills and
knowledge needed to implement evalu-
ation to demonstrate business impact.
Some have the luxury of a degree in
learning and development, instruc-
tional technology, or human resource
development. Most do not. Even then,
the curricula usually focus very little on
measurement and evaluation—perhaps
one course on evaluation and maybe
one that covers needs assessment. The
bottom line is that most learning and
development team members have no
formal education in this area.
Many learning and development
professionals transfer from other areas
in the organization. When individuals
enter the field, they have almost no in-
sight into, nor experience with, evalu-
ation. Preparation to address these
issues is minimal at best. The problem
is then exacerbated by the excessive
number of evaluation books offered to
these unknowing practitioners, often
leaving them confused.
Changes are needed
Not all of the comments and data
from the executive study point to a
negative situation. Many CEOs are
proud of the learning functions in
their organizations and note the
contributions that are made. These
executives are supportive and en-
couraging, and they are thoroughly
committed to making learning a part
of their growth and profit strategies.
JANUARY
2010 || T+D
LEARNING T+D || 57
55
 At the same time, even these CEOs
see a great deal of room for improve-
ment, leaving the learning and devel-
opment team, particularly the CLO,
with some important challenges.
Before taking action, it is helpful to ex-
amine the barriers described earlier to
see if you agree with our assessments.
Explore, examine, and evaluate the
barriers; make sure you clearly under-
stand the issues; and be prepared to
minimize or remove them.
The CLO is in a critical role here.
We’ve yet to see a successful evaluation
system implementation without the
support, commitment, and involve-
ment of the CLO. We’ve seen some en-
thusiastic learning and development
team members try, but eventually fail
without CLO support.
The challenge is to focus on re-
sults throughout the learning and
development cycle. The chart shows
the 10 key areas where the focus on
results should be addressed, detail-
ing what actions are needed, what’s
involved, and the payoff. These are
basics—the fundamentals—but they
are often ignored in some learning
and development functions.
Move quickly to make changes, and
take action in this area. Don’t put it off
to next quarter or next year. Here are a
few actions to take now:
1| Assess where you are now with the
results-based approach. Samples
of assessment instruments
are available directly from the
authors. With this assessment you
can plan specific actions.
2| Invest more in measurement and
evaluation by using one of the more
specific strategies suggested here.
3| Change practices and address
evaluation early and often in the
process, building evaluation into
learning and development.
4| Focus on objectives and expand
them beyond the learning
objectives. Require application
and impact objectives for 70
percent of new programs. This
alone will drive more results than
any other action.
5| Take a fresh look at a learning
scorecard, perhaps building
56 | T+D | DECEMBER
58 JANUARY 2010
2010
one that reflects some of the
data in this article. Don’t let
the top executive team design
your learning and development
scorecard. They don’t know how
to do it, and in all likelihood, it
will be something that’s almost
impossible to do. Instead, show
them something they can review.
6| Invest in technology, or at least use
the technology at hand, to assist
with the evaluation challenge.
7| Build a measurement culture.
Ask questions. Require data. Ask
people to think about results,
accountability, measures, metrics,
and analytics, but not to an
extreme. Make accountability a
routine part of conversations,
expectations, and ultimately, the
reward structure.
8| Conduct a few impact studies and
maybe, occasional ROI research, for
those programs that are substantial,
strategic, expensive, and high
profile. You know the ones—they
attract attention from the senior
team, and they often require higher
levels of accountability.
9| Start providing information about
successes to the appropriate
executives. Feed their appetites.
Don’t promise too much, but
deliver more. Make it routine.
Avoid the request for showing the
value of your entire function. If
that request is coming, it may be
too late.
10| Get your executives more
involved. Executive involvement
helps to keep the focus on results
and accountability.
11| Never miss an opportunity to
speak to an executive about
the success of programs in
their area. After all, it is their
team that delivered this great
performance, and these
successes are important to them.
Now is the time
As we emerge from the recession, there
will be a greater focus on account-
ability for all types of projects and
programs. Functions across the entire
organization will be making a plea for
increased funding.
Those programs that make the best
business case will ultimately prevail.
Executives will require more evidence
that increased funding will drive busi-
ness results. It is absolutely critical that
leaders of the learning and develop-
ment function prepare to face this
challenge and demonstrate the con-
nection to the business. We have not
done so well with this in the past, but
it’s fundamental, and it can be done.
Now is the time to pursue it.
Jack J. Phillips is chairman of the ROI
Institute and an expert on accountability,
measurement, and evaluation; jack@
roiinstitute.net. Patti P. Phillips is president
and CEO of the ROI Institute and an
authority on measurement and evaluation;
patti@roiinstitute.net.
INTERESTED IN ORDERING E-PRINTS?
Would a digital version of this article be a great
fit for your next course, presentation, or event?
Are you interested in e-prints of several T+D
articles on a specific topic?
Visit astd.og/TD/eprints for more information.
Photo by iStockphoto.com
A RESULTS-BASED APPROACH TO LEARNING AND DEVELOPMENT
SPECIFIC WHAT’S INVOLVED PAYOFF
ACTION
Validate that the learning solution is needed. Too
Make sure
often, programs are implemented when there Prevents unnecessary programs from
1 the solution is
isn’t a learning need, or it’s a solution in search being implemented.
appropriate.
of a problem.
Ensure the project or program is connected to Ensures business alignment at the beginning
Align the program
2 business measures by identifying the specific of the process. Ensures that more programs
with the business.
business need for the program. are driving business improvement.
Develop a clear understanding of what the target Enhances program success by identifying
Identify job
3 audience must do differently as a result of this those behavior, task, or action items that will
performance issues.
new program or project. be implemented.
Make sure that Examine the context of the learners’ workplace
the barriers to the to see what will enhance and what will inhibit Increases the likelihood that the program is
4 transfer of learning the transfer of the learning to the job. Take successful by examining what will make it
are minimized appropriate actions to either minimize or remove successful in the work environment.
or removed. those barriers, or enhance the enablers.
Extend learning objectives to application
Set objectives at
(defining what participants should do) and Provides the focus, guidance, and direction
5 the application and
impact objectives (the consequences of that necessary to achieve results.
impact levels.
application; business impact).
Create expectations
Detail responsibilities to ensure that participants Ensures that participants apply what’s
with participants
6 know what’s expected of them and their role in being learned and achieve success at the
and other key
achieving results. impact level.
stakeholders.
Make data collection a part of the program, even
Build in data Positions evaluation as application and
7 for follow-up evaluation. Provide “what’s in it for
collection tools. creates more incentives for the participant.
me” details.
Identify key Strengthen support and reinforcement for Involves the immediate manager. This is
8 managers who results with key managers (usually the perhaps the most important action to
influence results. manager of the participant). enhance results.
Move beyond your current dominant level of
Push evaluation to evaluation, which is usually around Level 1 Gains attention of the management team.
9
the next level. (reaction) and Level 2 (learning), and push the They are interested in levels 3, 4, and 5.
evaluation to application and impact.
Use data collected about the success, or
lack of success, of programs to make Improves programs so they are more
Use data to make
10 adjustments and changes, with the effective, efficient, and often, more focused
improvements.
understanding that the primary reason for on business results.
evaluation is process improvement.
THE BEST OF MEASURING & EVALUATING LEARNING
JANUARY 2010 | | T+D
T+D | | 59
57
 OvercOming Evaluation barriers Figure 1: Usage and value of the Kirkpatrick/Phillips model levels

Barriers
The survey probed into the perceived
barriers that prevent companies from Percentage who use the Percentage who say this
Responses corresponding level to level has high or very
using all of the evaluation levels. The
any extent high value*
barrier that looms largest is the dif-
ficulty in isolating learning as a fac- Reactions of participants 91.6% 35.9%
tor that has an impact on results (52

and valuing percent endorsed it to a high or very Evaluation of learning 80.8% 54.9%
high extent), which comes into play
mostly for Level 4 and Level 5. The next Evaluation of behavior 54.6% 75.0%

evaluation
most commonly noted barrier, cited by
41 percent of respondents to a high or
Evaluation of results 36.9% 75.0%
very high degree, is the lack of a useful
Return-on-investment 17.9% 59.4%
evaluation system with the learning
management system. Thirty-eight per- None of the above 0%
4.1%
By Laleh Patel cent of responding organizations noted
* Only among those who say they use this level to some extent
an additional barrier—evaluation data
An ASTD-i4cp study underscores the are not standardized enough to com-
progress organizations are making in pare well across functions. more respondents indicate that their These findings provide insight
the area of learning evaluation, as well Each barrier was negatively corre- evaluation data can’t be compared eas- into specific steps that organizations
as the gaps that need closure. lated with the Evaluation Success Index
(ESI), a measure of the extent to which
ily across functions, the less likely they
are to report organizational success
can take to avoid learning evaluation
pitfalls. Learning evaluation data loses

F
indings from the American
Society for Training & De-
velopment (ASTD) and the
Institute for Corporate Productivity’s
(i4cp) study, The Value of Evaluation,
highlight that progress is being made
in measurement practices, as the pur-
suit of excellent learning evaluation
continues. However, there is still room
for development because few orga-
nizations think that they’ve mastered
learning evaluation, and many admit
to facing some ongoing challenges.
ASTD and i4cp partnered to explore
the complex issue of learning evalua-
tion and the value of learning. A total
of 704 human resource and learning
professionals completed an online
survey during May 2009. The majority
of them (85 percent) were managers,
directors, vice presidents, or C-level
officers, and the sample was evenly
distributed in terms of workforce size,
annual revenue, and industry.
Kirkpatrick/Phillips
evaluation model
The findings revealed that the five-
level Kirkpatrick/Phillips model of
learning evaluation is the most com-
mon practice. The five levels include
participant reaction (Level 1), level of
learning achieved (Level 2), changes
in learner behavior (Level 3), business
results derived from training (Level 4),
and return-on-investment from train-
ing (Level 5). While this finding was
not shocking, there were some inter-
esting results to additional questions,
such as how frequently these levels of
evaluation are used and how valuable
they are to organizations.
The survey data answered these
questions, showing that the majority of
respondents (92 percent) measure their
learning programs to at least Level 1 of
the model. This isn’t surprising given
that Level 1 is the easiest metric to
track, usually with “smile sheets.” The
use of other types of evaluation drops
off as we move up each subsequent
level of the Kirkpatrick/Phillips model,
which is likely due to the increasing dif-
ficulty in obtaining the information at
each level.
A little more than four out of five re-
spondents said that their organizations
evaluate at Level 2, where companies
explore what knowledge was gained.
Level 3 registered another drop in us-
age, with more than half of the respon-
dents indicating that their organization
evaluates learners’ behavior. Level 4
measures an important area—results,
which only 37 percent evaluate to
any degree. Level 5 received the least
consideration, with only 18 percent
of respondent companies measuring
the impact on ROI. The survey results
indicate that translating the value of a
training program into dollars and cents
can be a difficult process.
The most used levels are not the
most valuable
One of the interesting insights from
the survey data was that the extent
to which training evaluation occurs
at the different levels of the Kirkpat-
rick/Phillips model does not tell us
much about its perceived value. Al-
though Level 1 is the most used form
of evaluation, it was found to be the
least valuable for organizations. Only
36 percent of respondents said that
Level 1 had a high or very high value.
By comparison, 55 percent said that
Level 2 had high or very high value,
and three-quarters said the same
about Level 3 and Level 4.
At first, these findings may strike
some as perplexing, but there is
almost always more value in knowl-
edge gained, behaviors changed, and
results achieved than in participants’
reactions. It therefore prompts the
question as to why companies are not
evaluating more programs at higher
levels. Of the few companies that
measured Level 5, nearly 60 percent
perceived it to have a high or very
high value.
respondents believe their learning
metrics are a worthwhile investment
of time and resources. The intention
was to assess which barriers are most
strongly associated with successful (or
unsuccessful) evaluation. The stron-
gest correlation with the ESI occurs
when evaluation data is not standard-
ized enough to compare well across
functions (r=-0.23). In other words, the
Figure 2: Barriers to the evaluation of learning in organizations
Responses
Too difficult to isolate
training’s impact on results
versus other factor’s influence
Our LMS does not have a
useful evaluation function
Evaluation data is not
standardized enough to
compare well across functions
It costs too much to conduct
higher-level evaluations
Leaders generally don’t care
about evaluation data
Evaluation data is too difficult
to interpret for most people
Evaluations are not seen
as credible
** Correlation is significant at the 0.01 level (2-tailed)
Note: Only statistically significant correlations are presented.
with overall evaluation efforts. Another some of its value when it cannot be
significant correlation between the ESI compared across functions. Therefore,
and a barrier occurs when the LMS taking the time to coordinate across
does not have a useful evaluation func- functions may be beneficial. Addition-
tion (r=-0.21). This negative association ally, taking steps to ensure the useful-
suggests that the more respondents ness of the evaluation function of an
believe that their LMS does not have organization’s LMS can be beneficial.
an evaluation function that meets their Phillips has also suggested using meth-
needs, the less likely they are to give ods such as control groups, trend-line
their evaluation efforts high marks. analysis, forecasting models, and im-
pact estimates to help isolate training’s
affect on results.
Percentage who see Correlation Conclusion
these as barriers to a high with evaluation Organizations are always searching for
or very high extent success index ways to demonstrate the increasing value
of training, especially in the current eco-
nomic climate. Although many are tak-
51.7% -.18**
ing the initial steps to achieve excellent
learning evaluation for their training
40.8% -.21** programs, there is still much ground to
cover in their pursuit to master the mea-
surement techniques and to overcome
38.0% -.23**
the challenges they face. T+D
For more information about the aSTD-i4cp study, visit
32.2% www.astd.org/content/research.
24.1% -.12** Laleh Patel is research associate for ASTD;
lpatel@astd.org.
18.9% -.20**
14.5%
Listen to this feature
at www.astd.org/TD/TDpodcasts.htm

60 | T+D | February
62 DECEMBER 2010 Photos by iStockphoto.com February
THE BEST OF MEASURING & EVALUATING 2010 | T+D | 61
LEARNING 63
 RetuRn on expectations (Roe) is Rapidly
RetuRnthe
sweeping on expectations
leaRning and (Roe) is Rapidly
development
sweeping the leaRning and development
industRy. the Reason is quite simple:
industRy.
when the Reason
used pRopeRly, is quite
it neveR simple:
fails to
when
show used
the pRopeRly,
value it neveR
of tRaining fails
in the to
teRms
show by
desiRed the value
key of tRaining in the teRms
stakeholdeRs.
desiRed by key stakeholdeRs.
This is because ROE is designed and executed in partnership
with theThis
people who will
is because ROEdetermine learning’s
is designed value. The
and executed Kirk-
in partnership
patrick
withFoundational
the people who Principles explain the
will determine ROE process
learning’s and Kirk-
value. The the
critical importance
patrick Foundationalof business partnership
Principles explainin achieving
the it: and the
ROE process
• critical
The endimportance
is the beginning.
of business partnership in achieving it:
By James D. Kirkpatrick and • ROE is the
• The endultimate indicator of value.
is the beginning.
Wendy Kayser Kirkpatrick • Business
• ROE is partnership
the ultimate is necessary
indicator oftovalue.
bring about
positive
• BusinessROE. partnership is necessary to bring about

ROE’s
• Value must be
positive created before it can be demonstrated.
ROE.
• A compelling
• Value must chain of evidence
be created beforedemonstrates your
it can be demonstrated.
bottom line value.chain of evidence demonstrates your
• A compelling
bottom line value.

Rising
staR
Why return on
expectations
is getting so much attention

listen to this feature

at www.astd.org/tD/tDpodcasts.htm
listen to this feature
at www.astd.org/tD/tDpodcasts.htm
Photo by Veer August 2010 | T+D | 35
62 | T+D | August
34 DECEMBER
2010
2010 Photo by Veer THE BEST OF MEASURING & EVALUATING
August
LEARNING
2010 | T+D | 63
35
 The end is the beginning instead of relating training to the
Showing the value of training begins accomplishment of the highest level
Next, learning professionals should
work with the managers of the in-
PricewaterhouseCoopers and ROE
when executives before the program even starts. Donald business goals. Here are some com- tended training participants. Together In April 2010, Jim Kirkpatrick delivered a Kirkpatrick Four Level Evaluation
ask for new training, L. Kirkpatrick says it best in Evaluating mon examples of true Level 4 results: they should identify the critical be- Certification Program in Vancouver, British Columbia. The group was treated
Training Programs: The Four Levels: • sales to a conversation between two PricewaterhouseCoopers leaders: Eric Andrew,
many learning “Trainers must begin with • employee retention
haviors of training graduates, which, if
performed consistently on the job, will national private company services leader for PwC Canada, and Deana Gill, a
professionals retreat desired results and then deter- • profitability most likely produce the desired results. program attendee and national PCS learning and development senior manager.
to their departments mine what behavior is needed • saving or extending lives Only then can the training professional Both strongly acknowledged that they are indeed strategic business partners,
to accomplish them. Then train- • prevention of damage to and that the success in Eric’s area was only possible through close collaboration
and start designing ers must determine the atti- the environment.
proceed with the traditional identifica-
tion of learning objectives. and an integrated team approach.
and developing tudes, knowledge, and skills that It is also critical for the training At the same time, the training pro- One of the other program participants asked Eric, “What do we have to do to
suitable programs. are necessary to bring about professional to find out how important fessional should ask what evidence will get the attention of and buy-in from senior business leaders like you?” Without
the desired behavior(s). The the initiative is to the organization and pause, he answered, “Show me how what you do contributes to my results.” Eric
sometimes, a training final challenge is to present the obtain some idea of the resources that
be required at each of the four levels
to show that the initiative was a suc- and Deana then led a lively discussion, with ROE taking center stage.
needs assessment is training program in a way that can be allocated to it. Not every single cess. When the measurement meth- It was determined that ROE must begin with a business partnership mental-
conducted, but it is enables the participants not initiative and program will warrant a ods, tools, and techniques are defined ity and approach. This involves the identification of the specific “needles” (busi-
only to learn what they need to complete execution plan as outlined in ness and human resource metrics) that must move in order for the initiative to
rarely taken beyond know but also to react favorably this article. The level of effort should
at the start of the initiative, it makes
evaluation much easier to perform. be a success. This approach must also include agreement on the level of effort
training and into critical to the program.” be in alignment with the importance (in addition to training) required to bring about those results.
performance and The clearer a training professional of the initiative to the success of Business partnership is After agreeing on the specific metrics and execution plan, training profes-
is on the desired Level 4 results of the organization. necessary to bring about sionals (such as Deana) and the identified program champions make it happen.
business needs. this is a solution (See Table 1), the better positive ROE They then need to present the chain of evidence to senior leaders (such as Eric)
where a lot of training chance that his time and effort will ROE is the ultimate Once there is a clear understanding showing how these efforts contributed to the overall result.
waste occurs. cost-effectively accomplish that end. indicator of value of the result to be accomplished, the
A common error many training When executives ask for new training, next step is for training professionals
professionals make is defining de- many learning professionals retreat to to work with business managers and
sired results at the level of training their departments and start design- supervisors to create a tactical execu-
ing and developing suitable programs. tion plan. This is necessary because mance of critical behaviors. The degree ASTD State of the Industry report).
Sometimes, a training needs assess- training events alone typically result in to which drivers occur directly relates It is only 10 percent of the time that
ment is conducted, but it is rarely only 15 percent transfer of learning to to the extent to which critical behaviors poor performance is due to something
Table 1 taken beyond training and into critical on-the-job behavior, according to are performed. Performance of critical that did or did not happen during
performance and business needs. This Robert Brinkerhoff. behaviors is what yields business-level formal training.
to what degree targeted outcomes occur, as a result of the learning is where a lot of training waste occurs: Historically, learning profession- results. Results determine the ROE from Very little time and attention is
level 4: results
event(s) and subsequent reinforcement. training that isn’t needed is conducted, als have focused most of their efforts the initiative. being given to the critical on-the-job
to what degree participants apply what they learned during training and the required training or other so- on training events (Levels 1 and 2); application environment for new
level 3: Behavior
when they are back on the job. lution that would be truly beneficial is however, the largest ROE occurs at Value must be created knowledge and skills. To create maxi-
to what degree participants acquire the intended knowledge, skills,
not even identified. Level 3, on the job. For maximum ROE, before it can be demonstrated mum ROE, it is essential that learning
level 2: learning Stakeholder expectations define there needs to be agreement about At this point in the process, desired professionals do everything possible
and attitudes based on their participation in the learning event.
the value that training professionals the intensity of effort required from results are clearly defined, and a cross- to partner with supervisors to support
level 1: reaction to what degree participants react favorably to the learning event. are responsible for delivering. Learning all departments before, during, and functional execution plan has been on-the-job behaviors and executing
professionals must ask the stakehold- particularly after training. Specifically, created. What remains is the imple- the required drivers. To accommodate
ers questions to clarify and refine their learning professionals need to partner mentation of the plan. this, fewer resources should be spent
expectations on all four Kirkpatrick with supervisors or frontline leaders Organizations spend more than 72 on Levels 1 and 2 to allow more train-
Table 2 levels, starting with Level 4. This is • at checkpoints during percent of their training evaluation re- ing involvement in Level 3.
a negotiation process in which the training design sources measuring Levels 1 and 2 (ASTD, While monitoring and encouraging
ROE ROI training professional makes sure that • before training to prepare “The Value of Evaluation” report). This on-the-job behavior, training profes-
Proactive, business partnership approach Defensive isolation approach that the expectations are satisfying to the participants shows how much money training pro- sionals should create an “early warning
that unifies teams separates functions stakeholder and realistic to achieve • after the training to support and fessionals are spending on employee detection system” in which preliminary
Defines training as a contributor to key
with the resources available. monitor trainee performance on training. It leaves just 28 percent of the measurements of identified metrics are
Defines training as an end in itself Learning professionals then need to
business results the job. training evaluation budget for demon- taken. Areas that are not hitting targets
convert the typically broad, unquan- When participants return to the job strating how the training supports on- should be identified so that adjust-
Value defined by business stakeholders
Value defined by predetermined formula tified expectations into observable, after training, required drivers—pro- the-job behavior and accomplishment ments to the plan can be made. This
in cooperation with training
measurable Level 4 results by asking cesses and systems that reinforce, mon- of key business results (Levels 3 and 4). keeps major long-term initiatives on
Focus on comprehensive evidence the question, “What will success look itor, encourage, or reward performance Seventy percent of the time, when track and creates the highest likelihood
Focus on single metric of numeric proof
and a compelling story of value
like to you?” These Level 4 results be- of critical behaviors on the job—must participants fail to consistently perform that the desired results will be achieved.
Easy to understand, flexible, come the targets on which collective be in place to provide both support and desired behaviors on the job, it is due to A good way to communicate the
Complex, rigid, and expensive
and cost-effective efforts are focused. accountability for the consistent perfor- factors in the work environment (2006 status of critical behaviors, required

36 | T+D | DECEMBER
64 August 2010
2010 THE BEST OF MEASURING & EVALUATING LEARNING
August 2010 || T+D
T+D || 65
37
 drivers, and leading indicators (short-
term observations and measurements
that suggest that critical behaviors are
on track to create a positive impact on
desired results) is with a dashboard.
A dashboard is a visual depiction of
goals set for each of these items and
the current status of each. Dashboards
are a highly effective driver and one
that executives favor.
A compelling chain of
evidence demonstrates
your bottom line value
Training is on trial, accused of failing to
deliver sufficient business value to the
bottom line. There is an urgent need for
training professionals to demonstrate
the value they do bring to the business
by presenting a chain of evidence at the
end of important initiatives.
This process is easy and fun when
the Kirkpatrick Foundational Princi-
ples are followed because the evidence
to be collected is clearly defined at
the beginning of the project by the
stakeholders, and data is collected all
along the way. It can be very rewarding
to assemble the story of how working
together in business partnership has
created the desired ROE.
The chain of evidence includes
both numeric data and the testimoni-
als and stories of those who partici-
pated in the process. This combination
of evidence shapes a story that appeals
to and is understood by everyone.
Training professionals should prepare
a compelling presentation that will be
delivered in person if possible, to share
this data with stakeholders and the
entire project team.
Telling this story will earn train-
ing professionals a “not guilty” verdict
for the accusation of failing to deliver
adequate value for their efforts. Since
the plan was created and executed in
partnership, the training professional
is then cast in a light of delivering a
positive story of success.
“But we were asked
to show our ROI”
It is common that training profes-
sionals are asked to “show their ROI.”
Return-on-investment has become
shorthand for value much the way
38 | T+D | DECEMBER
66 August 2010
2010
“Kleenex” has become the accepted
name for facial tissue. Be sure to ask
the stakeholders exactly what sort of
measurement of value they require. it is common that
training professionals
Generally what ensues is a conversation
in which the stakeholders say they want
to know what they are getting for the are asked to “show
time, money, and resources invested in their Roi.” Return-
on-investment has
learning and development efforts.
Some well-meaning training profes-
sionals don’t ask questions and just become shorthand for
assume that an ROI study to isolate value much the way
“kleenex” has become
the impact of training is required, and
proceed to spend even more money to
hire someone to deliver it. It is impor- the accepted name for
tant to clarify exactly what ROI is to facial tissue. be sure
to ask the stakeholders
understand how it differs from ROE.
ROI is defined as the ratio of money
gained or lost (whether realized or un- exactly what sort of
realized) on an investment, relative to measurement of value
they require.
the amount of money invested:
Net Program Benefits × 100 = ROI% THE LEAST LIKELY METRIC IS ACTUAL BUSINESS
Program Costs
OUTCOMES, WHICH NEARLY A QUARTER OF
How ROE differs from ROI
Table 2 indicates the key differences RESPONDENTS SAID THEY DO NOT MEASURE AT
between ROE and ROI. Brinkerhoff
comments on the use of ROI in
ALL IN TERMS OF LEARNING EVALUATION. ONLY
measuring the value of training
as follows:
before a member of their own team.
Earn your place on the team by being
22.4 PERCENT SAID THEY USE THIS APPROACH
“To isolate the impact of training is a cooperative member who works TO A HIGH OR VERY HIGH EXTENT.
methodologically impossible and collaboratively towards the over-
strategically counterproductive. If all success of important initiatives.
we’re trying to build partnerships Then watch the confidence in your
with managers of trainees, we can’t work grow and requests to “prove the
use an evaluation plan that iso- value” decrease. T+D
lates the impact of training alone
and ignores the critical manage- James D. Kirkpatrick is a senior consultant
for Kirkpatrick Partners; information@ From the 2010 ASTD Research Study The Value of Evaluation: Making Training
ment support after the training
kirkpatrickpartners.com. Wendy Kayser Evaluations More Effective. Visit www.astd.org/research for more info.
event. It’s like saying I’ve got a new
Kirkpatrick is director of Kirkpatrick
laptop; now let’s see if we can iso- Partners; information@kirkpatrickpart-
late the value of the B key.” ners.com.
Getting Started
Get started on building ROE in your
organization today:
• Identify the key stakeholders that
judge the “success” of training.
• Get to know your stakeholders
and identify their priorities.
• Create a cross-functional advisory
committee with representatives
from business and training.
• Select a strategic initiative, and interesteD in orDerinG e-Prints?
Would a digital version of this article be a great
use the ROE process. fit for your next course, presentation, or event?
The reality is that companies are Are you interested in e-prints of several T+D
articles on a specific topic?
more likely to let a “contractor” go Visit astd.org/tD/eprints for more information.
THE BEST OF MEASURING & EVALUATING LEARNING | T+D | 67