WHITE PAPER
MEASURING THE CUSTOMER-DRIVEN APPROACH
THE IMPACT OF
TRAINING AND
DEVELOPMENT
A WHITE PAPER BY In the world of training and development,
P AU L B E R N TH AL , measuring program impact is a frequent
PH.D. discussion topic. Everyone seems to be
M AN AG E R , C AB E R , D D I
searching for the perfect measurement system
to increase organizational capabilities and link
them with the business strategy. Human
resource practitioners, OD consultants, training
managers, and senior managers realize that any
training and development activities should
eventually show a positive return and improve
the bottom line. Otherwise, why even bother to
have a training and development process?
REASONS FOR MEASURING TRAINING
AND DEVELOPMENT ACTIVITIES
Why should an organization measure the
progress of its training and development
curricula? Some rationale includes:
• To justify the financial investment in the
training and development programs.
WHITE PAPER— MEASURING THE IMPACT OF
• To gather feedback for ongoing improvement
as a program is being delivered.
• To demonstrate the link between HR
TRAINING AND DEVELOPMENT
programs and the company’s strategy.
• To compare the effectiveness of two or more
training programs.
• To meet requirements set by professional
organizations or government regulations.
1
Because there are so many reasons for
measuring program impact, it’s important to
consider what an evaluation is supposed to
accomplish. The first step in doing so requires
you to identify the real customers or
stakeholders of the program. The customers
include the program participants, their
managers, the HR staff, and senior leaders.
Each of these groups probably has different
questions or expectations about the purpose of
the program. Once you have identified these
customers, it’s a good idea to brainstorm a list
of questions that each group might have about
the program. For example:
• Participants: “Is this going to improve
my skills?”
• Managers: “Will this program make my
employee more productive?”
• HR staff: “Did this program address the
right skill deficits?”
• Senior leaders: “Will this program provide a
positive return on our investment?”
The questions posed by the customers will drive
the types of measures that you select in the
evaluation process. There is no point in
spending time trying to answer questions that
no one has asked. Additionally, you need to
make sure that you answer the questions posed
by your most critical customers.

LEVELS OF EVALUATION
No discussion of training evaluation would be
complete without addressing Donald L.
Kirkpatrick’s four levels of evaluation. This
typology, developed in 1959, organizes the
many questions customers have into four
primary categories. Since its development,
significant progress has been made to better
understand evaluation. For instance, others
have built upon Kirkpatrick’s model by adding a
fifth level that focuses specifically on return on
investment (ROI). Also, recent research by
Kurt Kraiger and his colleagues (1993) has
greatly enhanced the understanding of the
relationships between the different outcomes.
The five levels of evaluation are described as
follows.
Level 1: Reactions and Planned Action
Question: How did participants respond to the
training?
Reaction measures (also known as “smile
sheets”) are the most commonly used form of
evaluation.
Reaction measures assess how the participants
responded to the training class, event, or
materials. Warr and Bunce (1995) describe
three kinds of reactions that are measured:
• Enjoyment of training (emotional
reaction)—“I found this training program to
be enjoyable.”
WHITE PAPER— MEASURING THE IMPACT OF
• Usefulness of training (perceived
value)—“What level of value does the
training content have for your job?”
TRAINING AND DEVELOPMENT
• Difficulty of training—“I found the
issues taught in the training to be difficult to
understand.”
The typical measurement instruments used for
this level are participant questionnaires and
2
© Development Dimensions International, Inc., MMIII. All rights reserved.
verbal feedback. This information is usually
collected immediately after the training has been
completed. Reaction measures are usually fairly
brief, but they can go into great depth about the
enjoyment, usefulness, or difficulty of the
training that participants received.
Level 2: Learning
Question: To what extent did the participants experience
changes in attitudes, skills, or motivations as a result of
the training?
Learning can be defined and assessed in many
ways. For example, participants’ ability to answer
questions about the training content or to
demonstrate newly acquired skills can be
measured. Kurt Kraiger and his colleagues
(1993) identified three types of learning that can
result from training:
• Cognitive Outcomes—These are usually
assessed by multiple-choice questions, open-
ended responses, a listing of facts, or similar
methods. Knowledge checks such as these are
similar to tests used in schools to assign
grades. Measures of cognitive outcomes can
be assessed immediately after the training or
later to assess knowledge retention over time.
• Skill-Based Outcomes—These outcomes
are typically measured by requiring participants
to demonstrate their new skills in the training
environment. Skill-based outcomes are not
the same as behavior-change outcomes (Level
3) that occur in the work environment.
Learning outcomes that focus on skills
measure only participants’ ability to
demonstrate the skills.
• Attitudinal Outcomes—These measures
focus on how participants feel or think about
the training content. They have implications
for participants’ motivation to use the training,
their confidence in using the skills, and their
ability to reach goals.
 To determine if learning has occurred,
participants’ knowledge and skill levels, pre-
and post-training, can be compared to
measure change. Having a control group also
helps evaluators to draw conclusions by
comparing the performance of trained and
untrained individuals.
Level 3: Behavior
Question: Can behavior change be observed on the job
as a result of the training (i.e., training transfer)?
Level 3 is all about how participants use their
skills or apply their new knowledge in the
workplace. In most cases, training success is
defined in terms of behavior change. However,
Level 3 focuses on what happens to training
participants after they leave the training
environment. At this point, the focus is on
training effectiveness, not evaluating the
training. Measures used at Levels 1 and 2 are
associated only with the influence of the
training; if results are positive or negative, the
training can usually be pointed to as the primary
cause. Levels 3, 4, and 5 include the influences
of many factors besides the content of the
training or its delivery.
Measuring behavior change usually requires
some type of pre-/post-training assessment.
Often, this type of assessment comes in the
form of surveys that measure estimates of
behavior change. Performance management
systems also have been used to track changes in
performance before and after the training.
WHITE PAPER— MEASURING THE IMPACT OF
However, this approach requires a carefully
structured performance management system to
keep accurate records of participants’
TRAINING AND DEVELOPMENT
accomplishments. Other methods of
measuring behavior change exist, but they often
are not practical. For example, training
3
© Development Dimensions International, Inc., MMIII. All rights reserved.
participants could be observed on the job while
their performance is monitored with checklists.
Or the number of errors they make when using
the skills taught in the training could be tracked.
Survey assessments can be administered easily
and can provide a quick read on skill application.
Ideally, a control group should be used to add
validity to the findings.
Level 4: Results
Question: How have organizational outcomes changed as
a result of the training program?
By far, Level 4 is the most difficult training
outcome to measure accurately. “Results” can
include almost any criterion by which
organizational success is defined. These
measurements are linked to the organization’s
business case, critical success factors, or strategic
objectives. Examples of results might include:
• Productivity.
• Customer satisfaction.
• Efficiency (e.g., scrap).
• Morale.
• Profitability.
These outcomes are usually tracked over time
(e.g., month to month) or measured before and
after the training. Changes in results might
appear in the form of significant deviations in
long-term trends or sudden jumps in monthly
measures. Unfortunately, it is not always clear
how long it will take for a training intervention
to have an impact on organizational results.
Changes could occur immediately or appear
years later. As in any study, measuring parallel
results from a control group adds validity to the
conclusions.
 Although business results are the most removed
from training, they often receive the most
attention. Occasionally, many factors interfere
with the influence of the training, so much so
that the training appears to have no real impact
on the bottom line. Numerous researchers
have pointed out that evaluating training at
Level 4 is inappropriate. The uncontrolled,
non-laboratory setting of an organization
makes isolating the impact of any one program
almost impossible.
In Level 4 the assumption is made that if an
organization trains and develops people, it
should realize positive bottom-line outcomes.
But measuring the effectiveness of training and
development curricula at this level can be an
expensive, time-consuming drain of resources.
Before undertaking this level of measurement,
an organization must carefully consider whether
doing so would be cost effective and warranted.
Level 5: Return on Investment
Question: Did the benefits of training outweigh
the costs?
Because it focuses on a calculated percentage
return, Level 5 is distinct from the more generic
Level 4. It measures the cost of the training
intervention versus the return on investment.
Usually, gains observed from the training
intervention (such as changes in results or
behavior) are converted into monetary values.
These returns are then compared to the per-
person cost of the training. Many factors are
WHITE PAPER— MEASURING THE IMPACT OF
used to calculate ROI. Entire books have been
written to explain this process, which also is
known as “utility analysis.” For example, ROI
TRAINING AND DEVELOPMENT
must account for the fact that money spent on
training could have been invested in other
company ventures; therefore, training not only
must return its costs, but also exceed the
potential value of alternative investments.
4
© Development Dimensions International, Inc., MMIII. All rights reserved.
Although most OD managers or training and
development managers would love to calculate
them, these measures can be difficult to obtain
and explain. Attempting to show cause-and-
effect relationships between training and ROI
takes expertise and patience. In the end,
HR practitioners should carefully weigh the
options and determine if ROI assessment is the
right choice.
MEASUREMENT CRITERIA
The following are some criteria to consider when
identifying the measures of effectiveness for
training and development curricula:
• Ensure that the measurement systems are easy
to understand by everyone involved.
• Decide what you are going to do with the data
you gather. Are you going to distribute it? Is
it going to be part of a presentation?
• Manage your internal customers’ expectations.
Make sure that they are comfortable with the
type of data you are collecting and that they
will have confidence in the results.
• Before deciding on which measurements to
use, identify the business needs. What is the
root problem that is driving the training?
• Identify the objectives of the training
curriculum for the short term and the long
term. Where should you see immediate versus
long-term effects?
• Candidly discuss any barriers, drivers, or
constraints to implementing the training
and development curriculum (e.g., resources,
level of investment, other concurrent
initiatives).
• Define the audience levels of employees who
will benefit from the training (e.g., number of
employees, location, shift). How will the
training affect these breakouts?
 • Link the business strategy and cultural
strategy to the training and development
curriculum. How does the training support
the strategy?
• Allow the measurement systems to be
influenced by those making the decisions.
If you don’t involve your primary
customers in the design of the
measurements, they will be less likely to
accept the results.
• Decide whether your measures will be “one
shot” or long term. Are you going to
evaluate the program once, using a single
group of employees, or do you plan to
collect data on a regular basis (e.g.,
monthly)?
• Identify a minimum acceptable level of
effectiveness for your training and
development curriculum. What kinds of
results do you expect? What will satisfy
you and your internal customers?
• Use any measurements of training and
development activities that can be
converted to dollars—the most effective
way to measure ROI.
WHITE PAPER— MEASURING THE IMPACT OF
TRAINING AND DEVELOPMENT
5
© Development Dimensions International, Inc., MMIII. All rights reserved.
ADDITIONAL MATERIALS
Barron, T. (1997, January). Is there an ROI in
ROI? Technical & Skills Training, 8, 21–26.
Kirkpatrick, D.L. (1998). Evaluating training
programs: The four levels. San Francisco:
Berrett-Koehler.
Kraiger, K., Ford, J.K., & Salas, E. (1993).
Application of cognitive, skill-based, and
affective theories of learning outcomes to
new methods of training evaluation. Journal
of Applied Psychology, 78(2), 311–28.
Parry, S.B. (1996, May). Measuring training’s
ROI. Training & Development, 50, 72–77.
Phillips, J.J. (1994). In action: Measuring return on
investment. Alexandria, VA: American
Society for Training & Development.
Phillips, J.J. (1996, March). Was it the training?
Training & Development, 28–32.
Phillips, J.J. (1996, February). ROI: The search
for best practices. Training & Development,
42–47.
Warr, P., & Bunce, D. (1995). Trainee
characteristics and the outcomes of open
learning. Personnel Psychology, 48, 347–375.
Many resources for evaluation methods also are
available on the web site for the American
Society for Training & Development
(http://www.astd.org).
.
 Level of
Measurement Definition Examples of Measurement

Level 1 How did participants • Post-session participant questionnaire.

Reactions and respond to the
• Verbal feedback.
Planned Action training?
• Observation of trainees.

Level 2 To what extent did • Pre-test/Post-test change scores on

Learning the participants measures of skill or knowledge.
experience changes in
• Retrospective assessment of skills
attitudes, skills, or
before training versus skills after
motivations as a result
training.
CONTACT INFORMATION of the training?
• Performance tests or simulations to
DDI HR BENCHMARK GROUP
PAUL BERNTHAL, PH.D. gauge learning.
DEVELOPMENT DIMENSIONS
INTERNATIONAL
1225 WASHINGTON PIKE Level 3 Can behavior change • Pre-test/Post-test measures of
BRIDGEVILLE, PA 15017
PHONE 412.257.7533 Behavior (on-the- be observed on the behavior (assessments, 360° feedback,
FAX 412.257.3093
E-MAIL HRBENCH@DDIWORLD.COM job application) job as a result of the surveys, interviews, etc.). Might
DEVELOPMENT DIMENSIONS
training (i.e., training involve a control group.
INTERNATIONAL transfer)?
PHONE 412-257-0600 • Retrospective assessment of skills
E-MAIL INFO@DDIWORLD.COM
WEB SITE WWW.DDIWORLD.COM before training versus skills after
TO ORDER A COMPLETE REPORT,
training (e.g., “How was I before
CALL: DDI BOOKS AND versus after?”).
PUBLICATIONS AT 1.800.944.7782.
ASK FOR PART NUMBER
MKTCPWP07-R.
Level 4 How have • Changes over time against baselines.
Results organizational
• Estimates of bottom-line impact
outcomes changed as
provided by managers or other
a result of the
observers.
training program?

Level 5 Did the benefits of • Cost of training and development

Return on training outweigh the versus ROI.
Investment costs?
WHITE PAPER— MEASURING THE IMPACT OF
TRAINING AND DEVELOPMENT

6
MKTCPWP07-R © Development Dimensions International, Inc., MMIII. All rights reserved.