How to measure
training effectiveness
By Roger Grice CEO
Special points of
interest:
Goal-based evaluation
Goal-based evaluation
Kirkpatrick’s four steps Each year globally many mil-
to measuring training lions of dollars are spent on
effectiveness training. It would seem reason-
able to establish just how
Input process and out- effective this training has been
put models if only to help justify the
budget. Evaluation techniques
Flowchart of training are not new, indeed the most
evaluation influential early work on
evaluation was performed in
the USA during the 1940s by
Tyler. Probably the most im-
portant aspect of Ralph W
Tyler’s work was the realiza-
tion of the importance of ob-
jectives in designing an effec-
tive school curriculum. This
Inside this issue: early methodology can be de-
fined as the scientific/
experimental approach. The
model emphasized the impor-
tance of knowing the educa-
Cost Effective 2 tional standards of the individu-
Training als in the control group before
the new education initiative
took place and then determin-
Evidence or 3 ing the change that had taken
proof place by measuring the differ-
ence in attainment after the
educational intervention was
The IPO Model 3 completed. In other words the
IPO Flowchart 3
Donald L Kirkpatrick’s “Four steps to measuring training effectiveness”
This model uses four separate
stages for the evaluation of the
About Sentrico 4 effectiveness of a training pro-
and Roger Grice gram. The four stages are
Reaction
Conclusion 4 the Kirkpatrick level 1
Learning
Behavior
Results
The first stage is about the
Newsletter Date: September 2010
aim was to determine the ef-
fectiveness of the training by
scientific means on the level of
performance of the individual.
Although there have been
some critics of this methodol-
ogy it has proved to be valu-
able in that organizations have
been able to quantify the rela-
tionship between their corpo-
rate objectives (often ex-
pressed as key performance
indicators (KPIs) today) and
the training aims and objec-
tives. The main criticism of the
scientific/experimental ap-
proach is that the methodology
is unable to take into account
the unexpected or uninten-
tional learning that can take
place within a dynamic and
changing learning environment.
Those readers who can re-
member reading about the
original, and now famous Haw-
thorne experiments to deter-
mine the effects of illumination
levels on production held at
the Western Electric plant in
Cicero, Illinois in the 1920’s
and the later experiments also
reaction of the trainee to the
training. This sort of measure-
ment is concerned with how
the trainees “feel” about the
course. The usual course feed-
back sheets are an example of
evaluation. Most organizations
do not do any more than this
type of measurement and
analysis. The drawback is that
we do not really know if the
conducted by the National
Research Council showed that
human behavior is indeed diffi-
cult to quantify exactly. Never-
theless the greater attention
that was given to training
methodology and in particular
the effectiveness of manage-
ment development during the
1960s and 1970s has led to
much better understanding and
the emergence of systems
evaluation methodology. As we
will see this approach can be
divided into two main schools;
the narrow focused approach
being attributed to Donald L
Kirkpatrick in the early 1960s
and the somewhat broader
model advocated by P.Warr et
al (The Evaluation of Manage-
ment Training) known as the
CIRO model. The CIRO model
looked at context evaluation,
input evaluation, reaction
evaluation and outcome evalua-
tion.
trainee has actually learnt any-
thing. What really seems to be
being asked of the trainee was
how “happy” were you with
the course; hence the some-
what derogatory description
that is often applied to the
“happy sheets”!
We will now look at the next
level of evaluation concerned
with Learning.
Page 2
“Four steps to measuring training effectiveness” continued
Level 2.
Things can be improved by
using a pre-test and post-test
and comparing the results.
The model sug- The questions need to be
objective and closely related
gested by to the course objectives
Kirkpatrick is
(more about that later). In this
one of the most way we can determine if the
popular and can training actually delivered
knowledge and this was un-
be a useful ap- derstood by the trainees at
proach to evalu-
the time. An organization that
ating the effec- does this can be confident that
tiveness of train- the trainee has actually learnt
ing. something at that time.
Why do I make the point that
Level 4 Evaluation
“These are Kirkpatrick is now concerned
metrics which with the training to determine
really have an if it has actually been trans-
lated into tangible benefits to
impact on the the organization. Quite simply
“bottom line” has productivity and or quality
and for that been improved? Have the
number of accidents or inci-
reason feature dents been reduced? Has plant
in the availability and or plant utiliza-
companies tion been improved? Has the
morale of the workforce
balance sheets changed for the better? These
and KPIs”. are metrics which really have
an impact on the “bottom
line” and for that reason fea-
Cost effective training
If we all now accept that the
majority of organizations carry
out training to change the
behavior of the workforce and
improve their performance in
the workplace we also must
accept that this cannot be
achieved without a clear plan
Eliminating waste by and strategy. This rational
approach to training and de-
ensuring the training
velopment is best demon-
carried out delivers
strated by the Competence
what is required by
Profile (s) or Competence
the organization is
essential for a sound Map (s) for a particular job (s)
in the organization (s). If the
business case.
approach in constructing these
How to measure training effectiveness
we have to make the meas-
urement and consider the
learning at a certain time?
Well because we do not
know if the learning has had
time to be internalized and
become “concrete”. All too
often trainees (and delegates
at conferences for example)
will have difficulty remember-
ing what was in the course or
seminar they attended a few
days later let alone months
later. There are various tech-
niques that we can employ to
improve the level of recall but
basically “if we don’t use it we
will lose it”!
ture in the companies balance
sheets and KPIs.
We have to ask ourselves is
this not the real reason for
training? Training has to make
a real difference in perform-
ance and effectiveness; this is
tied closely to competence.
Training has be proven to
deliver results and be cost
effective.
It might sound simple but it is
not for most organizations.
has been methodical we will
have at our disposal a data-
base of learning objectives. I
call these TCOs or terminal
competence objectives. These
TCOs must relate to the
needs of the organization and
will contain the
“underpinning” knowledge
required to understand the
tasks or duties the employee
must perform as part of his/
her duties as well as the prac-
tical skills necessary. In this
way we are able to ensure
that the training is appropriate
Level 3
This is concerned with
“behavior”. By that we mean
the measurable change in an
individual as a result of their
attendance on the training
course. This is, in my opinion,
the least we should be expect-
ing from any training program.
After all what is the point of
spending money and using
resources if the training does
not effect some measurable
change in the behavior of the
trainee?
The reason is that most do
not have in place any system
for measuring the improve-
ment in competence of the
individual; let alone a system-
atic approach to identifying
the most effective means of
assisting the individual to be-
coming competent.
to the individual at that time in
their development and truly
meets both their needs and
those of the organization.
If we use these TCOs to de-
velop the training program
(whether it is a classroom
based course or a workshop
activity) we will have a course
“tailored” to the needs of the
organization and furthermore
we will be able to measure the
improvement in performance
and contribution to the team’s
KPIs.
By Roger Grice CEO
Evidence or Proof ?
A goal can be considered to
be something that the organi-
zation strives to achieve
through the meeting of spe-
cific objectives. Achieving the
individual objectives in a me-
thodical and logical way effec-
tively maps out the process
towards the eventual achieve-
ment of the goal.
If we look at an important KPI
for any organization it might
be expressed as “zero lost
time injuries”; in other words
no employee being away from
work for more than 3 days
because of an injury sustained
while in the company or per-
Input, Process and Output
I have presented an argument
for mapping out the compe-
tences required to perform a
job and to linking these com-
petences to the delivery of
training interventions. In effect
this means that the strategic
goals of the organization must
be cascaded down to team
level within the organizations.
In time new competences will
be required for team mem-
bers as the industry “drivers”
like new technology and com-
petition encourages or per-
haps forces the organization
Flow chart of input-output approach to training evaluation
Trainee quali- Value adding activities
fications/
experience/
current com- Plan
petence
Instructor
abilities
Training ma-
terials
Subject matter experts
Resources
Competence Management System Inputs
Expenditure on training
forming work related duties
(like driving a truck for exam-
ple). If a company with over a
1,000 employees for example
achieves a figure of say
6,000,000 man hours without
a LTI then clearly their HSE
systems must be working. This
can be translated into real
company benefits and major
savings to the organization.
There are many tangible bene-
fits and other benefits which
are very important but more
difficult to quantify.
Benefits can include:-
Better productivity
to adopt new work systems
and processes.
Some leading companies rec-
ognized this change model and
one (IBM) introduced the IPO
paradigm. The concern being
is training delivering the de-
sired results within the organi-
zation? The IPO model was
designed to confront and
solve problem.
In this model a distinction is
made between output (short
term benefits) and outcomes
Design Develop Deliver
Coaching /mentoring
Page 3
Lower insurance premiums
Improved morale
Enhanced reputation
Reduced stress
No legal costs
No compensation cases The proof of the effective-
ness of a training interven-
Lower staff turnover tion can be overwhelm-
ingly shown by hard facts
Employer of choice and figures, as well as
other sources of evidence
obtained by staff inter-
views and focus groups.
(benefits which are longer
term but actually determine “ The ultimate
the availability of future train- payoff or
ing resources).
added value of
The following quote is particu- an employee’s
larly relevant here
learning
“ The ultimate payoff or added
experience is
value of an employee’s learn-
ing experience is how well he how well he or
or she performs on the job.” she performs
David S Bushnell writing about on the job.”
the “Input, Process, Output: A
model for Evaluating Training”
1990 Training and Development
Journal
Trainee reac- Profitability
tions improved
Customer
Knowledge and satisfaction
skills gained increased
Productivity
Improved job enhanced There are several factors
performance Reduced acci-
dents and
we have to consider when
incidents planning a training inter-
vention, but the most
important is how it will
make appositive impact on
business performance.
Roger Grice Associates Ltd
Windsor House
107 Talbot Road,
Rhondda Cynon Taff,
CF72AE
United Kingdom.

AboutRoger Grice Associates Ltd and

Sentrico ™ Competency Management System.

Phone: 44(0)1443 228612 Roger Grice is the CEO of a Consultancy that spe-
E-mail: cialises in business solutions for medium and large
post@rogergrice.co.uk enterprises, particularly in the energy, oil and gas
sector. With our team of highly qualified and experi-
enced consultants we are able to assist our clients in
organizational development and business improve-
ment through the development of their own people.
We believe passionately in what we do and strive to
provide a service which is truly “world class”. Major
projects that we have been involved in have received
international recognition for their novel approach
and the sustainable development of the local work-
Please visit our website force to competence standards recognised through-
www.rogergrice.co.uk out the world.

Roger has over twenty years experience in the oil and gas
business and is a “world class” training expert with an MSc Eur Ing Roger Grice MSc BA
in Training and Performance Management . (Hons) C.Eng. MIEE. FCIPD.

Conclusion
We have seen that evaluation
of training interventions such
as conventional courses are
generally a neglected area as
regards evaluation of their
effectiveness. In general the
“happy sheet” type of evalua-
tion is ineffective as it is in-
clined to be subjective, and
influenced by considerations
which are actually outside the
course itself. That is the
course content could be poor
or inappropriate but was de-
livered by a good instructor
who managed the group well
and they actually “enjoyed”
the experience but learnt
little!
Instead I have suggested that
we owe it to the organization
to identify the competences
that are really required by the
individual to do his or her job
well. These competences are
the skills and underpinning
knowledge which should be
part of the company database
and accessible through its
Competence Management
System.
The training intervention has
to be designed to meet the
We should avoid training
evaluation which is only based
on subjective and often flawed
data. Instead we should be using
metrics which enable us to
measure improvements in per-
formance that improve the
organizations competitiveness
and profitability.
needs of the organization by
satisfying the competence
development requirements of
the individual. By achieving this
goal we have a rational and
justifiable case for training and
we are able to prove that
training really does deliver
tangible benefits, not least to
the “bottom line”.
In this global marketplace it is
essential to maintain a com-
petitive advantage through the
workforce; this is best
achieved by an effective Per-
formance Management system
underpinned by a robust
Competence Management
System such as Sentrico™.