TALENTACQUISITIONANDDEVELOPMENT

The Real ROI of Leadership Development:

Comparing Classroom vs. Online vs. Blended Delivery

The Real ROI of Leadership Development:

Comparing Classroom vs. Online vs. Blended Delivery
Contents
1

Introduction

Using A Truly Blended Approach

The Evaluation Process

The Kirkpatrick/Phillips Five Levels of Learning

Results: Comparing approaches at each level of evaluation

Level 1 and Level 2

Level 1-6 American Express Measurement Process

Level
Level
Level
Level

3
4
5 ROI
6 - Transfer Climate

10 Conclusion

TALENTACQUISITIONANDDEVELOPMENT

The Real ROI of Leadership Development:

Comparing Classroom vs. Online vs. Blended Delivery

Introduction
With its primary focus on business results and return on investment, American Express once again takes center stage
and pioneers the research on how to most effectively deliver
best-in-class training for its global population of people
leaders. While companies all over the globe are scrambling
to offer their employees a variety of training media, there
exists little, if any, research to support their ultimate investment decisions. American Express, a $64 billion, global
organization with more than 65,000 employees in 45 countries, has compared and analyzed three delivery options for
its leadership training program - online, instructor-led, and
a blended approach, using the results to inform its training
strategies and maximize return on investment (ROI).
Despite the growing pressure on training organizations to
deliver tangible and measurable results, a vast majority of
companies cannot quantify the behavior change, business
impact, and ultimate ROI of their initiatives. Add to this, the
challenge of creating an ROI for a Leadership program and
results get even more elusive. This study not only measures
ROI results for a leadership development program, but also
uses these findings to compare ROI across various delivery
modes. Further, a new measure is introduced, where the
leading indicators of impact - those things that help or hinder learners from applying their new knowledge and skills
- are identified and discussed.
As an organization in the service industry, American Express
realized as part of its overall talent strategy, it needed to
inspire its employees to be even more engaged and motivated, as it is their people who ultimately create the American Express customer experience that is unparalleled by
competitors. The company determined it could achieve this
by increasing leadership capability - by equipping its people
leaders with the knowledge and skills to truly engage all
employees.

In 2003, American Express launched a leadership program

called Situational Leadership II (SLII) and delivered it
to its 20,000+ population of People Leaders. The specific
leadership program was chosen because of its ability to
target specific leadership competencies and tasks, while
remaining flexible enough to translate across an incredibly diverse leader population. It provides a model whereby
leaders partner with direct reports to diagnose and develop
unique action plans.
Over the next three years, American Express managers
and executives across its many lines of business and staff
groups were offered the leadership training via three primary delivery modes:
In the first group, employees were trained using web-based,
online only delivery. They were able to access the SLII
courseware through a learning management system for selfdirected learning with little or no supporting sustainability
tools built into the program.
In the second group, learners attended a one- or two-day,
instructor-led classroom training event with no online or
other formal supporting events. Here, instructors were either provided externally by the vendor or trained internally to
deliver the course content. Supporting sustainability tools
were available but not consistently applied.
The third group of learners was offered a blended delivery
approach designed by Ninth House, which included an
executive sponsor communication, an in-person or virtual
classroom kick-off event, a self directed online course, a
wrap-up event that discussed the focus of the program, and
a sustainability tool to facilitate one-on-one follow-up conversations with their leaders.

Using a Truly Blended Approach

A truly blended approach to their leadership training has been adopted by American Express. The methodology included
a classroom type kickoff event led by an American Express business leader who discussed the focus of the program, the
learning process and expectations, as well as how learners may apply the learning to their business. Participants then
took the self directed online portion of the course and six weeks after completion attended a wrap up event that was a
virtual classroom experience. During this event participants were encouraged to talk about what they had learned with
their teams and their managers, how best to apply the learning and were given an internally developed sustainability tool
to facilitate one on one follow-up conversations with their leaders.
This particular initiative which was among the first weve done using the blended learning approach has set the stage
for how all leadership development training should be done. Its absolutely critical for the participants to have the tools
they need to take what theyve learned and use it more each day - Ariane Grunbaum, Leadership Development Analyst,
American Express

The Evaluation Process

Over 2000 managers and directors development efforts
were evaluated. The participants came from all American
Express business units and staff groups in all of their four
global regions. The evaluation and overall impact of the
leadership development efforts were measured in part on
the traditional Kirkpatrick/Phillips 5-level approach (Table
1 below). This was augmented with an additional critical
measure - Level 6 which refers to the transfer climate.
Developed by Dr. Paul Leone, Talent Acquisition and Development at American Express, Level 6 specifically identifies
the factors in the participants immediate work environment
that predict the impact of the program, looking at what

might inhibit or help the transfer of learning. Dr. Leones

research has shown that these climate factors can quite
literally make or break a companys training investments.
Through this new level of analysis, American Express was
able to identify the leading indicators of program impact.
The challenge was to identify the factors that made the SLII
training extremely effective for some groups, while others
experienced far less behavior change and business impact.
Identifying and understanding these crucial factors helped
American Express ensure greater impact and ROI as rollout
and implementation continued across the company.

The Kirkpatrick/Phillips Five Levels of Learning,

Plus the crucial sixth level developed by Dr. Paul Leone, American Express

Level 1: Reaction
Level 2: Knowledge
Level 3: Behavior

Level 4: Impact
2

Level 5: ROI
Level 6: Transfer
Climate

Method of Evaluation
Learner responds to survey upon completion of learning event
(satisfaction with experience)
Learner responds to survey upon completion of learning event
(new knowledge and skills acquired)
Assessment completed by manager, learner, and direct reports
three months after learning event
(observed improvement in leadership skills)
Assessment completed by manager, learner, and direct reports
three months after learning event
(improved productivity of direct reports)
Cost vs. Benefit analysis based on increase in sales productivity
of direct reports over three months
Assesses factors in learners work environment (climate) that will
help or hinder the transfer of learning

Table 1

Measurement

The Evaluation Process

Employees participating in the leadership program were

given a survey immediately after training. This survey captured satisfaction with the experience and the level of new
knowledge and skills acquired (Levels 1 and 2). Three
months after the training was completed, participants were
invited to take another survey online and were instructed to
invite their leaders and direct reports to complete a similar
survey. This survey captured behavior change and application (Level 3), as well as increases in productivity and per-

formance (Level 4). A return on investment (ROI) analysis

(Level 5) was then conducted for all participants reporting sales revenue as their primary performance measure.
Included in self-assessment across the organization was a
transfer climate index which asked participants about the
work environment they returned to post-training (Level 6).
See the summary of the measurement and data gathering
process below.

Level 1-6 American Express Measurement Process

2
3

Participants completed
learning experience*:
Online (n=105)
Instructor-led (n=1,479)
Blended (n=550)

Level 1-2 surveys

Learner satisfaction
Knowledge transfer

*n = employees who participated in the study

3
Months

Self assessment (n=2134)

Improvement in competencies
Assess PMP practices
Assess transfer climate factors
Open-ended comments

Direct report assessment (n=2056)

Improvement in competencies
Assess PMP practices
Assess improvements in
specific leadership behaviors
Open-ended comments
Measure changes to productivity

Leader assessment (n=486)

Improvement in competencies

7 Level 5

ROI study

Table 2

Level 3 & 4

Core leadership program SLII

selected for (a) flexible delivery;
(b) address 3 core leadership
competencies:
Drives results
Builds diverse talent
Communicates effectively

RESULTS: Comparing approaches at each level of evaluation

Level 1 and Level 2
5

4.34

4.55

4.47

4.50

4.54

4.53

Level 1 - Satisfaction
Online

Level 2 - Learning

Instructor-led

Blended

Table 3

All three modes of delivery receive

similar high ratings as to how well
the program was received (Level 1)
and the amount of learning that took
place (Level 2); however most organizations never look past these Level
1 or 2 measures, often using these
rather limited metrics to extrapolate
the ultimate value and potential ROI
from their programs. The following
sections show how American Express
looked beyond these very first levels
of data which are not necessarily
predictive and do not accurately discriminate between how the various
modes of delivery ultimately impact
the participant and their business.

Level 3
Level 3 - Behavior Change
% or participants showing High Improvement in leadership competencies
100%
90%

Worse None Little Some Significant

Exceptional

80%

High Improvement

70%
60%

51%

50%
40%
30%

42%
32%

20%
10%
0%

Level 3 - Behavior Change

Online

Instructor-led

Blended

Table 4

When the question turns to behavior change (Level 3) measurement,

the research first looked at the percentage of participants who were
demonstrating high performance,
meaning the individual, direct reports and leaders were all rating
the improvement of this participant
either significant or exceptional
on the survey scale. The graph below
indicates that for the self-directed
web-based program, 32% of participants showed high improvement,
42% showed high improvement for
the instructor-led program, and for
the blended approach it was 51%.
These findings clearly indicate distinct differences in impact with the
blended approach proving more effective.

RESULTS:

Comparing approcahes at each level of evaluation

Performance measures varied across lines of business. Primary

productivity measures of direct reports included cycle time, conversion rates, fund asset, revenue impact, forecast accuracy, sales,
TBASS (customer sat), AHT (handle time).

Level 4

Beyond quantitative metrics and reporting, qualitative data was

also gathered to give the most accurate measure of Level 4 Business Impact. Qualitative data included success stories, which
looked at personal definitions of how improvement was defined
and how new leader behaviors were impacting discretionary effort
and increases in productivity. The main themes from this data are
outlined in the chart below, together with some insightful quotes
from participants direct reports about how their productivity was
affected by their leaders.

Here American Express study separated its entire sample of direct reports into two distinct groups: Group A
direct reports of high improvement leaders vs. Group
B - direct reports of no improvement leaders. When
looking at the extent to which each of these two groups
of direct reports improved on their primary measures
of performance, some quite astounding differences
emerged. The direct reports of high improvement
leaders increased productivity by an average of 42%,
while the direct reports of no improvement leaders
showed increases of only 16%. (see graph below).

Give More Coaching/

Constructive
Feedback

More Often/Open
Recognition of
Accomplishments

The third group of learners was offered a blended

delivery approach designed by Ninth House, which
included an executive sponsor communication, an inperson or virtual classroom kick-off event, a self directed online course, a wrap-up event that discussed
the focus of the program, and a sustainability tool to
facilitate one-on-one follow-up conversations with their
leaders.

Increased/
Maintained 1:1
meeting to
follow-up

Productivity Increase of Direct Reports

I now do write ups on people who perform very

good within a 3 month period so that they will
be recognized and rewarded at our Quarterly
meeting. It really builds morale and increases
productivity

42%

40%

Better at Assessing
Strengths and
Weaknesses 7%

Direct Reports of
Direct Reports of
High Improvement No Improvement
Leaders
Leaders

Table 5

16%

21%

We saw increases by applying new

leadership competencies in our daily
efforts and having increased 1:1
discussions. Following up on training
to point out specific behaviors that
15% reflect high performance and
opportunities to improve really made a
difference.

More Concentration
on Development/
Career Planning 9%

80%

0%

30%

Setting Clearer/
more aligned
goals 10%

100%

20%

One of my reps showed little interest on one

product he had to sell so I really coached him
and gave him more regular feedback: He
jumped from the last position in the team
straight to the top 5.

Since the training I have been able to

develop a tool that helps my team set and
track goals and accomplishments. We
utilize this tracking tool during team
meetings and share best practices. This
allows for the team and myself to
understand our shared progress.
I now make time to talk about
development plans no matter what other
subjects need to be discussed! Sales
performance really depended on these
development discussions!
I now better steer my Sales people to
get the best out of them I recognize
what they can do well and where they are
lacking. This ensures poor quality output
is prevented and performance is
optimized.

Percentages listed above represent how many participants expressed something similar

Table 6

American Express next used Level 4 Business Impact,

to assess how high improvements in certain key leader
behaviors actually impact the business. To do this,
they focused on understanding how improving leaders
behavior actually affects their direct reports levels of
productivity.

60%

American Express Level 4

Measures of Productivity

RESULTS:

Comparing approaches at each level of evaluation

Level 4 (Cont.)
gram (expressed as another percentage). That is, on the
direct report survey, it was asked how much of their productivity increases could be attributable to improved leadership (e.g. more direction and support from their leaders)
and on the participant survey it was asked how much of
the improved leadership could be attributable to the training experience. These two percentages are multiplied for
the Level 4 calculation total productivity improvement
directly attributable to the training. When looking at these
products across delivery modes, it was again found that
there were clear differences in impact between the three
delivery options. See the level 4 calculation and the results
of each delivery mode below:

Level 4 Calculation:
%DR productivity
lifted by leadership

% leadership lifted
by SLII (Self Survey)

Total productivity
increase attributable
to training

=
12%

10%

10%

5%

0%

Level 4 - Business Impact

Online

Instructor-led

Blended

From these results we see that for the blended solution, 12% of the direct reports productivity improvement is a direct outcome of his/her leader attending the leadership training. This
means that a manager can improve his/her direct reports performance on a given metric
by 12% should he/she demonstrate an improved leadership style. American Express then
decided to examine what a 5, 10, or 15% increase in productivity ultimately means to the
corporate bottom line.

Table 7

While the Level 4 data makes an incredible case for how the
behavior of a leader can impact productivity, the next step
was to isolate just how much of this difference in productivity can be directly attributed to the training, and once again,
compare this impact across delivery modes (classroom vs.
online vs. blended). After all, there may be a multitude of
factors that can either make an employee perform better
or make their leader improve their leadership skills over
a three-month period. To isolate the unique influence of
the leadership program, American Express factored in how
much each direct report was influenced by their leaders
specific changes (expressed as a percentage) and how
much each leaders changes were influenced by the pro-

Level 5 - ROI

To do this part of the Level 5 analysis, attention turned to

Sales Specialists across the organization where their primary metric was new sales contributions/revenue generated.
While the American Express research was based on actual
and confidential operational data, this illustrative example
is based on representative data not actual measures. For
example, assume on average, a sales specialist contributes
$50,000 in revenue over a three-month period. If this employees contribution was to increase by 10% due to better
direction and support from his/her leader, the added benefit
to the company would equal approximately $5000. Now
consider that each leader in the study impacted not one
employee, but at least five. Assuming these new leadership

behaviors are affecting all of a leaders direct reports, that

brings the benefit to the company up to $28,000. If the
benefit is $28,000 in additional revenue, and the cost of
the training was approximately $2,600 and the return on
investment is approximately 1000%.
Using this calculation to compare ROI across programs,
the research quickly shows that although the impact of the
instructor-led training is close to that of the blended training approach (10% compared to 12%), the cost of delivering each of them is dramatically different. With the cost of
training for the blended approach being almost half that of
the instructor-led program, the ROI for the blended becomes
almost 75% greater!
As depicted in the table below, American Express found that
the blended approach, offering significantly higher impact
at a much lower cost, was resulting in an ROI almost 75%
higher then that of instructor-led or strictly online training
approaches:

Instructorled Training

Online
Learning

Blended
Learning

Individual Sales Specialist

- Estimate Sales Contribution (3 months)

$50,000

$50,000

$50,000

Increase in Productivity

10%

5%

12%

Impact on Individual Contribution

$5,000

$2,500

$6,000

# of Sales Specialists per Leader

5.6

5.6

5.6

Total Impact on Contribution

$28,000

$14,000

$33,600

Cost of Training per Leader*

$2,611*

$1,368

$1,766

Return on Investment

972%

923%

1,599%

Further, it could be argued that

the blended formats 75% more
effective results are actually understated. Estimates show that
the results are in fact over 100%,
or double, due to the fact that for
the Level 5 analysis, costs were
based on only one-day instructorled classes. A large, but unverifiable portion of the training was
delivered via two-day training programs that are considerably more
expensive and would substantially
alter the results.

Table 8

The Level 5 findings clearly show where the research became

business-critical. This section outlines how American Express has turned the performance increases found in Level
4 into real dollars, using the numbers in an ROI calculation.
And the impact of the training becomes a jaw-dropping benefit for the organization.

Level 6 - Transfer Climate

The huge difference in the Level 5 data led the research
team to wonder what it was about the blended approach
that made it more effective. The simple answer to this question is Transfer Climate. That is, those factors in the participants immediate work environment that were predictive
of how much training would be applied and sustained on
the job.
To measure this, American Express included a transfer climate index within the post-program impact survey. Dr.
Leone notes, When we looked at the work environments
that these learners were going back to post-training which
was our Level 6 we found three factors were consistently
predicting whether a learner would have no improvement
or high improvement. These factors were:
1. Did the learner have one-on-one meetings with his/her
immediate manager to discuss how to apply the training in
his/her specific role?
2. Did the learner perceive his/her manager endorsed and
supported this specific training?
3.Did the learner expect to be recognized or rewarded for
the training-related behavior change?

What the analysis ultimately displayed was that learners

having significant and sustained impact from their training (as opposed to learners showing no improvement) were
clearly taking their learning back to a different work environment. Even more importantly, these three factors are
all very directly influenced by the participants immediate
managers. This proved one of the major hypotheses an
immediate leader has the potential to either make or break
any training effort.
The illustration below depicts just how much these factors
could discriminate between levels of impact. For example,
72% of learners in the high improvement group were
having one-on-one conversations about how to apply their
learnings, while only 17% of the no improvement group
were having these conversations. This shows all 3 factors
were present and pervasive for the high improvement
group, while they were all significantly lower for the no
improvement group.

Transfer Climate Factors

86%

82%
72%

46%
17%

Had conversation with

leader about how to apply?

Has a leader who

supports training?

38%

Expects to be
rewarded?
No Improvement Group
High Improvement Group

Table 9

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

RESULTS:

Comparing approaches at each level of evaluation

Based on this research, American Express established that

the key criteria for a high transfer climate includes having a
manager who clearly communicates endorsement and support for the training and sets goals and expectations before
the employee initiates the learning event; who follows up
with the participant after the event to discuss what was
learned; and who recognizes and rewards improved leadership behavior.
So to answer American Express original Level 6 question:
the real reason why the blended solution was more effective is because it, inherently, as part of the design, fosters
an environment of manager involvement and accountability.
With intentional components like leader-led kick-offs and
wrap-ups, a high degree of overt personal investment and
implied support for the training by the learners immediate managers, and consistent follow-up conversations built
right into to the blended approach, its not surprising that
this group of learners were more effectively transferring
the learning back to their jobs. In fact, the climate was so

important to the impact of the training that when the climate was held constant across all delivery modes (where
online, instructor-led, and blended training participants had
equally supportive and involved immediate managers) there
were no significant differences in impact.
Leone explains, If you have a high climate, differences
created by the mode of delivery disappear and you will
get similarly effective results. The climate is really what
is predicting the impact and not necessarily the delivery
mode. The blended approach had more positive results
here because many of the sustainability tools and conversation starters are built right into the training design. This
led to the high climate, which led to higher improvement in
the leadership competencies, greater impact on the business, and finally greater ROI. The overall message is that
although any delivery mode can be highly effective, in reality the blended solution does more to ensure the climate
is there and maximized. The illustration below summarizes
these important relationships:

Predictive Effect of a Strong Transfer Climate on Leadership Development ROIs

Higher
transfer
climate

Higher
improvement in
leadership
competencies

Greater
business
impact

Greater ROI

Manager clearly communicates endorsement and support for the training set goals and expectations
before learner initiates learning event
Manager follows up with participant after the event to discuss what was learned and how to apply
Manager recognizes and rewards improved leadership behavior

Table 10

Blended =
High touch
communication,
business leader
support, kick-off,
follow-up
sessions

Conclusion
In 2003, American Express launched a global leadership
development effort in three formats. At the same time, the
Talent Acquisition and Development group also launched a
comprehensive evaluation effort to determine the sustained
effectiveness of the development efforts.
10

American Express evaluated the three training delivery options: basic online courses; instructor-lead classroom training events; and a blended learning solution combining both
online and classroom training, including executive sponsorship communications, kick-off and wrap-up events and supportive manager discussions throughout the courses.
In evaluating each method, American Express used the traditional Kirkpatrick Level 1-4 and Philips Level 5 measures
and found the following results:
The leadership training received similar positive satisfaction (L1) and knowledge (L2) scores (88-90%), regardless of how it was delivered.
The training resulted consistently in high improvement
in on-the-job performance (L3) (32% online; 42% classroom; 51% blended).
The direct reports of high improvement leaders reported from 5% to 12% net improvement in productivity
(L4) attributed to the training itself.
Due to cost savings in the American Express approach
to blended learning delivery, ROI (L5) for blended was
75%-100% greater than either online or classroom
alone.

They found that the key measurement was Return on Investment (ROI) and then American Expresss own, crucial Level
6 measure, Transfer Climate.
The Transfer Climate measure, developed by American Express researcher Paul Leone, is the extent to which factors
in a participants immediate work environment were either
helping or hindering the transfer of the learning back to
the job. This additional level of analysis not only lets the
organization know why training is far more effective for
some groups over others, but also informs business groups
what they can do to maximize the effects of their employees
training in the future.
In this case, the new measure Level 6 Transfer Climate,
indicated that the blended learning approach, which created and fostered a climate of high, immediate manager
support, offered significantly higher impact at a much lower
cost. This resulted in an ROI almost 75% greater than that
of the other two delivery modes, or nearly double if the full
costs of the two-day instructor led options are considered.

Conclusion

These findings clearly indicate that the true impact of a

training program will best be predicted by the work climate
each participant returns to after the event. More specifically, this refers to the type of leader they work with and report to after their respective training. It also deals with the
relationship between the management team presenting the
training and the employees taking the courses. There needs
to be a high level of support from management in order
for the powerful sustainability to take effect. A highly supportive transfer climate needs to be present in order for a
solid and lasting transfer of the information back to the job.
The research finds the blended solution to be more effective because it finds key ways to impact climate elements
by inherently creating higher leader involvement through its
delivery approach.
In fact, Leones findings show that these climate factors
can quite literally make or break your training investments.
This research also underscores the crucial need for a Level 6
analysis to understand why the training has such incredible
impact for some, while in other cases, it appears to be a
simple waste of time and resources. Through such analysis,
a company will be able to quantify how much these factors
affect the ultimate ROI.

TALENTACQUISITIONANDDEVELOPMENT

The blended approach used by American Express included

components like leader-led kick-off and wrap-up session
a high degree of overt investment and implied support for
the training, and consistent follow-up conversations built
right into to the program. Leone explains, The blended
approach had more positive results because it had many
of the sustainability tools and conversation starters built
right into the training. This led to the high climate, which
led to higher improvement in leader competencies, greater
impact on business and finally vastly greater ROI. In the end
analysis, we found that the blended approach was resulting
in an ROI 75-100% greater than that of the Instructor-Led
training or strictly Online training approaches.
With leaders at all levels of an organization able to critcally
impact the effects of training programs, the importance of
understanding and creating a high transfer climate should
swiftly move to the forefront of any training initiative or
strategy. As clearly evidenced in this particular research, the
right leader with the right approach will truly work wonders
to maximize any training investments.

11

1.