Ways to Measure Your

Craft Skills Program

Maturity Matrix
LEVEL 11
LEVEL
NOT ENGAGED
ENGAGED
NOT

LEVEL 22
LEVEL
EXPERIMENTING
EXPERIMENTING

LEVEL 3
ENLIGHTENED

LEVEL 44
LEVEL
GOOD PRACTICE
PRACTICE
GOOD

LEVEL 55
LEVEL
BEST PRACTICE
PRACTICE
BEST

Skill and Knowledge

Assessments

S&K assessment has not been

performed. There is little or
no formal understanding of the
specific requirements for each
position beyond a rudimentary job
description

Some S&K assessments have been

performed, but the results do not
seem inflected in the design of the
training programs for craftsmen.

Formal S&K assessments have

been performed for all craft positions
with the results of this assessment
being fully integrated into the design
of the training program. This was a
one-time significant effort.

Job Task Analysis is the method

of performing skill assessments,
with resultant skill and knowledge
banks used to design the training
programs.

Job and Task Analyses and S&K gap

assessments are routinely analyzed
to determine training needs and to
anticipate future workforce needs.

Task Procedures and

"Standard Work"

The organization has placed

no emphasis on the concept
of standard work and has not
documented any of their critical work
procedures.

Task procedures exist, but they are

informal in nature and consist largely
of copies from owners manuals and
vendor data. It is clear that these
documents do not affect the quality
of work being performed.

The organization has started placing

an emphasis on the development of
standard work and has documented
an extensive library of standard task
procedures.

The standard work process is well

developed with a criticality analysis
being applied to identify those
procedures deemed most important.
Formal active training delivered.

The standard work process is well

developed with an appropriate
library of task procedures being
developed and formal active training
being delivered. These standard
procedures are updated and
audited.

The organization informally solicits

training needs (beyond basic
craft skills) from employees and
supervisors.

Program regularly surveys and

reports gaps and provides training
and intervention to resolve craft
S&K gaps, especially with respect
to future workforce needs, ROI
from cross-crafting, knowledge
capture, while also addresses other
gaps such as RCFA, predictive
maintenance, etc.

A standard process (SDCA) is used

to routinely assess craft knowledge
gaps most critical to business
needs --- includes crafts skills and
knowledge / cross-craft needs,
future workforce needs, and other
domains such as RCFA, TPM,
Predictive Maintenance, Problem
Solving, Planning and Scheduling,
etc. Addresses work environment
factors affecting trainees.

Level 4 plus a complete career

development path defined and
documented for each position to
include not only technical skills,
but interpersonal, leadership, and
problem solving, and analytical
skills where appropriate. Applies
to new hires, job progression,
apprentice-needs where appropriate,
incumbent craftsmen, refresher
strategies, RCFA, multi-crafting, etc.

ANALYSIS

Elements

DEVELOPMENT

Gap Awareness

Basic craft skill and knowledge gaps

(skill-specific, cross-craft, multi-craft,
etc.) are not routinely captured and
analyzed for impact with respect to
workforce development training.

Basic craft skill and knowledge gaps

are the focus of the training program
(technical maintenance skills).

The training program has been

largely developed informally, many
without learning objectives defined
for the training interventions.
Although they many not based on
documented learning objectives,
Various training offers are provided
and there is basic management
documentation for craftsmen training
(new hires, advancement, advanced,
predictive training, RCFA, etc.)

Curriculum (S&K) maps for new

hires and for any craftsmen
progression program are published
and form the basis for the training
programs, designed appropriately
with linkages to the S&K
requirements. Properly developed
learning objectives and job
performance expectations outlined.
Job requirements are tied to
training strategies and performance
objectives, all linked.

Strategies and curriculum maps

are used to guide and manage
the training needs of craftsmen
whether new hire, a craftsman with
a basic level of knowledge wanting
to take the next step, predictive
technologies training, etc. In all
cases, the training interventions
are robustly developed with clear
linkages to the S&K evident in its
development. Properly developed
learning objectives and job
performance expectations have
been defined and are stressed
during delivery.
Course Development Plans are used
both prior to course development
and as part of Continuous
Improvement. These list the media /
mode strategies for each significant
craftsman training offering. The
program utilizes many mixed media
such as video, classroom, CBT, and
printed material.

Blended: The curriculum has been

developed using an appropriate mix
of media to include printed word,
video, instructor-led, self study
modules, hands-on and computer
based training. These media are
available to personnel on demand
with significant guidance and
opportunity for self development
evident.

Training Objectives,
Course Outlines and
Curriculum Maps

The training program developed is

informal, relying largely on publicly
available materials, vendor manuals,
and rudimentary photocopies of
technical information. Applies
to new hires, job progression,
apprentice-needs where appropriate,
incumbent craftsmen, refresher
strategies, etc.

Media/Mode

In this category, training offerings

may tend to be either heavily on
computer-based learning without
enough hands-on, or self-study, or
lecture, or too generic, or too reliant
on unstructured OJT, etc. Course
Development Plans are not used or
available.

There are places where some

training offerings have included a
blended approach of media / modes;
however it seems largely attributable
to individual training providers or
offerings.

Course Development Plans are used

prior to course development These
list the media / mode strategies for
each significant craftsman training
offering. Vendors and in-house
training staff consciously decide
upon the appropriate blend of media/
modes prior to course development.

Instructor Guides &

Participant Guides

The curriculum consists exclusively

of participant materials and
handouts. The materials are
informal and few or no instructor
guides or notes are provided to
ensure consistency of delivery.

When courseware is developed

internally instructor guides are
generally developed, but not
always. Many courses that are
part of the standard catalog do not
have instructor guides - especially
externally developed training.

Instructor guides are developed as

part of the standard training delivery
package - but are seldom used by
the instructors - inconsistencies exist
from instructor to instructor.

Instructors are required to own their

Instructor Guides and to ensure
they are up-to-date so that other
instructors can teach the same
course the same way.

Instructor guides that drive

consistency in delivery are used to
continuously improve the training
process and to help new instructors
easily learn the required training
standards for craftsmen training
offerings.

On-the-Job Training
and Job Performance
Measures

The on-the-job training program

is not formally documented
and consists primarily of an
assignment to accompany another
job incumbent for some period of
time. The expectations from the
training not clear or defined. No
documentation of the process exists.

On-the-job training occurs with

multiple incumbents such that
a variety of view points and
experience levels are leveraged.
No formal structure for the process
exists and the measures of
success are subjective at best. No
documentation of the process exists.

The on-the-job training process

is structured and formal with a
task level listing and performance
expectations are defined.
Expectations are a pass/fail with
subjective feedback given informally.
The process is documented with no
follow-on actions - everyone passes.

OJT formally defined, but peer

feedback process is poor or nonexistent. Performance expectations
defined, but are subjective in nature.
There is a formal documentation
process with candidates required to
improve there skills in certain areas
- not everyone passes the first time
through.

The OJT process is well defined

with expectations and peer
feedback occurring based on the
OJT performance. Instructor
expectations for the OJT are
defined. Continual improvement
element to the process clearly
exists. Quantative performance
expectations clearly defined and
implemented.

No job aids are delivered. All of the

learning and reference material is
provided in the classroom with little
effect visible on the shop floor.

Job aids are incorporated into the

student guides as a reference, but
never appear on the shop floor. It is
clear that these job aids do not effect
the quality of work performed on the
shop floor.

Job aids are used extensively and

posted on-the-job floor to excess.
It is difficult to determine which job
aids are important. The job aids
are delivered passively on the shop
floor with no active communication
occurring.

Evidence exists that a robust

process of shop floor reinforcement
of concepts learned in the classroom
being applied via job aids. Active
communication exists regarding
these documents.

Job aids are develop and exist on

the shop floor with the proper priority
applied. They are periodically
updated and replaced but they are
actively delivered to the workforce.
It is evident that these job aids
have an effect on the quality of
work performed. Continuous
improvement evident.

Job Aids

Continued on back...
North America Latin America Middle East Asia-Pacific
GPAllied Asia-Pac
Level 18, 333 Ann Street
Brisbane QLD 4000
o. +61(0)7.3232.1265
f. +61(0)7.3232.1200

World Headquarters
4200 Faber Place Drive
Charleston, SC 29405
o. 888.335.8276
f. 843.414.5779
www.gpallied.com

GPAllied EMEA
Guldensporenpark 21-Blok C
B-9820 Merelbeke, Belgium
o. +32(0)9.210.17.20
f. +32(0)9.210.17.28

Ways to Measure Your

Craft Skills Program

Maturity Matrix
LEVEL
LEVEL 11
NOT ENGAGED
ENGAGED
NOT

LEVEL
LEVEL 22
EXPERIMENTING
EXPERIMENTING

LEVEL 3
ENLIGHTENED

LEVEL
LEVEL 44
GOOD PRACTICE
PRACTICE
GOOD

LEVEL
LEVEL 55
BEST PRACTICE
PRACTICE
BEST

No curriculum based approach is

applied. Training is delivered in
mass to the entire population and
is delivered to address the current
problem of the day. No strategic
approach.

Training maps are developed,

but are not specific to the job
description. The level of support and
coaching provided to the individual
is lacking. Development of skill and
knowledge is discussed and address
infrequently, generally when pressed
by an outside function.

IDPs are developed to the position

level but are administered poorly.
Some individuals understand and
gain benefit from the IDP, but they
are largely an administrative task
and do not have a significant effect
on the workforce development
efforts.

IDPs are designed to the position

level and are delivered to all
individuals holding that job position.
Some follow-up and coaching is
being done.

IDPs are designed and delivered to

the individuals within the workforce.
Continual coaching and progress
checks applied. IDPs vary based
on the individual needs and current
skill level.

New courses added to the

curriculum are not reviewed by the
job incumbents and no validation or
pilot proof-of-concept process exists.

A cursory review of the course

materials is performed by the job
incumbents, but the process is
done informally- likely via email with
limited feedback and no interaction
of the incumbents during the actual
pilot delivery.

A formal course review is often

performed for high profile course
deliveries, but is not the standard
operating procedure. Improvement
opportunities for high profile courses
are recorded and followed up on, but
not universally for all courses.

A formal review process exists that

is applied to all courses, but the
follow up on opportunities is spotty
and generally a one time effort continual improvement not evident.

The course review and approval

process is rigorous and formal. The
pilot session is delivered to the
most qualified job incumbents with
specific and meaningful feedback
provided. This feedback is
incorporated into the course prior to
providing the training to the general
population.

No dedicated training facility exists

- training is given in any available
space - space is largely unsuitable
for training delivery (Noisy,
temperature, interruptions, etc.)

Dedicated training facilities exist,

but they are somewhat neglected
and poorly maintained. The training
experience suffers due to the
condition of the training facilities.

Dedicated training facilities exist,

and periodic upgrades exist over
the years. The facility periodically
becomes outdated and does
not support all of the training
needs beyond simple classroom
instruction. Many interruptions and
derailers to the training experience
exist.

Dedicated training facilities exist

which are properly designed to
support the optimal training delivery.
Facilities support both classroom,
CBT, and the requisite laboratory
exercises. Minimal interruptions
derailers to the training experience
exist.

Proper dedicated training facilities

exist with a dedicated training
manager assigned to ensure the
continued upkeep and indicated
improvements to the training facility.

Instructor qualification is rarely

considered. Feedback on
instruction is not asked for or given.

Instructor feedback forms are used

for training classes, but they are
informal and consist primarily of
subjective opinions. The results are
not recorded or tracked. No actions
are taken based on the results of
this feedback.

Instructor feedback forms are used

with feedback formally tracked and
corrective actions taken. Forms
consist primarily of subjective
opinions.

The instructor certification

process exists, but is informal
and undocumented and consists
primarily of incumbent or peer
opinions. Feedback forms used,
with corrective actions being taken.

A formal instructor certification

process exists with an individual
task certification process in place
with subject matter experts,
task qualification, and instructor
performance audits being
performed. Corrective actions taken
to address deficiencies.

Learning Contracts
and Performance
Expectations

Learning contracts are not used in

any form. The expectations placed
on those being trained once they
return to the workplace are informal.

Learning contracts are not used,

but performance expectations of
those attending the training are
verbally delivered at the beginning
of the training session. The direct
supervisors of those being trained
are usually not aware or informed
of specific training expectations or
their role in reinforcing and coaching
learned skills on the job.

The direct supervisors of those

trained attend the opening of
the training session and are part
of the discussion regarding the
performance expectations upon
return to the job.

Learning contracts are developed

prior to the execution of the training
and reviewed between the trainee
and their direct supervisor. Clear
and measureable performance
expectations are established for the
trainee upon return to the job.

Learning contracts are used

extensively and are properly
developed (level 4). Follow up
reviews are performed by the
trainers or departmental sponsors
in order to measure the change
in workplace performance, with
a feedback loop existing back to
the development and execution
elements of the model.

Exam Evaluation

Examinations or performance
tests are not given in any form.
The effectiveness of the training
is generally not measured. In
this category there may be a
tendency toward written exams
without enough hands-on, practical
examination.

Exams are reviewed informally posttraining, but the review is performed

in an informal manner and is not
documented. Corrective actions are
not identified and are rarely followed
up on.

Examination reviews are performed

and documented, but the follow-up
and corrective action process
is insufficient. Exams remain
largely unchanged through multiple
deliveries of the training sessions.

Formal examinations are provided

with a single version of the exam
existing. Formal exam reviews to
the question level exist. The follow
up process is somewhat informal
and opportunities for improvement
are often missed. No statistical data
on the examination performance is
maintained.

Written and hands-on examinations

are provided as part of the training
experience with multiple versions
of the exam existing. Statistical
measures are taken to the question
and specific answer level with
problem areas addressed and
corrected via existing feedback
loops to the exam and training
delivery process.

Training effectiveness audits not

considered or performed.

Training effectiveness audits

performed upon return to the
workplace post-training, but they
consist largely of subjective opinions
solely from the participants.

Training effectiveness audits

performed upon return to the
workplace post-training, but they
consist largely of subjective opinions
provided not only by the participants,
but also from the participant's
managers.

Training effectiveness audits

performed, with a mixture of
subjective and quantitative
performance measures (number
errors, productivity, etc.). The
process to make adjustments to the
training based on the audit results is
spotty and inconsistent.

Training effectiveness audits

performed, with a mixture of
subjective and quantitative
performance measures (number
errors, productivity, etc.). A robust
process to make corrections and
upgrades to the training delivery
based on results exists.

IMPLEMENTATION

Elements
Individual
Development Plans
(IDP) & Training
Records

Course Validation and

Pilot -Testing

EXECUTION

Training Environment

EVALUATION

Instructor
Qualification

Training Effectiveness
Audits

North America Latin America Middle East Asia-Pacific

GPAllied Asia-Pac
Level 18, 333 Ann Street
Brisbane QLD 4000
o. +61(0)7.3232.1265
f. +61(0)7.3232.1200

World Headquarters
4200 Faber Place Drive
Charleston, SC 29405
o. 888.335.8276
f. 843.414.5779
www.gpallied.com

GPAllied EMEA
Guldensporenpark 21-Blok C
B-9820 Merelbeke, Belgium
o. +32(0)9.210.17.20
f. +32(0)9.210.17.28