Escolar Documentos
Profissional Documentos
Cultura Documentos
Revisiting Kirkpatrick’s model – demic programme, it is necessary that every course should
have an in-built monitoring and evaluation system. Here
an evaluation of an academic training we first review the contemporary theories of evaluation
course of training programmes in general and secondly, reveal a
case study on the application of a widely accepted theo-
P. Rajeev*, M. S. Madan and K. Jayarajan retical model to evaluate an academic training course
offered to postgraduate students in life sciences.
Indian Institute of Spices Research, Marikunnu PO,
Calicut 673 012, India
With the changing socio-economic and technological
relevance of training, the definitions, scope, methods and
In scientific and research organizations, the training evaluation of training programmes have also changed.
needs facilitator roles and methods have undergone a One of the earlier classic definitions of training is ‘bring-
change necessitated by rapid information and techno- ing lasting improvement in skills in jobs’2. The present-
logy boom. There is ample evidence to show that evalua- day definitions take a multi-dimensional perspective en-
tion and objective assessment of effectiveness and veloping the needs of individuals, teams, organizations
outcomes of training programmes being implemented and the society. The steps in the training programme
by organizations are not given due importance as that development are planning, programme implementation,
of their planning and implementation. An attempt is and programme evaluation and follow-up. The evaluation
made in this communication firstly to analyse the of any training system helps measure the ‘knowledge
theories of training evaluation in general; the study
gap’, what is defined by Riech3 as ‘the gap between what
also illustrates a case study of training evaluation of
the academic training courses being carried out at the the trainer teaches and what the trainee learns’. Evalua-
Indian Institute of Spices Research by revisiting the tions help to measure Reich’s gap by determining the
popular Kirkpatricks’s model. The three-step evalua- value and effectiveness of a learning programme. It uses
tion model is a combination of formative and summative assessment and validation tools to provide data for the
approaches using multiple methods which measure evaluation. Bramley and Newby4 identify four main pur-
reactions, perceptions, learning and behavioural com- poses of evaluation:
ponents of the trainees combining quantitative and
qualitative tools and aims at assessing the usefulness 1. Feedback – Linking learning outcomes to objectives
of the course in providing an adequate learning cli- and providing a form of quality control.
mate. 2. Control – Making links from training to organizational
activities and to consider cost-effectiveness.
Keywords: Formative evaluation, Kirkpatrick’s model, 3. Research – Determining the relationships among
summative evaluation, training evaluation models. learning, training and transfer of training to the job.
4. Intervention – The results of the evaluation influence
TRAINING is an essential human resource development the context in which it occurs.
(HRD) function of any organization. In the Indian organi-
zational development context, the training needs, strategies, Evaluation of training systems, programmes or courses
methods and investments on training have all undergone tends to be a demand of a social, institutional or economic
a sea change since the last decade. Science and techno- nature5. The demand of a social nature became stronger
logy is growing faster than in the past and to tackle the with the participation of trainees themselves and other
global competition in many sectors, the erstwhile tradi- non-traditional actors like the resource persons in the
tional approach in training policies of organizations has evaluation process. Evaluation is also a growing institu-
undergone a change to one which is more liberal, con- tional demand, since the decision-makers not only need to
cept-based, comprehensive, systematic, well planned and understand the processes, but also to control and act to
dynamic1. For instance, in scientific and research organi- improve the effects of their guidelines. Evaluation is still
zations devoted to life sciences, these influences are due an economic demand given the community involvement
to fast technological developments in frontier areas like in funding. However, despite the importance of the evalua-
biotechnology, bioinformatics, biochemistry and micro- tion process, there is evidence that evaluation of training
biology. The challenges opening up provide a wide range programmes is often inconsistent or missing6–8. The lit-
of opportunities subject to the acquisition of relevant erature on evaluation needs to be classified into education
skills, knowledge and concepts. Realizing the need for and training. The latter reveals many difficulties as regards
HRD support, most scientific institutions have taken up a evaluation. Scientific and quantitative methods are not
facilitating role in providing infrastructure and academic popular. Evaluation appears to be undertaken reluctantly
training support to clientele groups. This communication and with the simplest of methods. Behavioural objects are
is based on the hypothesis that for the success of any aca- rarely even set by the trainers. Progress in the techniques
of evaluation has been slow, though a good deal of res-
*For correspondence. (e-mail: rajeev@spices.res.in) earch has been done. The literature is small, but growing9.
272 CURRENT SCIENCE, VOL. 96, NO. 2, 25 JANUARY 2009
RESEARCH COMMUNICATIONS
Programme evaluation research involves two general or examination. The learning evaluation requires post-
approaches. Formative evaluation (also known as inter- testing to ascertain what knowledge was learned during
nal) is a method of judging the worth of a programme the training. In addition, the post-testing is only valid
while the programme activities are forming (in progress). when combined with pre-testing, so that one can differen-
This part of the evaluation focuses on the process. This tiate between what he already knew prior to the training
approach has been termed as process evaluation10. In es- and what he actually learned during the training pro-
sence, the questions being answered in formative evalua- gramme.
tion are: ‘How was the programme implemented?’ and Level 3. Behaviour: The intention at this level is to
‘Was it implemented as planned?’. Results of formative assess whether job performance changes as a result of
evaluations are often used to improve programme imple- training. This performance testing is to indicate the
mentation by providing a feedback, which can be used to learner’s skill to apply what he has learned in the class-
modify future implementations11. For example, formative room. This evaluation involves testing the students capa-
studies examine issues such as the consistency with which bilities to perform learned skills while on the job, rather
a programme was implemented, other programmes that than in the classroom. Level-three evaluations can be per-
were implemented during the study, attitude towards the formed formally (testing) or informally (observation and
programme and duration of implementation. As applied to judgments).
HR programmes, formative evaluation offers a means of Level 4. Results: The intention at this level is to assess
assessing and improving on programme validity12. The the costs vs benefits of training programmes, i.e. orga-
summative evaluation (also known as external evaluation) nizational impact in terms of reduced costs, improved
is a method of judging the worth of a programme at the quality of work, increased quantity of work, etc. It mea-
end of the activities (summation). The focus is on the sures impact, which includes monetary efficiency, moral,
outcome. Summative evaluation assesses the extent to teamwork, etc. Collecting, organizing and analysing
which the intervention achieved the outcomes described level-four information can be difficult, time-consuming
by its goals. Often summative evaluations utilize quasi- and more costly than the other three levels, but the results
experimental research designs such as pre-test/post-test, are often quite worthwhile when viewed in the full con-
randomized control group design, time series, or a com- text of its value to the organization29.
bination of each. After examining the methodological framework, a case
The most influential framework for the evaluation of study is presented on the evaluation methodology at the
training programmes has come from Kirkpatrick6,13–16. Indian Institute of Spices Research, Calicut. Among other
Kirkpatrick’s model17 follows a goal-based approach. programmes, training courses are organized presently at
Even now this is the most popular model being used with the Institute in the field of bioinformatics and biotechno-
required modifications and is applicable to any organiza- logy and biochemistry. The trainees are postgraduate stu-
tional setting. For instance, this model has been applied dents in life sciences and the objective of the course is to
in evaluating training imparted to child-welfare profes- impart knowledge on concepts and methods in these
sionals as well as entrepreneurship development training fields and impart skills in using various scientific tools.
programmes18,19. Most of the models in use today are The curriculum involves hands-on training on skills
modified versions of Kirkpatrick’s four-level frame- related to analytical techniques in biochemistry like GLC,
work20,21. Several weaknesses have been identified with GCMS, HPLC, etc., isolation of proteins, enzymes, DNA,
Kirkpatrick’s model, including overemphasis on the reac- RNA, plant tissue culture and micropropagation, DNA
tions of trainees, low correlation between reactions and markers, preparation of molecular maps and molecular
performance, low correlation between measures at differ- approaches in the detection and isolation of plant patho-
ent outcome levels, and incompleteness of the model22–26. gens. The course duration is 30 days and trainees are
Other models for training evaluation have been presented selected based on acceptable standards of their perfor-
in the literature, including contextual evaluation27, res- mance in the regular courses they do in order to ensure
ponsive evaluation28 and balanced score card. homogeneity as far as possible.
Kirkpatrick’s model is based on four simple questions These training programmes have an inbuilt monitoring
that translate into four levels of evaluation. These are: and evaluation system; formal evaluation of participants
Level 1. Reaction: At this level, data on the reactions in terms of their achievements and self-appraisal by par-
of the participants at the end of a training programme are ticipants in terms of their perceptions about training with
gathered. This level is often measured with attitude que- the ultimate objective of assessing the usefulness of aca-
stionnaires that are passed out after most training classes. demic training in providing an adequate learning climate
This level measures the learner’s perception (reaction) of that could be linked with their career development. A
the course. three-stage evaluation method is followed: training orien-
Level 2. Learning: The intention at this level is to assess tation and pre-training evaluation, concurrent evaluation
whether the learning objectives for the programme are and post-training evaluation. Knowledge gain, on job per-
met. This is usually done by means of an appropriate test formance of skills and training effectiveness index are the
CURRENT SCIENCE, VOL. 96, NO. 2, 25 JANUARY 2009 273
RESEARCH COMMUNICATIONS
three major dependent variables in the method. Assess- gramme was organized in May–June 2008, in which 24
ment of cognitive learning is measured as knowledge trainees from various academic institutions in India par-
gain30 and skill in terms of performance. Perceptions of ticipated. The sample was homogenous with respect to
the trainees on the organizational effectiveness are mea- age and the academic background as standardized during
sured as the training effectiveness index. the selection procedure.
During the training orientation session, a platform is The mean pre-training knowledge test score was as low
provided in an open discussion meeting to build faculty– as 8.67 out of a possible score of 25 and the range of scores
trainee rapport and for collecting ‘open responses’ on was wide, i.e. 3.25–17.75. The coefficient of skewness,
‘what are the trainees’ expectations’ in undergoing the +1.25 indicated that the scores of the sample were dis-
course. During this session each trainee is assigned an tributed more above the reported mean. The coefficient of
advisor faculty for guidance throughout the duration of variation was as high as 39.11, which indicated a higher
the course. variation among the samples on knowledge domain
During pre-training evaluation, a comprehensive, quiz- before commencement of the training.
type knowledge test is administered to assess the initial The mean knowledge score in the post-training know-
level of knowledge. This test is prepared with more ledge test increased to 17.45 after completion of the train-
knowledge-type questions rather than understanding and ing and the range narrowed down to 12.5–22. The
skill-oriented questions. coefficient of variation (CV) decreased to 15.08, which
During the concurrent evaluation session, performance- indicated that the sample was more homogenous as far as
oriented tests are given. At this stage a mix of objective knowledge scores are concerned. However, the coeffi-
and subjective tests is employed. This consists of an cient of skewness was negative at –0.02, which indicated
unannounced surprise performance test comprising more that the sample distribution was below the reported mean.
in-depth understanding and descriptive type of questions, The t value of 1.35 indicated that there was significant
judgment of performance of trainees in practical labora- difference in mean knowledge scores prior to and after
tory sessions (judges’ rating) and judging the comprehen- training at 0.01 level of significance.
sion and communication skills through a pre-decided The mean of the objective-cum-descriptive perfor-
term paper prepared by the trainees and also presented by mance test was 11.78, out of a maximum possible score
each trainee in a seminar. Concurrent evaluation is ori- of 20, with a wide range, i.e. 4.3–18.3. The CV was
ented towards reflecting upon the performance, soft skills acceptable at 29.31. Here also, the negative coefficient of
and measuring the knowledge as well as skills (beha- skewness of –0.33, indicated a distribution of trainees
vioural component) likely to reflect in the sphere of activity below the reported mean. The mean rating score of prac-
of the trainees. Here, knowledge and skill constructions tical skill session judgment was as high as 3.45, with a
are assessed at three levels – personal, professional and maximum possible score of 5 and with a low CV of 22.3.
disciplinary31. Coefficient of skewness was positive at 0.71. The mean
During the post-training session, the knowledge test, on training assignment score was as high as 6.5 out of the
similar to the one given during the pre-training session is maximum possible 10, with a low CV of 12.33, which
repeated with the purpose of measuring the knowledge indicated uniform performance by the sample. Here also
gain. At this stage a questionnaire is also administered32 the coefficient of skewness was –1.66. The mean score
to measure the perceptions, reactions and attitude of the for seminar presentation was 11.85, when the maximum
trainees about organizational effectiveness of the training possible score was 20, with a narrower range of 8.8–16.6
including logistic support. This questionnaire consists of and lower CV of 17.89. Here, coefficient skewness was
pre-tested fixed alternative Likerts rating-scale questions positive at 0.66.
and some open-ended questions. The dimensions in the The efficacy of training was measured using a five-
scale rated are satisfaction in terms of educational experi- point rating scale on four dimensions, namely relevancy
ence, attention, methodology, interest, training materials, to needs, content of training, quality of instruction and
instruction in terms of knowledge and quality and practi- practical and overall effectiveness. The mean rating was
cal sessions. The quantitative data on test scores and rat- highest on the ‘content’ dimension (4.54) and the dimen-
ing scores is analysed using appropriate statistical tools sions of ‘relevancy to needs’ and ‘quality of instruction’
like central tendency measures, skewness coefficient, followed next with a mean rating score of 4.11.
coefficient of variation, indices and paired t-test and mul- The training organizational effectiveness index on di-
tiple correlation coefficients. The open responses are sub- mensions of logistics was as high as 0.85. This indicated
ject to subjective interpretation. a high score on qualitative aspects of the training organi-
The above methodology is being adopted for evaluat- zation. The open responses in efficacy and organizational
ing all the training programmes of the institute. The aspects were also listed and analysed. The most frequent
results of the evaluation of one such course offered in the response was to increase time allotted to hands-on train-
disciplines of biochemistry, biotechnology and micro- ing and improving laboratory support and provision of
biology are presented here as an illustration. The pro- on-campus accommodation for the trainees. With regard
274 CURRENT SCIENCE, VOL. 96, NO. 2, 25 JANUARY 2009
RESEARCH COMMUNICATIONS
Table 1. Statistical analysis