1

1 11 11
An Overview of Monitoring and Evaluation
M
any professionals w ho w ork in social service program s cringe w hen
they hear the w ords m onitoringand evaluation. W hen asked to
w rite their im pressions of these term s, participants in a recent M & E w ork-
shop responded w ith descriptions such as cum bersom e, a w aste of
valuable tim e, and som ething im posed from above. A fter learning
m ore about the benefits and techniques of M & E, how ever, these prac-
titioners w ere converted into enthusiastic supporters. W hy the change
ofheart?
The changes in attitudes cam e from :
an understanding that the prim ary purpose of m onitoring and evalua-
tion is project im provem ent;
a recognition that m onitoring and evaluation can be tailored to fit the
specific needs of a project and usually its budget;
a better understanding and m astery of actual M & E processes.
B ut the prior skepticism of these professionals w as not unw arranted.
Traditionally, M & E have been tacked on to a project, and in a m anner
often quite rem oved from planning and im plem entation processes. M oni-
toring system s, w here they have been in place, often have been used
alm ost solely for the production of national level reports. Evaluations
frequently have been lim ited to external team s or individuals w ho arrive
at the com pletion of a project to look at existing data, speak w ith indi-
viduals involved, collect im pressions and w rite a reportall often w ithin a
few w eeks. The m otivation for even these lim ited efforts usually has been
the requirem ents of a governm ent or donor.
This m anual reflects a very different orientation.
2
What Are Monitoring and Evaluation?
The body of social science know n as evaluative r evaluative r evaluative r evaluative r evaluative resear esear esear esear esearch ch ch ch ch is the system -
atic collection of inform ation on the design, im plem entation and effect of
projects on targeted populations (R ossi and Freem an, 1993). Ideally, the
process is divided into an ongoing m onitoring system m onitoring system m onitoring system m onitoring system m onitoring system and periodic periodic periodic periodic periodic
evaluations evaluations evaluations evaluations evaluations w ith som e special studies special studies special studies special studies special studies designed to answ er specific
What monitoring and evaluation can offer
Monitoring and evaluation can . . .
assess the quantity, quality and timeliness of project inputs (M)
identify operational constraints to project effectiveness thus help-
ing planners and managers improve implementation (M)
determine if a process or service, e.g., food fortification, is meet-
ing national or some other accepted/set standard (M)
determine whether a project is serving intended beneficiaries (M)
provide information to improve targeting (M)
help to identify effects that are attributable to a project (E)
provide information which will permit cost-effectiveness com-
parisons with other projects seeking to accomplish the same ob-
jectives (E)
meet donor accountability requirements (M, E)
serve as a vehicle to increase community participation (M, E)
inform decision-making on the future of a project (M, E)
3
questions about the project. M onitoring and evaluation are distinct,
though related efforts, w ith different overall objectives, and, therefore,
require differently designed system s.
M onitoring M onitoring M onitoring M onitoring M onitoring
1 11 11
is concerned prim arily w ith the ongoing collection and re-
view of inform ation on project im plem entation, coverage and use. B y
collecting inform ation on a regular basis throughout the life of a project a
m onitoring system can be used to assess the quality of project inputs
and services, the tim eliness of service delivery, the degree to w hich the
targeted individuals and com m unities are reached, the acceptability and
actual use of services, the costs involved in im plem enting the program ,
and the extent to w hich actual im plem entation coincides w ith the
projects im plem entation plan. A n effective m onitoring system also pro-
vides an im portant input for project staff supervision.
M onitoring data are often entered into a m anagem ent inform ation system
(M IS) w hich, in turn, provides inform ation in an easy-to-use form at to
keep track of project activities, budgets, and personnel. Inform ation gen-
erated from a m onitoring system provides valuable clues as to w here
problem s are occurring, w hy operations are succeeding or faltering, and
w hich specific aspects of a project need to be adjusted to im prove tar-
geting, coverage and im plem entation. M oreover, since m onitoring infor-
m ation is collected and review ed at regular intervals, areas of concern
can be addressed as they arise and corrective m easures can be insti-
tuted, thus im proving the chances for project success.
O ne im portant characteristic of nutrition project m onitoring is that , in a
w ell-functioning project m ost data needed for an M IS is already being
collected for program m atic purposes so that establishing a m onitoring
system should not im pose an additional burden. In an integrated com m u-
1. M onitoring is som etim es referred to as process evaluation or im plem entation
evaluation.
4
nity-based project, for exam ple, this w ould include ongoing grow th m oni-
toring data, as w ell as inform ation on activity attendance, supplem ent
distribution, and clinic referrals.
W hile m onitoring inform ation is used prim arily for m anagem ent decisions,
it is also im portant for providing contextual inform ation for evaluations.
Evaluation Evaluation Evaluation Evaluation Evaluation seeks to m easure project effects, i.e., w hether and to w hat
extent the projects inputs and services are im proving the quality of
peoples lives. Evaluations provide inform ation on the changes in the
behavior and conditions of targeted com m unities and individuals (R ossi
and Freem an) by assessing the effectiveness of the project in attaining
its originally stated interm ediate and overall objectives. A s w ith project
m onitoring, how ever, evaluations m ay also reveal unexpected findings,
both positive and negative, w hich can be used to alter and im prove
project design and im plem entation.
G enerally, follow ing an initial baseline survey, one or tw o m idterm evalua-
tions take place in the m id to late stages of a project and an endpoint
2
evaluation is conducted upon project com pletion or at the end of a fund-
ing cycle. Ideally, there should be an efficient ongoing m onitoring system
in place from the start of the project. If such m onitoring indicates that
im plem entation is proceeding reasonably w ell, the form al evaluation can
be lim ited to (a) the verification of the m onitoring system and (b) the pro-
vision of inform ation on outcom es and im pacts.
3
In the absence of a
2. The term endpoint evaluationw ill be used throughout this guidebook to denote
studies w hich are conducted upon the com pletion of a project to m easure outcom es
and im pacts. O ther term s com m only used are sum m ative or im pact evaluation.
3. In cases w here m onitoring indicates that im plem entation is seriously substandard,
evaluations of project effect m ake little sense. In one recent situation in southern
A frica, form al evaluations w ere canceled despite the existence of baseline data and
control groups because delivery of project inputs w as so flaw ed that no im pact could
be expected.
5
Field Insight: Neglecting the Reasons for Change
I
n Vietnam, community health and nutrition workers observed
that, despite comparable socioeconomic status, some children
were growing adequately while most suffered from varying degrees
of under-nutrition. Using the positive deviance* approach, the be-
havioral characteristics of successful households were identified.
These behaviors included the collecting of shrimp and crabs while
working in the fields and supplementing a childs diet with them.
An educational campaign promoting such positive deviant behav-
iors was implemented. After several months, an evaluation was
conducted to investigate the effect of the project on nutritional sta-
tus. Evaluators were pleased to discover that nutritional status had
substantially improved since the baseline. However, the evaluation
collected only anthropometric data and neglected to determine
whether the adoption of positive deviant behaviors had, in fact,
taken place. While the overall improved growth of children in the
project area is cause for celebration, the conclusion that this resulted
from the educational campaign may have been incorrect. In fact,
deworming medication had been introduced into this area during
the same period and may have contributed substantially to the de-
creased rates of under-nutrition. Simply examining impact relegates
the reasons for change to a black box. Unfortunately, this approach
to evaluation is all too common: improvement in impact indicators,
where it is observed, is assumed to be attributable to the project
without examining process (what we later will define as output and
outcome) indicators.
*Positive deviance refers to situations in which individuals or households are
doing better than would be expected given their social and/or economic circum-
stances. Their time and resource allocation strategies may be worth disseminat-
ing more broadly.
6
reliable and com prehensive m onitoring system , how ever, an endpoint
evaluation w ould have to include an explicit assessm ent of the im ple-
m entation process to determ ine the extent to w hich the target population
actually w as reached and services delivered. W ithout this inform ation,
any absence of positive im pact w ill leave unansw ered the question of
w hether the problem w as a structural defect or faulty im plem entation.
Sim ilarly, even positive results cannot be attributed to the program inter-
ventions w hen inform ation about the process is absent.
W hile project m onitoring clearly serves the interests of program funders,
program m anagers and staff, and beneficiaries, all of w hom benefit from
a process that im proves project operations, each of these groups m ight
resist an im pact evaluation out of concern that the result m ight be nega-
tive. Funders m ight have to adm it to m istakes in judgm ent; program m an-
agers and staff m ight consider their jobs threatened; beneficiaries, if they
are receiving food or other goods or services m ay fear their loss. O ver-
com ing such resistance to evaluation is not alw ays possible, but experi-
ence suggests that resistance can be reduced if the stakeholders of a
project are involved in planning for the evaluation and review ing evalua-
tion data as they are being com piled.
Since m onitoring data are essential to effective m anagem ent, all projects
should be m onitored. M ost projects have som e form of m onitoring system
in place for precisely this reason. Far few er projects conduct regular
evaluations. O f 97 feeding program s in Latin A m erica analyzed by
M usgrove (1991), only 10 included an evaluation, and, of these, only
three used generally accepted evaluation procedures. H um anitarian
w orkersattitudes that every available penny m ust go to feeding hungry
children, and program m anagersconcerns w ith tim e and financial con-
straints, w ere explanations m ost often given w hy so few evaluations are
conducted (M usgrove, 1991). In other cases, project funders and
im plem enters believe that the project is so obviously beneficial, and the
potential for negative effects so sm all, that evaluation is sim ply a w aste of
scarce resources. They m ay argue that evaluation resourcesnot just
7
m oney, but staff tim e and disruptionw ould be better spent to expand
the project.
This reasoning m ay be dangerous. There are m any exam ples of projects
that have proven to be ineffective, or, in som e cases, have even had
negative effects, despite high expectations for their success. R esources
spent to evaluate a project m ay result in far m ore effective use of the
rem aining resources available to the project. For exam ple, one ineffective
com ponent of a generally effective project m ay be scaled back, saving
resources that can then be used for the m ore effective com ponents. O r
an evaluation m ay find a project is effective in addressing the needs of
one target group but not another, so that som e resources can be redi-
rected in m ore effective w ays. O r negative side effects of a generally
effective program can be reduced or elim inated by suitable program
m odification.
C om m itted project staff and m anagem ent often believe strongly in the
value of the project they operate; they w ill focus on the successes and
perhaps neglect to see the less successful aspects of the project. O nly
system atic evaluation can truly verify or m odify these positive im pres-
sions. For exam ple, a com prehensive review of supplem entary feeding
program s, conducted in the 1980s, show ed that targeted supplem entary
feeding of m alnourished infants and preschoolers in the absence of
com plem entary health services show ed no effect on the nutritional status
of this target group (A nderson, 1977; B eaton, 1982). The evaluation w as
considered virtual heresy at the tim e, but it eventually led to careful ex-
ploration of the reasons for this result. A s a consequence, supplem entary
feeding program s are now far m ore likely to be im plem ented in the con-
text of com prehensive, integrated health and nutrition services. B ut at the
tim e this evaluative review w as carried out, m any of us w ere so set in our
conviction about the autom atic translation of food supplem ents to im -
proved nutritional status that w e w ould have argued (and som e did) that
evaluating such program s w as a w aste of resources, virtually taking food
from the m ouths of hungry children.
8
Evaluations need not alw ays be elaborate, lengthy, or costly. If m onitoring
data strongly suggest the existence of positive effects, an evaluation m ay
sim ply verify that these effects are attributable to the project, by com par-
ing current w ith baseline data, and by com paring beneficiaries in the
target area w ith com parable individuals or households in areas w ithout
the project. The com plexity of an evaluation depends in part on resource
availability and in part on the com plexity of the project itself, but often a
focus on a few m easurable im pact indicators is sufficient, if ongoing
m onitoring data are sound. This underscores the need for effective and
com prehensive m onitoring, w hich serves both m anagem ent needs and
the needs of evaluation at key points in the life of a project.
How M&E Fit into a Project
W hile m onitoring and evaluation are com plem entary, they are tw o distinct
processes. M onitoring follow s a m anagem ent m odel w ith a focus on
im proving day to day operations. Evaluation uses a research m odel to
assess the extent to w hich project objectives have been m et or sur-
passed. H ow ever, m onitoring and evaluation are m ost effective as inter-
w oven activities. Together they can provide inform ation that w ill help
decision-m akers choose an appropriate course of action for the future of
the project or on the direction of future projects. D epending on the M & E
findings, decision m akers m ay decide to:
continue the project, either as it is currently im plem ented or w ith
revisions;
expand the project by increasing the target population;
replicate the project in a new setting; or
curtail the project and reallocate the resources elsew here.
O ften, as indicated at the outset, evaluations, or m ore accurately, assess-
m ents, are exercises tacked on to the end of projects to exam ine project
im plem entation and im pact. A s w ill be m ade clear in the sections w hich
9
Field Insight: The Benefits of Constructing
a Comprehensive M&E System
B
y designing a comprehensive M&E system during the planning phase, staff
from the first Tamil Nadu Integrated Nutrition Project (TINP) were able to use
M&E data in a number of useful andinnovative ways. The system consisted of the
following six components.
1) Ongoing monitoring of the quality, delivery, coverage, acceptance and utiliza-
tion of the services provided
2) Ongoing monitoring of project costs
3) Ongoing monthly impact snapshots using the growth monitoring data
4) Longitudinal data collection of a sub-sample of households or individuals to
track the participation and benefits accruing to potentially under-served groups
5) Formal evaluations of 1% of the targeted population (consisting of a baseline,
two midterm evaluations, and a final evaluation)
6) Other special studies
This M&E system gave the project a comprehensive feedback system which al-
lowed staff to (a) make timely management decisions rather than having to wait
for evaluation results, (b) monitor on an ongoing basis changes in the nutritional
health of the population, (c) calculate costs for services delivered or impacts
achieved which could then be compared to other programs with similar inputs and
objectives, (d) gain valuable insights on characteristics of drop-outs and non-par-
ticipants, (e) draw conclusions about the effectiveness of the project in producing
short-term outcomes and long-term impacts, and (f) gain deeper insights into the
internal dynamics of the project.
Source: Adapted from World Bank. 1980. Tamil Nadu Nutrition Project Implementation Volume.
Washington, DC: The World Bank, Population, Health and Nutrition Department.
10
follow , this process is rarely capable of evaluating a projects im pact, and
is, in fact, the type of exercise likely to create resentm ent am ong stake-
holders. B y contrast, m onitoring and evaluation should be built into a
project during the design stage. Project planning should alw ays include
the developm ent of parallel M & E system s. B y incorporating M & E from the
beginning, project staff w ill be providing them selves w ith a thorough and
ongoing feedback system that w ill allow them to m ake tim ely m anage-
m ent decisions w ithout having to w ait for the results of an evaluation. A t
the sam e tim e, early planning m eans that a valid baseline survey can be
conducted and control groups established, significantly increasing the
likelihood that the findings of the endpoint evaluation w ill be credible.
Initiating an evaluation after the project is under w ay m akes it m ore diffi-
cult to attribute changes in behavior or condition to the project or quantify
the m agnitude of the change. B oth m onitoring system s and evaluations
are m ost useful if they are incorporated into a project from its inception,
but both are valuable even if introduced later.
Som e w ays in w hich M & E can be used throughout the life of a project are
sum m arized in Table 1.1.
Who Should Be Responsible for Monitoring and Evaluation
There are three basic options for structuring M & E responsibilities:
contracting external
4
m onitoring and evaluation personnel
having a m ix of external and internal (project) personnel
relying on project personnel alone
4. The term externalhere m eans external to the project.
11
Table 1.1 The Role of M&E throughout the Life of a Project
Planning or Late
Redesign Phase Implementation Implementation
(Monitoring and Phase or Post-Project Phase
Evaluation) (Monitoring) (Evaluation)
Focus is on: The design of the Project coverage, delivery, Determining the interme-
project and how it costs, intermediate out- diate outcomes and more
will improve the comes, and other substantial impacts of the
lives of a particular management concerns. project on peoples lives.
population group.
Types of Are the goals, ob- Are the specified inputs What, if any, are the
Questions jectives, and and services reaching outcomes and/or im-
to be activities appro- the targeted populations, pacts of the project on
Answered priate in light and on time? the targeted popula-
by Monitor- of the projects tions?
ing and context? Are inputs of the
Evaluation desired quality? Have the originally
Are the project stated objectives and
inputs and acti- Are inputs being well goals been met by the
vities (including used by the population? project?
training and
materials) likely Do actual project What other effects, in-
to achieve these activities correspond tended or unintended,
objectives? with those spelled out did the project have on
in the project design local communities,
Will the projects or implementation plan? project staff, or govern-
monitoring and ment policies?
evaluation system What are the project costs
produce the infor- and do they correspond to
mation needed the budget plan? If not, what
for critical components of the project
decision-making? are over and under budget?
Are the criteria Is there evidence of short-
used for targeting term, intermediate outcomes
appropriate? that will produce long-term
impacts?
12
In general, the m ore externalthe process, the m ore objective it is likely
to be. A t the sam e tim e, w holly external evaluations often are out of touch
w ith project realities and w ith ongoing m onitoring processes. W holly inter-
nal processes, conversely, w ill assure full fam iliarity w ith the context and
its nuances, but are often considered inadequately objective by decision
m akers and other observers. A dditionally, w holly in-houseevaluations
m ay not have all of the expertise necessary for such a process.
D ecisions about the internal/external balance in an evaluation w ill neces-
sarily vary from project to project. In seeking the ideal balance for a given
project, the follow ing three scenarios m ay be helpful:
Scenario I : An ongoing external evaluation presence
In large, expensive nutrition projects, it m ay w ell be w orth contracting
w ith an external institution w hich w ould be actively involved in evaluation-
related activities throughout the life of the project. Such an entity, w orking
closely and interactively w ith internal M & E staff, could be responsible for
the follow ing:
selection of control groups
collection of baseline data on a representative sam ple from the project
and control populations
subsequent collection of m idterm and end-point data (prim arily on
outcom es and im pacts) on participants and controls
regular quality checks on the m onitoring data being collected inter-
nally by the project
periodic disaggregation of the m onitoring data to assure that particu-
lar groups (e.g., religious, caste, food-insecure, fem ales, those resid-
ing on the outskirts of villages) are not excluded and are sharing in
project benefits
13
special studies identified at the design stage or during project
im plem entation
periodic assessm ents of the perceptions of service providers and
beneficiaries on project effectiveness, constraints and m eans of ad-
dressing them (using the program constraints assessm ent m ethodol-
ogy described in A nnex 1); and periodic assessm ent of field w orker
job satisfaction
analysis of evaluation data together w ith internal staff
Scenario I has the advantage of reducing the w orkload of internal staff
w hich can then devote their energies to project im plem entation and
m onitoring. It also assures an integration of evaluation activities w ith
ongoing m onitoring. (Several recent state level external evaluations of the
Indian IC D S program did not interface w ith ongoing m onitoring efforts,
thereby not only depriving them selves of crucial processinform ation,
but also resulting in confusing conclusions.) A ssum ing a fully com petent
and responsible external institution, the quality of the entire process is
likely to be high, but the evaluation costs w ill also be high. (A s a rule,
m onitoring and evaluation costs should total 35% of total project costs.
If they are m uch higher, less expensive options should be considered. If
they are m uch low er, the M & E process is probably being short-changed.)
Scenario I I : Periodic external presence
In m edium -size projects, it m ay not be necessary to have an external
institution involved in all of the above. Instead the external entity, w orking
at all tim es w ith internal staff, could take responsibility for designing the
evaluation, assisting in the identification of control groups, participating
in the baseline data collection and the m idterm and end-point evalua-
tions, and participating in the analysis. In this scenario, quality checks on
m onitoring data w ould be carried out, at least occasionally, by internal
M & E staff, and special studies w ould be contracted out or conducted
internally.
14
Scenario I I I : External presence at beginning and end only
In sm aller projects, an external evaluator, often a single individual w ith
M & E expertise, w ould be present at the beginning of the project to ad-
vise on the M & E system as a w hole and specifically issues of control
group (or a reasonable substitute), sam ple size and critical indicators,
and, in som e cases, provide necessary orientation for staff w ho w ill be
responsible for data collection and analysis. The external evaluator w ould
then rejoin internal M & E personnel at the conclusion of the project to
review the m onitoring and evaluation data collected and the analysis
carried out, and w ould m eet w ith project m anagers, service providers
and groups of beneficiaries to discuss the process and the conclusions.
W henever an external entity or individual is utilized, care should be taken
to provide clear term s of reference and necessary project docum entation.
R egardless of w hich M & E staff structuring is em ployed, relevant training
of those responsible for M & E operations is essential to ensure quality
data collection, analysis and interpretation and effective action. In nutri-
tion projects, this is likely to include skills in nutritional assessm ent plus
m ore generic ones associated w ith interview ing, focus group facilitation,
and data processing.
What to Monitor and Evaluate
This guidebook breaks dow n m onitoring and evaluation system s into four
principal com ponents
5
that appear particularly w ell suited for utilization in
5. Though in this guidebook the w ords inputs, outputs, outcom es and im pacts are
used to describe the principal project com ponents to be m onitored and evaluated,
there is a w ide range of term s currently being used in project M & E. O ther fram ew orks
have used term s such as perform ance/processes/im pacts; provision/utilization/cover-
age/im pact; and inputs/processes/outcom es/im pacts. R egardless of w hich term s are
used, it is necessary that definitions be clear and that those involved in the M & E of a
particular project (e.g. project staff, external evaluators, and donors) understand
w hich vocabulary is being used.
15
nutrition projects. B eyond its specific M & E utilization, such a fram ew ork is
a useful tool for project design because it provides a m eans for planners
and other staff to (a) articulate how they anticipate project inputs and ac-
tivities w ill achieve the desired effects, (b) reach consensus on the details
of the project, and (c) clarify the term inology that w ill be used for their par-
ticular M & E system . D ividing a project into various com ponents also m akes
it easier later to identify the specific constraints to project effectiveness.
M onitoring focuses on the appropriate and tim ely provision and use of
project resources focusing prim arily on inputs inputs inputs inputs inputs and outputs outputs outputs outputs outputs; evaluation
focuses on w hether the expected im pacts im pacts im pacts im pacts im pacts w ere achieved. B oth m onitor-
ing and evaluation system s assum e that before the project w as im ple-
m ented, the designers conducted a problem or situation analysis,
determ ined the proxim ate and underlying causes of the problem they
w anted to address, and developed the project to deal w ith these specific
causes, linking project inputs inputs inputs inputs inputs w ith desired outcom es outcom es outcom es outcom es outcom es and im pacts im pacts im pacts im pacts im pacts. In
the context of this analysis of the linkages from inputs to outputs to out-
com es to im pacts, financing is considered a given; the projects inputs
are the resources and services purchased w ith project funds.
A Framework for Monitoring and Evaluation
R esources
used to
support the
prim ary
activities of
the project.
Assumptions
Expectations
regarding
the effective-
ness and
quality of
the project
inputs.
The delivery
of goods
and
services.
Assumptions
Expectations
regarding
the w ays
these goods
and services
w ill be used
by the target
population.
C hanges in
behaviors/
practices.
Links pro-
vision of
goods and
services to
im pact.
N utritional
status
m easures.
Effects
resulting
directly from
project
outputs or
indirectly
through
outcom es.
B roader
effects.
Effects
resulting
from the
achievem ent
of im pacts,
usually in
com bination
w ith other
factors.
Inputs Outputs Outcomes Impacts Benefits
16
Inputs Inputs Inputs Inputs Inputs are the m aterials, goods and actions necessary to carry out the
prim ary project activities. These include item s to be delivered to the tar-
get population (e.g. food, m icronutrient capsules or injections, education
m aterials), training of project personnel, and preparation of project sites
or equipping factories. Tim ely availability of adequate equipm ent and
supplies needed for project im plem entationsuch as trucks and gas for
the delivery of food supplem ents or w arehouses for food storageare
also inputs into the im plem entation of the project. M onitoring should yield
inform ation that can answ er questions regarding the procurem ent, pro-
duction, delivery (to project staff) and costs of these resources. For ex-
am ple, are vitam in A capsules being delivered to project staff on tim e
and in the proper quantities? W hat percent of targeted com m unity nutri-
tion w orkers (C N W s) have received training? To w hat extent do the
projects actual input costs coincide w ith the budget plan?
There often w ill be m ore than one task associated w ith a particular input.
The delivery of training to com m unity nutrition w orkers, for exam ple, re-
quires prior recruitm ent, developm ent and production of m aterials, and
perhaps the training of trainers. Each of these sub-com ponents can be
m onitored under the heading of inputs. (A s discussed in Section 2,
m ore com plex and sequential input system s m ay require their ow n flow -
chart or input tracking system such as that illustrated in Table 2.1.)
Input assum ptions Input assum ptions Input assum ptions Input assum ptions Input assum ptions are the expectations regarding the effectiveness and
quality of the project inputs (e.g., vitam in capsules have not lost potency;
fortification equipm ent is installed properly) and the expectations regard-
ing the process of getting these inputs to the output or delivery stage
(e.g., adequate num bers of w eighing scales and grow th charts have
been delivered to project sites; the target population has been properly
identified).
W ith respect to effectiveness and quality assum ptions, the follow ing ex-
am ple m ay be useful. In a project that includes training of village health
w orkers there is an assum ption in the overall design of the project that
17
the w orkers w ill understand the concepts and techniques taught and w ill
be able to convey this inform ation in a useful w ay. If that assum ption is
false, the likelihood of achieving effective outputs w ill, in turn, be ad-
versely affected. A ccordingly, it m ight be useful to m onitor this assum p-
tion by collecting inform ation periodically on the effectiveness of training
activities. Sim ilarly, if an intervention uses food supplem ents, it is as-
sum ed that the rations produced are of adequate quality and caloric
density. In the case of nutrition projects that focus on behavioral change
through nutrition com m unication, it is crucial that the educational m eth-
ods be appropriate for the target population, and that they be targeted to
the behaviors that need to be changed. A s w ith inputs, input assum p-
tions can be system atically m onitored.
W ith respect to process assum ptions, m onitoring m ay include a m echa-
nism to track the placem ent of necessary staff and the delivery neces-
sary inputs at each service delivery point. M onitoring m ight also include
checks on target populations selected. This m ay be less im portant w here
target groups are m ore easily identified, e.g. pregnant w om en, children
under tw o years of age, but m ore im portant w here the target is food inse-
cure households, requiring a transparent process w ith reliable and w ell
understood indicators.
O utputs O utputs O utputs O utputs O utputs refer to the provision of project goods and services to the target
population; these constitute the prim ary project activities. The types of
questions that can be answ ered w ith inform ation on outputs include, H ow
m any of the children eligible for the project w ere given vitam in A cap-
sules last m onth? W hat percent of pregnant m others in the project area
received iron folate supplem ents? H ow m any of the targeted school age
children received dew orm ing m edication in the last six m onths?
O utput assum ptions O utput assum ptions O utput assum ptions O utput assum ptions O utput assum ptions are those m ade about the target population and
their utilization of the goods and services received. Though the delivery
of inputs and services m ay run sm oothly, positive outcom es w ill only
result if certain assum ptions about the target population are m et. D o they
18
understand the m essages? D o they have the resources to put them into
practice? C an the environm ent support the intervention? For exam ple,
doproject participants consum e enough fat to perm it the efficient con-
version of beta-carotene into retinol? D oes the provision of food supple-
m ents for w om en and children increase their total daily caloric intake?
(Even efficient delivery of a supplem entthe output output output output output w ill not lead to im -
proved grow ththe im pact im pact im pact im pact im pactif the supplem ent substitutes for food nor-
m ally consum ed at hom e rather than confirm ing the output assum ption output assum ption output assum ption output assum ption output assum ption,
nam ely that the supplem ent w ill be additive and increase total daily
intake.)
A lthough m any output assum ptions w ill have been addressed in the de-
sign stage of a w ell prepared project (e.g., pre-testing of nutrition educa-
tion m essages w ill have addressed resources, lim itations and literacy
concerns), their critical im portance and the possibility of changed condi-
tions m ay suggest the value of periodic m onitoring.
Inform ation on both inputs and outputs should be collected routinely, and
inform ation on input and output assum ptions at least periodically, to
m onitor a projects operations and thus inform m anagem ent decisions. In
cases w here m onitoring data yield shortcom ings, these indicators can
help pinpoint design and im plem entation w eaknesses of the project.
M onitoring inputs and outputs is also necessary to provide context for
m idterm and endpoint evaluations.
The term s outcom eand im pactare com m only used by developm ent
practitioners in an interchangeable fashion. It m ay be useful, how ever, to
distinguish betw een interm ediate outcom es and m ore substantial im -
pacts. O utcom es O utcom es O utcom es O utcom es O utcom es, as defined here, are the interm ediate effects resulting
directly from project outputs that m ay be necessary to achieve a desired
im pact. In m any nutrition projects, outcom es take the form of behavioral
changes in the target population, such as im proved child feeding prac-
tices or m ore equitable intrahousehold food distribution resulting from
N utrition C om m unication efforts. These behavioral change outcom es m ay
19
then translate into im proved nutritional status, w hich w ould be consid-
ered the im pact.
O utcom es can also include interm ediate changes in the conditions of the
target population. If, for exam ple, a nutrition project designed to im prove
the grow th of children supplies dew orm ing m edication in addition to
other inputs, an interm ediate outcom ew ould be a decrease in parasitic
load. The outcom e outcom e outcom e outcom e outcom e in a take hom e food supplem entation program for
children w ould be the actual consum ption of that food by the child.
In general, even nutrition projects that have been conscientious about
M & E, often fail to collect inform ation on interm ediate outcom es. It should
be noted that for som e nutrition interventions there is no m easurable
interm ediate outcom e; instead the output leads directly to the desired
im pact. This is the case in projects that distribute vitam in A capsules
(im pact), w here im provem ents in m icronutrient status result directly from
taking the capsule (output) (See A nnex 2, Table A 2.1)
6
.
Im pacts Im pacts Im pacts Im pacts Im pacts are the m ore m eaningful changes in the condition of the target
population and generally reflect the prim ary objectives of the project. For
nutrition purposes, it is generally convenient to speak of im pact in term s
of change in nutrition status using anthropom etric and m icronutrient sta-
tus indicators. H ow has the prevalence of iron deficiency anem ia am ong
w om en changed as a result of iron supplem entation? W hat effect has the
project (perhaps directly through on-site food supplem entation- an out-
putor through nutrition counselingan output, resulting in behavioral
6. In areas w ith high rates of infection, this m odel m ight also include an additional
assum ption to the right of the outcom es colum n (or an additional output assum ption if
there are no outcom es), that a food or nutrient consum ed w ill be adequately absorbed
by the bodys intestinal tract. W here infection rates are particularly high, or w here one
nutrients absorption is seriously inhibited by the deficiency of another, this problem
m ay w ell have to be addressed in order for outputs or outcom es to translate into
im pacts.
20
changean outcom e) had on the incidence of w asting in targeted chil-
dren under three years? W hat effect have food supplem ents to pregnant
w om en w ith low body m ass index (B M I) had on the incidence of low birth
w eight (LB W )?
Inform ation on outcom es should be collected during the life of the project
through ongoing m onitoring or, alternatively, through special studies,
m aking clear w hether service delivery has had som e first-level effect on
the quality of life of the target population, such as increased food intake
or im proved caring practices. In cases w here outcom es do not becom e
apparent until the later stages of a project, they can be assessed along
w ith im pact indicators as part of an endpoint evaluation. Together, infor-
m ation on interm ediate outcom es and final im pacts is used to m ake deci-
sions on the future of the project
Finally, benefits benefits benefits benefits benefits are the broader, sustainable changes in public health or
econom ic status that a program seeks to achieve but w hich are inevitably
influenced by a w ide range of other factors. These benefits, such as
decreased infant, child, and m aternal m ortality, im proved econom ic con-
ditions resulting from greater productivity, and increased lifespans, m ay
not be seen until m any years after the project is com pleted. B enefits
usually are not included as indicators of project success unless there is a
special interest (and corresponding funding) for such inform ation, usually
in a long-term research context. Fortunately, m uch has already been
learned and docum ented about the associations betw een im provem ents
in nutrition status (e.g. im proved grow th, higher birth w eights and de-
creases in m icronutrient deficiencies) and im provem ents in m ortality,
m orbidity and productivity.
7
A ccordingly, equipped w ith a particular set of
evaluation-generated im pact data, project personnel m ay be able to
m ake projections on a range of benefits likely to accrue.
7. See, for exam ple, tables 6A and 6B of the W orld B ank Toolkit #3 (Phillips and
Sanghvi 1996).
21
O verall inform ation on inputs and outputs should be collected regularly
as part of a projects m onitoring system and entered into a m anagem ent
inform ation system . Input and output assum ptions also should be m oni-
tored, although usually less frequently and often through special studies.
O utcom es (often) and im pacts (alw ays) from participants and control
groups, should be included in evaluations. This allow s determ ination of
the extent to w hich observed changes betw een baseline data and data
collected subsequently am ong participants, can be attributed to the
project. A t the sam e tim e, it w ill alm ost alw ays be useful to include snap-
shotsof outcom e and im pact variables am ong project participants, even
w ithout control group data, as part of a m onitoring system (rather than
having to w ait tw o and a half years for a m idterm evaluation). In m any
nutrition projects im pact snapshotscan be taken using grow th m onitor-
ing, pregnancy w eight gain, or birth w eight inform ation regularly col-
lected and com paring it both w ith other project areas and w ith data
collected from the sam e area over the course of the preceding year (ide-
ally w ith the sam e m onth in the previous year to assure seasonal consis-
tency) . O utcom e data, usually behavioral in nature, and often collected
through special studies, is also im portant to m onitor w ith som e regularity.
If a nutrition com m unication project designed to increase food consum p-
tion during pregnancy is not having this effect on participants, the project
staff should know this and be acting on it w ell in advance of a form al
evaluation.
It should be noted that this m odel assum es reasonably hom ogeneous
projects from area to area. In a m onitoring system , m onitoring form s, data
collection regim ens and M IS indicator colum ns assum e a discrete set of
inputs, outputs and other categories of inform ation w hich w ill be reason-
ably constant across project areas. Sim ilarly a w ell functioning evaluation
assum es that the indicators used in baseline and evaluation surveys w ill
be the sam e in each area. W hat can be done, then, in projects, such as
the Iringa N utrition Project in Tanzania w hich place high prem ium s on
local determ ination of project activities, and w here, as a result, project
activities vary from com m unity to com m unity?
22
In such cases, M & E options appear to be tw o-fold. First, if the locally
determ ined activities or activity com binations fall into a sm all num ber of
categories, and if the project as a w hole is large enough so that evalua-
tion sam ple size requirem ents w ill be m et, each of these activity com bi-
nations can be considered a cohort, w ith particular sets of m onitoring
data collected in each, and w ith evaluation data analyzed separately for
each (perm itting also a com parison of the cohorts). W here activities from
area to area are too heterogeneous, project m onitoring w ill have to de-
volve largely to the local level, w hile project evaluation w ill necessarily be
lim ited to assessing the com posite im pact of this heterogeneous set of
activities on a pre-determ ined set of im pact indicators.