Benefit Monitoring and

Evaluation
Krishna Prasad Pant, Ph D
 Benefit Monitoring and Evaluation (BME)
• Benefit: expected end result of any planned action.
• Monitoring: process of scanning the internal and external
environment that affect the desired benefit in the long run.
• Evaluation: making of a judgment about the amount,
number, or value of something.

• ADB says: BME is an instrument for assessing a project‘s

socioeconomic impact on the target beneficiaries.
• The World Bank says: BME refers to a group of activities
which seek to improve the beneficial impacts of loans as
well as to incorporate lessons of experiences into future
lending operations.
• BME increases effectiveness of development projects and
gives lessons to future projects.
 BME comprises three activities
1. Baseline: Preparation and analysis of benchmark information on
persons and population groups benefiting from the project as well
as the affected population prior to the project's commencement,
2. Monitoring: observe and check the progress or quality of project
outputs and outcomes delivered to intended beneficiaries over
the project period.
3. Evaluation of project impact a few years (usually three to five
years) after completion when all project facilities and services
have been fully developed.

• BME produces managers and planners with information needed to

enhance the efficiency and effectiveness of development
investments.
 1. Benchmark information
• Gathered prior to intervention
• Describes data and analyses of socioeconomic and institutional
characteristics of persons or groups affected by the project
• Provides information to:
– Help planners and implementers ensure that interventions respond to
the needs and desires of those for whom the intervention is intended
to benefit
– Enable evaluators to assess the impact of the intervention.

• Approaches for preparing benchmark information

– Traditional socioeconomic surveys
– Socio-technical profiles - consultation between project designers and
beneficiaries
– Application forms
– Rapid appraisal methods
– Participatory methods
 2. Monitoring the Benefits
• Management information system (MIS) is a system
using formalized procedures to
– provide management at all levels and all functions with
appropriate information based on data from internal and
external sources,
– enable them to make timely and effective decisions for
planning, directing and controlling the activities for which
they are responsible.
• More effective is informal monitoring- personal
contacts, field visits, informal meetings, telephone and
exchange of correspondence.
• Monitoring conducted by managers or development
agency to help them ensure that benefits are derived
from the development investments.
 General characteristics of the Monitoring information

• Timeliness and frequency – real time, daily,

weekly, monthly, quarterly, biannual, yearly
• Appropriateness and level of detail- relates to
task and responsibilities, necessary details
• Understandability- no unnecessary jargons
• Accuracy- confidence on decision making
 What to monitor?
Sources of information

• Delivery of services to the recipients- reasons for deviations

if any
– official records of service delivery agency
– Information obtained from recipients of the services
• Use of services by the recipients – whether using for
intended purpose
– Agency
– Recipients- informal contacts
• Effects of services: Immediate direct effects or benefits from
using the services
– Records of the agency
– Publications of other agencies
– Survey of beneficiaries
 3. Evaluating benefits

• Measure outcomes and impacts of an activity and

distinguishing these from the influence of other, external
factors.
• Help to clarify whether costs for an activity are justified.
• Inform decisions on whether to expand, modify or
terminate the project, program or policy.
• Draw lessons for improving the design and management
of future activities.
• Compare the effectiveness of alternative interventions.
• Strengthen accountability for results.
 Models of Impact Evaluation
Model Design Example
1. Randomized Subjects are randomly assigned to Water supply and sanitation,
pre-test post-test project and control groups. Data housing, community
evaluation collection instruments are applied to infrastructure etc where
both groups before and after demand > supply
2. Quasi-expt. Control group is selected which Low-cost housing programs
design matches the characteristics of the Pilot Project for Climate
project group as closely as possible. Resilience (PPCR)
3. Ex-post Data collected on project beneficiaries Assessing the impacts
comparison and a control group after implemented. of micro-credit programs
of project and Multivariate analysis is often used to
non-equivalent statistically control for differences
control group in the attributes of the two groups.
4. Rapid Participatory methods allow Assessing community
assessment groups to identify changes resulting managed water supply
ex-post impact from the project, who has benefited projects
evaluations and who has not, and what were the
project’s strengths and weaknesses.
(triangulation with KIS, case studies).
 Logical framework (Logframe)
• Management tool used to improve the design of interventions,
most often at the project level.
• It involves identifying strategic elements (inputs, outputs,
outcomes, impact) and their causal relationships, indicators,
and the assumptions or risks that may influence success and
failure.
• It thus facilitates planning, execution and evaluation of a
development intervention.
• Results-Based Management (RBM): A management strategy focusing on
performance and achievement of outputs, outcomes and impacts.
• Management by objectives (MBO): a process of defining objectives
within an organization so that management and employees agree to the
objectives and understand what they need to do in the organization in
order to achieve them.
 Steps in logframe
• Establishing objectives,
• Establishing cause-and-effect relationships among
activities, inputs, outputs and objectives,
• Identifying assumptions underlying the causal
linkages, and
• Identifying objectively-verifiable measures for
evaluating progress and success.
 Benefits of logframe

• It’s the core reference document through out the entire

project management cycle (design, implementation,
evaluation)
• It’s based on international standards, methods, and
terminology
• Improves causal logic: Establish a causal link between inputs,
activities, results, purpose and overall objective (vertical logic);
• Establish a system for M&E of project performance
• Assumes projects are a learning process
• Establish a communication and learning process among the
stakeholders (clients/beneficiaries, planners, decision-makers
& implementers).
SUMMARIZED DESCRIPTION INDICATORS VERIFICATION ASSUMPTIONS
OF OBJECTIVES FOR OBJECTIVE MEANS
VERIFICATION

GOAL: What is the overall objective
that the project will contribute to?
PURPOSE (Specific Objective): What
is the purpose of project to be
realised by the project?
(hypothesis on the intended impact
or benefit).
Results (Outputs): What are
concrete visible results to
contribute to realisation of project
purpose? What changes
and improvements will be achieved
by the project?
ACTIVITIES: What activities are
required and in what order
in order to achieve the expected
results ?
What indicators are What are information Developments, conditions or
linked to overall sources for these external decisions required for
the sustainability of the benefits.
objective? indicators?
Indicators showing the What are information Factors and conditions not
extent to which sources under direct project control,
for these indicators?
project but necessary to achieve the
purpose has been overall project objective?
achieved?
What are the indicators What are information Factors and conditions not
showing whether sources for these under direct project control,
expected results have
indicators? but necessary to achieve the
been achieved?
project purpose?
Means: What Costs: What are the costs Factors and conditions not
resources are required for each of under direct project control,
the means?
for implementation of but necessary for results to
What are the
listed activities? means/actions be achieved as planned?
costs, breakdown of the
budget?
Preconditions:
What pre-requisites must be
fulfilled for project
implementation to start?
 Column 1: Summarized Description of Objectives

• Goal: Project importance to society in terms of the long-term

benefits which are not achieved by the project alone
• PURPOSE: Central specific objective(s) of the project in terms
of sustainable benefits to be delivered to the beneficiaries.
Intended situation at the end of the project.
• RESULTS: Tangible products and services delivered or
competences and capacities established directly as a result of
project activities by the completion date.
• ACTIVITIES: Specific tasks (work programme) to be undertaken
during the project’s lifetime in order to obtain results.
 Column 2: Indicators for Objective Verification (OVI)

• GOAL: Measures the extent to which a contribution to the overall

objective has been made.
– Features which can be measured or at least described precisely in terms of
quantity and quality respectively and which show a change in situation.
– Used during evaluation. However, it is often not appropriate for the project
itself to try and collect this information.
• PURPOSE: Conditions at the end of the project indicating that the purpose
has been achieved (quantity, quality and time).
• OUTPUT: Measures of the quantity and quality of results. Result indicators
are predominantly used during monitoring and review.
• ACTIVITIES: Resources required for implementation of listed activities?.
– Indicate what are the inputs required, main resources to be applied e.g.: TA,
personnel, equipment, training, studies, supplies, etc.
 Means of Verification and Assumptions

Column 3: Means of Verification:

• Indicate where and in what form information on the
achievement of objectives and results can be found, e.g.
reports of ministries, project reports, laws, statistics,
assessments, etc.
Column 4: Assumptions (External Factors):
• Assumptions are conditions which could affect the progress of
the project but which are not under direct control of project
management.
• An assumption is a positive statement of a condition that
must be met for the project’s objectives to be achieved.
 Performance indicators
• Are measures of project impacts, outcomes, outputs,
and inputs that are monitored during project
implementation to assess progress toward project
objectives.
• Are also used later to evaluate a project’s success.
• Indicators organize information in a way that clarifies
the relationships between a project’s impacts,
outcomes, outputs, and inputs and help to identify
problems along the way that can impede the
achievement of project objectives.
 Performance indicators continue

• A quantitative or qualitative factor or variable that

provides a simple and reliable means to measure
achievement, to reflect changes connected to an
intervention, or to help assess the performance of a
development actor (DAC Glossary of Key Terms in Evaluation, 2002)
• A description of the project’s objectives in terms of
quantity, quality, target group(s), time and place (EC
2001. Manual Project Cycle Management)
 Types of Indicators (1/3)

• Impact indicators: Measure highest objectives – improved

welfare and living standards
• Effect (outcome) indicators: Measure the results (quantity
and quality) from the goods and services produced by the
program/project
• Output indicators: Measure quantity of goods and services
created or provided with inputs
• Input indicators: Measure resources allocated to program or
project
 Types of indicators (2/3)
• Risk indicators: measure the status of the exogenous factors
identified as critical through the risk and sensitivity analysis
performed as part of a project’s economic analysis.
• Efficacy indicators: show how well the results at one level of
project implementation have been translated into results at
the next level:
– the efficiency of inputs,
– effectiveness of project outputs, and
– sustainability of project impact.
• Efficiency indicators: usually represent the ratio of inputs
needed per unit of output produced—for example, physical
inputs, dollars, or labor required per unit of output.
– Accountability indicators: Is a subset of efficiency indicators. They
measure the extent to which resources are available for and
appropriately applied to the activities for which they were targeted.
 Types of indicators (3/3)
• Effectiveness indicators: represent the ratio of outputs (or the resources
used to produce the outputs) per unit of project outcome or impact, or
the degree to which outputs affect outcomes and impacts.
– For example: Number of vaccinations administered (or their cost) per unit decline in
morbidity rate (illness prevented) or per unit decline in mortality rate
– Number of farms visited per measured change in farm practices (number of farmers
adopting new practices), or number of farmers adopting new practices per unit increase
in agricultural productivity
• Sustainability indicators: represent the persistence of project benefits
over time, particularly after project funding ends.
– Disease incidence trends after external funding for a vaccination project ends
– Persistence of changed farming practices after extension visits are completed
– Maintenance and use of roads after highway construction ends
– Persistence of institutions (programs, organizations, relationships, and so on)
created to deliver project benefits.
 Direct and Indirect Indicators
• Direct indicators - refer directly to the subject they have
been developed for
– Possible for operational and more technical subjects.
• Indirect (proxy) indicators, which only refer in an in direct
way to the subject
– Subject of interest cannot be measured directly (qualitative
subjects, like behavioural change, living conditions, good
governance, etc.)
– Subject of analysis can be measured directly, but it is too
sensitive to do so, eg. level of income, safe sex (in the context of
an HIV/AIDS intervention)
– Use of an indirect indicator can be more cost-effective than the
use of a direct one.
– Typical management tools - managers are not looking for
scientifically reliable data but for management information.
 Features of Good Indicators
• Is a direct and clear measure of progress;
• Is relevant
• Varies across areas, groups, over time and is sensitive to
changes in policies and program
• Not easy to manipulate e.g. repetition rates
• Can be tracked in a cost effective manner at desired
frequency
 SMART Indicators

• S Specific
• M Measurable
• A Achievable
(acceptable, applicable, appropriate, attainable, agreed upon)

• R Relevant (reliable, realistic)

• T Time-bound
 Steps in formulation of SMART indicators
1. WHAT: Brainstorm on the variables, which may provide means to measure
change in the objectives or phenomena.
• Direct indicators: Cost-effectiveness.
• Indirect indicators:
– validity: causal relation with phenomena of interest
– accuracy: is the variable measurable in a sufficient precise way
– sensitivity: is the variable reacting quickly and clearly enough
– cost-effectiveness: right balance between reliability and efforts needed to obtain the data
2. HOW MUCH: Define the magnitude of the change we want to achieve
3. WHO: In order to clarify who belongs to the target group. Often specific
information on who belongs to the target group is necessary, for example
in cases when gender specificity is required.
4. WHERE: includes specific information on the intervention area, if this does
not yet become clear from step 3.
5. WHEN: includes the definition of the timeframe.
 Disaggregating Indicators

• Geographic areas - urban/rural, mountain/hill/terai

• Administrative units – district, regions
• Gender
• Age group
• Income or consumption groups- rich, poor
• Socially defined groups- Dalit, Tribal, Madhesi, Chhetri-Bahun
 General principles for selecting indicators
• Relevance- to the basic sectoral development objectives of the
project and, if possible, to overall country objectives.
• Selectivity- should be a few and meaningful. No more than a
dozen indicators, at least half of them impact indicators that
explicitly measure project impact against each major
development objective.
• Practicality of indicators and data collection. The data
required to compile the key indicators must be easily available;
if collecting the data will require a special effort, this need
should be identified early in the project cycle and included in
the project design.
 Theory of Change
• Reflects the underlying process and pathways through
which something hoped for change is expected to occur
– knowledge, behaviour, attitudes or practices, at the individual,
institutional, community or other level.
• TOC defines the pieces and steps necessary to bring about
a given long-term goal and describes the types of
interventions (whether a single programme or a
comprehensive community initiative) that bring about the
results hoped for.
• TOC:
– Demonstrates the pathway of how to get from here to there (i.e.
what is needed for goals to be achieved)
– Requires underlying assumptions to be detailed out in a way
that they can be tested and measured
– Puts the emphasis first on what we wants to achieve rather than
on what we are doing.
 6 Steps of TOC
• Step 1: Clarify Goals. First, identify the ultimate impact you want to
achieve in your community. The impact will generally be an
ambitious visionary statement that stands for a healthy, thriving
community – not specific program clients or the results of a specific
strategy.
• Step 2: Identify Powerful Strategies. Consider specific strategies or
programs which address your ultimate impact. These strategies may
include program strategies, campaigns, initiatives, collaborations,
public awareness efforts, capacity-building efforts, community
mobilization efforts and so on.
• Step 3: Create "So That" Chains. Take the first strategy listed and
create a "so that" chain based on the following question:
"We do X strategy so that (blank) results for individuals, families,
organizations or communities?" The answer should be the direct
outcome or result of the strategy. Repeat this question until you have
linked each strategy to your goal.
 Sample “so that” chain of TOC
STRATEGY: Communicate with key organizations, service providers, business networks and advocacy
groups about the Making Connections vision
So That
• There is increased attention on the focus of the Making Connections initiative [Influence]
So That
• 1) There is increased knowledge of and interest in the Making Connections agenda [Influence] and
• 2) There is increased awareness of how partners can work together and what each partner can
contribute to achieve Making Connections results [Influence]
So That
• 1) There is increased shared commitment to a common vision for the community [Influence] and
• 2) There is increased investment of local resources in the Making Connections agenda [Leverage]
So That
• 1) A shared plan of action is developed [Influence] and
• 2) There are increased resources available in the community to support children and families
[Influence]
So That
• There are positive changes in the lives of vulnerable children and families [Individual Impact]
 TOC Steps
• Step 4: Link Strategies with Outcomes and Goals. Place the outcomes that
form the "so that" chain between the strategy and the goal.
• Step 5: Test the Logic and Relevance. Test whether logical linkages occur
between the strategies, outcomes and impacts; whether the most
relevant outcomes are included and whether you have included all of the
relevant strategies.
• Based on these tests, revise, test and revise some more. It is typical for a
theory of change outcome map to be revised several times before it
provides a complete and clear picture of your community change effort.
• Step 6: Articulate Assumptions. While the outcome map offers a visual
sketch of the pathways to achieving outcomes, this work is embedded in a
context.
• Complete the story by articulating the assumptions that influenced the
map's design.
• It is useful to provide narrative information about the principles and belief
system that underlie the outcome map.
 Conclusions
• BME comprises benchmark information, benefit
monitoring and benefit evaluation.
• Logframe is a thinking tool for BME
• Finding good indicators and their precise
measurement determine the success of BME
• Recent development in BME is theory of change (TOC)
that assess the underlying process and pathways for
accessing the changes.
 Exercise
• Develop a logframe with SMART indicators for
the following projects:
– Group A: Childcare Center Development project in
TU central campus, Kirtipur.
– Group B: Fruit Tree Plantation project in TU
central campus, Kirtipur.
– Group C: Ringroad to CEDECON Shuttle bus
service project in TU, Kirtipur.
– Group D: Solar photo voltaic system fitting in
CEDECON in TU central campus, Kirtipur.