Monitoring & Evaluation of

Social Marketing Programs

Report

Module 7
Presented by Poppet Celdran

Monitoring & Evaluating

Advocacy Communications & Social Mobilization (ACSM)
Programs

WHO. 2007. Advocacy Communication and Social Mobilization (ACSM) for Tuberculosis Control. A Handbook for Country Programs.

Monitoring
1) conducting short-term and long-term monitoring and tracking;
2) recognizing problems via feedback from the field; and
3) making mid-course corrections based on feedback.

Before Advocacy Communications & Social

Mobilization (ACSM) activities begin, create
monitoring mechanisms to receive feedback on
the interventions and identify any problems
early.

1. Conducting short-term and long-term monitoring and

tracking
It is impossible to anticipate every problem, a monitoring system will help identify
difficulties quickly so that they can be addressed. Determine the roles of partners
in solving problems. Process evaluation is the day-to-day monitoring of ACSM
activities and operations. Determine the objectives of a process evaluation prior to
establishing the monitoring mechanisms.
The objectives should focus on:
whether ACSM activities are on track,
how close they are to meeting the projected timeline and budget, and
whether staff members understand and perform their roles correctly.

Once objectives have been determined, select monitoring and tracking mechanisms. Some
suggested sources of information for tracking the various components of ACSM activities include:

inventory of materials;
distribution list;
activity reports;
public diaries;
television and radio logs;
media-clipping services;
staff surveys or focus groups;
partner feedback;
timeline and budget assessments;
news and information searches;
legislative tracking systems; and
attitude or household surveys.

2. Recognizing problems via feedback from the field

Responding to relevant information in real time allows a programme to improve
immediately, rather than realizing in retrospect what it should have done. The
following strategies for effectively using feedback from the field can be
implemented:
Involve key decision-makers, stakeholders and TB advocates in helping to
analyse and use feedback;
Use process evaluation to uncover problems or opportunities for the ACSM
intervention during implementation;
Conduct preliminary evaluations to identify potential improvements and highlight
and share successes before the completion of ACSM activities; and
Use summative research to make future ACSM programme decisions part of a
process rather than just an end point.

3. Making mid-course corrections based on feedback

Consider the feedback questions below when deciding whether mid-course corrections
are necessary.
Have goals and objectives shifted as activities have been conducted? If so, revise the
original goals and objectives to meet the new situation.
Are particular objectives not being met by programme or ACSM interventions? Why?
What barriers are being encountered? How can the barriers be overcome?
Has a strategy or approach met all its objectives or does it seem not to be working at
all? Consider ending that tactic.
Is there any new intervention or treatment information that should be incorporated
into the messages or design?
Which strategies or activities have not succeeded?
Review why they do not work and determine what can be done to correct any
problems.

Evaluation

1)
2)
3)
4)
5)
6)
7)

types of evaluations;
setting evaluation goals;
determining suggested indicators;
developing an evaluation plan;
selecting monitoring and evaluation methodologies;
developing and pre-testing data-collection instruments; and
writing an evaluation report.
At some point all programs need to ask the question, How effective were the
ACSM strategies? This is the time to reflect on what has been achieved, what
has worked and what has not, and to make recommendations to improve
future efforts. A summative evaluation will not only help to answer these
questions, but also to demonstrate to funders and partners the effectiveness of
activities.

1. Types of evaluations

When conducting an outcome evaluation the

limitations and constraints can include:

Outcome evaluation

limited funds;
limited staff time or expertise;
length of time allotted to the ACSM
intervention and its evaluation;
restrictions on hiring consultants or
contractors to conduct the evaluation;
policies that limit the programmes ability
to collect information from the public;
difficulty in defining or establishing
consensus on ACSM objectives;
difficulty in isolating the effects of ACSM
intervention from other influences; and
a perception by funders or others involved
in the programme that evaluations are not
valuable.

measures how well

the ACSM intervention has met its objectives and what
should be changed to improve future ACSM activities.
Follow the steps below to conduct an outcome
evaluation:
determine what information the evaluation must
provide;
define the data to collect;
decide on data collection methods;
develop and pre-test data collection instruments;
collect data;
process data;
analyse data to answer the evaluation questions;
write an evaluation report; and
disseminate the evaluation report.

Impact evaluation connects behaviour change to health or

social outcomes. An impact evaluation answers the question: Did
the people who adopted new actions or behaviour experience
improved health and well-being related to their TB status?
Impact evaluations are not often used by planners of ACSM
activities mostly because of the high costs usually involved in
carrying them out successfully. However, evaluators can select a
few programmatic objectives to measure, such as changes in the
rates of treatment completion or increases in case detection.

2. Setting evaluation goals

To set evaluation goals, first determine who wants to learn from the evaluation. This
will be primarily the stakeholders involved in ACSM activities or those with a direct
interest in the programme.
Other interested parties such as the broader community, the various levels of the
health system, people from the municipal/district/regional/national governments and
donors should also be invited to suggest evaluation goals or participate in other
ways.
There may also be people and organizations that are interested in learning from the
evaluation although they may not have been directly involved in the process.
Potential future partners might also be invited to participate, to provide an external
perspective and to further their understanding of the approach used and its results.

What are the ACSM objectives?

How long will the intervention last?

What should the members of the intended

group think, feel or do as a result of the
ACSM activities, in contrast to what they
thought, felt or did before?

What kind of changes in attitude, awareness,

behaviour or policy, for example, can be
expected in the time period? Sometimes,
when the outcomes are measured, activities
or interventions will not have been in place
long enough for objectives to have been
met.

How is change expected to occur?

Will it be slow or rapid? What measurable
intermediate outcomes steps toward the
desired behaviour are likely to take
place before the behaviour change can
occur?

Which outcome evaluation methods can

capture the scope of the change that is
likely to have occurred?
When sample sizes are small (usually due to
funding constraints), the evaluation is said
to lack statistical power and only fairly large
changes will be statistically significant.

3. Determining suggested indicators

Changes in the evaluation indicators, or key outcomes to be measured, indicate
whether objectives have been achieved. Select the indicators that identify where
ACSM initiatives have been the most successful or where additional work is
needed. Link indicators to the objectives set during the planning process. All
indicators must be measurable.

Individual-level indicators measure:

knowledge;
attitudes;
beliefs;
behaviour;
stage of willingness to change; and
demographics.

4. Developing an evaluation plan

At a minimum, an evaluation plan should answer
several key questions:
a) What are the projects objectives and
expected outcomes? What questions should
be asked?

What elements of the project worked?

What elements did not work?
What were some successes?
What were some failures?
What still remains to be done?
What is the vision for the future?
How has capacity been strengthened during the
intervention?
Which results obtained during the intervention
are likely to be sustained or improved upon?

b) What information is needed to answer the

above questions and how will the information be
collected?
List the instruments and methods that will be used to
collect the information, such as:
repeated surveys (baseline and follow-up);
stories (peak moments, peak achievements);
drawings (e.g., of project history);
role plays to present important milestones or events;
in-depth interviews with project participants and
observers;
group discussions;
picture-card pile sorts (sort interventions that worked
or did not);
rankings (rank initiatives or interventions, from those
that worked best to least); and
review of the project proposal, reports and documents.

c) Who will collect the information?

It is more cost-effective to use existing human resources and platforms, if possible, to
collect information. Ideally, staff should not collect information that directly measures the
success or failure of their own efforts.
d) What resources/materials will be needed?
List who needs support in collecting the information from different sources and what
support they need.
e) When will this information be collected?
Create a timeline for data collection.

5. Selecting the monitoring and evaluation methods

For the broadest view of areas where the ACSM programme has been
most effective and those where improvement may be needed, use
quantitative and qualitative research methods to collect evaluation data.
Qualitative methods may include in-depth interviews, focus groups
or anecdotal feedback mechanisms such as diaries and observational
studies (watching people in a natural setting without their awareness
and observing their behaviour).
Quantitative methods may include sales data, service statistics or
surveys.

6. Developing and pre-testing data collection instruments

Most outcome evaluation methods involve collecting data on participants through observation, questionnaires or participatory
methods. To develop data collection instruments or to select and adapt existing ones ask the questions:
Which data?
The data collected should be directly related to the evaluation questions. For example, if members of the intended group
need to know more about a topic before behaviour change can take place, ask knowledge-related questions in the evaluation.
From whom?
Evaluators should decide how many members of each group are required in the evaluation to measure change. There should
be adequate resources to collect information from the number of people required. Different data-collection instruments and
methods for different groups may be necessary.
How?
Assess available resources before deciding how to collect data. Are skilled interviewers accessible or can staff members be
trained? Will self-reports from participants be used? If so, do any confidentiality issues need to be addressed?
Consider also whether participants will be comfortable with the data collection-methods.
Will they be willing and able to fill out forms?
Will they share personal information with interviewers?
Will the interviews and responses need to be translated?

7. Writing an evaluation report

To prepare an evaluation report, have staff with appropriate expertise analyse the
outcome evaluation data and work closely with the evaluators to interpret the data
and develop recommendations.
An evaluation report should:
present the lessons learnt in a clear format that can be easily digested by others
who may be planning future ACSM activities;
demonstrate accountability to employers, partners and funding agencies;
provide evidence of the effectiveness of the ACSM interventions and activities; and
create a formal record to serve as an institutional memory of what has been tried,
which partners had strong skills or experience in specific areas, what problems
were encountered and what successes were achieved.

Monitoring & Evaluating

for Social and Behavioral Change Communication

Weiner, Renay.

Monitoring and evaluation plan

A monitoring and evaluation (M&E) plan is a document that outlines how
an implementation research project is monitored and evaluated, and that
links strategic information obtained from various data collection systems
to decisions about how to improve the project on an ongoing basis.
The M&E plan serves several main purposes, including:
stating how achievements of the programme/project will be measured;
documenting consensus, thereby encouraging transparency,
accountability and responsibility;
guiding implementation of M&E; and
preserving institutional memory.

Standards for an M&E plan

An effective M&E plan should conform to the following standards:
Utility: It must be useful and serve the practical and strategic information needs of the
intended users for decision-making purposes, these may range from assessing programme
performance to allocating resources, etc.
Feasibility: Be realistic and practical. Given the scarcity of resources, the M&E plan should
make the best use of existing data collection systems. However, if new data collection
systems are involved, resources (cost and technical capacity) must carefully be considered.
Ethically sound: Abide by ethical principles with regard to those involved in and affected
by the M&E activities.
Accuracy: Provide technically accurate and useful information for decision-making and
programme improvement.

Key components of an M&E plan

There are four key components that form the foundation
upon which the M&E plan should be built. Answering these
four corresponding questions is critical to M&E planning:
What does the project want to change and how?
What are the specific objectives that are designed to
achieve this change?
What are the indicators and how will they will be
measured, and,
How the M&E data will be collected and analysed?

Key steps in developing an M&E plan

Stakeholder consultation and participation
Stakeholder consultations and participation should be regular occurrences throughout the
entire process of developing and implementing your M&E plan. These consultations ensure
dialogue, a clear understanding of the project goals and objectives, and how these will be
assessed.
They also ensure that various perspectives are understood and integrated, and that
authentic needs are being met. Stakeholder participation in the design of the M&E plan
facilitates the selection of appropriate and useful M&E indicators. Furthermore, taking extra
measures to promote stakeholder participation creates a sense of ownership and
responsibility among partners.
Stakeholder involvement increases the probability that the information and results guided
by the M&E plan will be consistent with their expectations.

Developing the M&E plan

Consensus should be reached on key questions in the following areas:
What do we want to know at the end of the project?
and
What do we expect to change by the end of the project?
Again, answering the question of what you expect your project to change will
guide decisions about what strategic information is needed for project
management decisions as well as what elements should be monitored and
evaluated in order to assess progress.
The rigor and scope of your M&E plan will depend on what you commit to and
what results or outcomes your project is accountable for.

Developing the M&E plan provides your team with a clear picture of the following:
How project activities are linked to expected outputs, outcomes and population-level
impacts.
How different types of information will be collected and used by different levels of the
health system.
What elements need to be measured (e.g. resources, service statistics, coverage and
quality, costs, and outcomes associated with the project).
Appropriate indicators to be selected. To enable standardization and comparison with other
similar projects, indicators should be consistent with international/national standards. They
should also be feasible and realistic to collect. The data sources identified must provide the
information needed to measure the indicators.

Determining the M&E methodology

Once your team has developed the M&E plan, defined the indicators, and
identified the data sources necessary, the appropriate methods by which data
can be collected and analysed should be determined.
For example, your team should determine whether you will use existing data
collection systems or if new systems need to be developed.
Your team must also determine how information will be recorded, analysed and
reported.
Furthermore your team should also carefully consider the resources available
in terms of technical competencies, costs, and time when determining the
methods and tools to be used.

Assign responsibilities for

implementation
After developing the M&E plan, the roles and responsibilities of the
different stakeholders should be described clearly.
This step will determine how the M&E plan will be specifically
implemented and what reporting system will be adopted.
The implementation of the plan should include the data collection plan
(i.e. who is responsible for collection of specific data; ensuring quality
control at each stage; how often the data will be collected; format of
the data (e.g. raw, summary); what resources will be required at each
stage; who will analyse the data) and the dissemination plan.

Setting targets
Target should be set in consultation with all stakeholders so that everyone understands what the
project has committed to achieve.
By setting targets, you will have a concrete measure by which to judge whether the project is
progressing as expected.
The process of target setting, must focus on answering the question:
What can realistically be achieved given the resources and the environment in which the
project is operating?
The factors to consider include: baseline levels; past trends; expert opinions; research findings; what
has been achieved elsewhere; client expectations; and the capacity and logistics to achieve targets.
When setting specific project targets, you must also decide the direction of any potential changes
that may be indicated over time.

Defining reporting system, dissemination and

utilization of results
Here are some practical considerations in planning information reporting and utilization planning:
Design the M&E dissemination plan around the information needs of the users. It is important to be mindful that the
content and format of data reports will vary according to their intended use. For example, is the M&E required to
monitor processes? To conduct strategic planning?
To comply with requirements? Help identify problems? Justify a funding request? Or to conduct an impact evaluation?
Identify the frequency of data reporting needs. For example, project managers may want to review M&E data
frequently to assess project progress and make planning decisions, whereas donors may need data only once or twice
a year to ensure accountability.
Tailor the reporting formats to the intended audience. Since reporting may entail different levels of complexity and
technical language, the report format and media should be tailored to specific audiences and different methods used
to solicit feedback.
Identify appropriate outlets and media channels for communicating M&E data. This should consider both internal
reporting, such as regular project reports, to management and progress reports to donors, as well as external
reporting, such as public forums, news releases, briefings, and websites.

Implementing the M&E plan

Implementation of the M&E plan occurs in three stages,
namely:
checking and measuring progress;
analysing the situation; and
reacting to new events, opportunities and issues.

Checking and measuring progress

Ideally, monitoring focuses on the projects three main characteristics of
quality, time and cost.
The project manager coordinates the project team and should always be
aware of the status of the project.
When checking and measuring progress, the project manager should
communicate with all team members to find out whether planned activities
are implemented on time and within the agreed quality standards and
budget.
The achievement of milestones is measured and reflects the progress of the
project.

Analysing the situation

The second stage of monitoring consists of analysing the
situation. The status of project development is compared to
the original plan, and causes and impact of potential
deviation are identified.
Actions are identified to address these causes and the
impacts of any deviations.

Reacting to new events,

opportunities and issues
It is important to anticipate and react quickly to new
situations, events, opportunities and issues, and to identify
the possible actions to be taken.
If appropriate, various options are considered and
discussed with the project team and a decision is taken
regarding the most appropriate path to pursue.

Adjustments to/updating the M&E

plan
The M&E plan should be seen as dynamic and should always reflect the reality of what is known and
understood. Each time a deviation from the original M&E plan is identified, whether or not it requires
any further action, the M&E plan should be revised and changes documented accordingly.
Here are examples of questions that can be considered to help your team assess how well the M&E
plan is working.
Are M&E activities progressing as planned?
Are the evaluation questions being answered sufficiently?
Have new evaluation questions been raised and, if so, should they be incorporated into the M&E
plan?
Are there any methodological or evaluation design issues that need to be addressed?
Are there any outside factors (political, environment) that are affecting the M&E plan?
Are appropriate staff and funding still available to implement the M&E plan?
Are M&E findings being disseminated and used by stakeholders for decision-making and
programme improvement?

Monitoring & Evaluation of

Communication and Social Mobilization Guidelines

PMI.

Minimum Monitoring and Evaluation Standards for

Communication Activities
Following is a list of required minimum monitoring and evaluation standards for communication
activities.
Describe the process of strategy, materials, and activity development and implementation, including
methodology and when the steps happened. The following steps should be described in detail, including
how, when, how many, and target audience:
Audience research including instruments, methodology, and results
Strategy formulation
Print materials developed
Broadcast materials developed
Local communication channels
Pre-testing including instruments, methodology, and results
Production
Broadcast
Dissemination/Orientation
Training/orientation
Supervision
Monitoring including instruments, methodology, and results
Evaluation including instruments, methodology, and results

The Rapid Monitoring Checklist

The Rapid Monitoring Checklist below gives examples of questions that can be used to collect relevant monitoring information, a list
of tools to collect the information, and suggested approaches for using the tools. As mentioned earlier, this checklist can be used to
receive feedback on the interventions and to identify any problems early so that they can be addressed quickly.

References:
WHO. 2007. Advocacy Communication and Social Mobilization (ACSM) for
Tuberculosis Control. A Handbook for Country Programmes. Chapter 9
Weiner, Renay. Monitoring and evaluation for social and behavioural change
communication
PMI Communication and Social Mobilization Guidelines
(http://www.pmi.gov/resources/publications/communication_social_mobilization_g
uidelines.pdf.)