TA 4581: Developing a National Result-Based M&E System For Key Projects

Overview: Ten Steps to a Results-Based Monitoring & Evaluation System

March 2007 20073

Ten Steps to a Results-Based Monitoring & Evaluation System

Select Indicators to Monitor Results Plan for Improvement Select Results Targets

Conduct a Readiness Assessment

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

From Ten Steps to a Results-Based Monitoring & Evaluation System: A Handbook for Development Practitioners, Kusek & Rist, The World Bank, 2004.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Foundation step for results-based monitoring and evaluation system. Can be considered at the country, sector, province, local, program or project level. Key questions asked in the readiness assessment:
1. 2. 3. 4. 5. 6. 7. 8. What demand is there for a results-based M&E system? Is there an advocate or champion for it? What is motivating the advocate? Who will own the system and who will benefit from it? How much information do they need? How will the system directly support better resource allocation and achievement of program goals? How will the organization, the champions and the staff react to negative information? Where does capacity exist to support a results-based M&E system? How will the M&E system link Project, Program, Sector and National Goals?

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Consider strategic priorities and desired outcomes Process:

Identify specific stakeholder representatives Identify major concerns of stakeholder groups Translate problems into statements of possible outcome improvements Disaggregate to capture key desired outcome Develop a plan to assess how a government or organization will achieve these outcomes

More about this later in this training

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Indicators are required for all levels of resultsbased monitoring and evaluation systems. Results have to be translated into their correct indicators. More, later in this training.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

The baseline is the information that shows the starting point. It is needed so we can understand what change has been brought about by the project or program. Data sources must be established and initial data gathered. More about this later in this training.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Targets are the quantifiable level of performance that we want to achieve, by a given time. More about this later in this training.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Monitoring begins as soon as the project does. Results-based monitoring is of outputs and outcomes. More about this, later in this training.

Monitoring
Monitoring
Systematic collection of data to provide stakeholders with indications of progress towards project or program objectives and the use of allocated funds.

Monitoring
Designed to provide information on project or program implementation and immediate administrative and management issues. Uses project or program plan and design data as starting point. Systematic reporting on provision of inputs. Systematic reporting on production of outputs. May also collect data on outcomes. Directly linked to a distinct project or program, using quantitative information.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Evaluation is used to:

Help make resource allocation decisions Help re-think the causes of a problem Identify emerging problems Support decision making on competing alternatives Build consensus on causes of problems and how to respond.

Different types of Evaluation are used to answer different questions. More on this later in this training.

Evaluation
Evaluation
The systematic and objective assessment of an in-progress or completed project or program. Includes assessment of its design, implementation and results.

Evaluation
Designed to provide strategic level information to high level decision makers. Uses strategic plan and baseline data on pre-project or pre-program situation as its starting point. Collects data on indicators for outcomes. May collect data on impacts. Systematic reporting on progress towards achieving outcomes. Reporting on outputs focuses on how they have contributed to outcomes. Uses both quantitative and qualitative information and can assess multiple projects and programs.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Monitoring and evaluation reports can be used:

To demonstrate accountabilitydelivering on political promises to citizens and other stakeholders To convinceusing evidence from findings To educatereporting findings to help organizational learning To explore and investigateseeing what works, what does not, and why To documentrecording and creating an institutional memory To involveengaging stakeholders through a participatory process To gain supportdemonstrating results to help gain support among stakeholders To promote understandingreporting results to enhance understanding of projects, programs, and policies.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Findings of Results-based Monitoring and Evaluation systems can be used to:

1. 2. 3. 4. Respond to elected officials and the publics demands for accountability Help formulate and justify budget requests Help make operational resource allocation decisions Trigger in-depth examinations of what performance problems exist and what corrections are needed 5. Help motivate personnel to continue making program improvements 6. Formulate and monitor the performance of contractors and grantees 7. Provide data for special, in-depth program evaluations 8. Help provide services more efficiently 9. Support strategic and other long-term planning efforts (by providing baseline information and later tracking progress) 10. Communicate better with the public to build public trust.

Conduct a Readiness Assessment

Select Indicators to Monitor Results

Plan for Improvement Select Results Targets

Use Evaluation Information

Use Findings

2
Agree Results to Monitor & Evaluate

4
Measure baseline data

6
Monitor Results

8
Report Findings

10
Sustain the Monitoring & Evaluation System

Key factors for sustaining a Results-Based Monitoring and Evaluation System are:
Continuing demand Clear roles and responsibilities Reliable and credible information Accountability Capacity Incentives

International Practices
Monitoring and Evaluation is done at
1. National/sector level 2. Program level 3. Project level

Variety of approaches, according to local circumstances, especially level of centralization/decentralization and approach to financial controls

China Monitoring and Evaluation National level

Began with mid-term review of the 10th FiveYear Plan. Government has made M&E a priority. Prime Minister Wen Jiabao called on government agencies to establish accountability system and undertake M&E activities. 11th Five-Year Plan contained an M&E indicator matrix with 22 indicators 8 with quantitative targets, 14 with benchmarks. Used the Ten Steps Model, with help of World Bank.

China Monitoring and Evaluation National level

Five-balances as guiding principle. 3 to 4 goals for each Balance. About 80 candidate monitoring indicators. Future:
refining and finalizing the indicators; deciding on targets and benchmarks; institutional set-ups organizational structure, legal framework etc. mid-course evaluation of 11th FYP.

International Practices National/sector level

Some countries have national coordinating departments for Monitoring and Evaluation at national level, looking at high-level development goals:
Malaysia Implementation Coordination Unit (in Prime Ministers Department), coordination across national, state and district development councils;

Many countries such as USA, UK, Australia, use oversight agencies (such as National Audit Office and state Auditors-General) to monitor and evaluate not just financial probity but also to do performance reviews, through national and provincial audit offices.

International Practices Program and Project levels

Program evaluations are common in OECD countries public services, as are major project evaluations. Many government departments at both national and provincial level have their own internal monitoring and evaluation functions. Major reviews are carried out independently, and are commissioned by the relevant government departments. National governments may encourage monitoring and evaluation through regulations or laws mandating reviews as a condition of funding.

International Practices Program and Project level Case example

Victorian State Government (Australia) Gateway process:

coordinated by an inter-department working group; reviews projects at any of six Gates throughout project life:
1. 2. 3. 4. 5. 6. Strategic Assessment Business Case Procurement Strategy Tender Decision Readiness for Service Benefits Evaluation projects rated high/medium risk on project profile model; projects nominated by Expenditure Review Committee as part of a budget process; projects nominated by any sub-committee of Cabinet.

Criteria for review:

Review done by an independent Review Team. Stakeholders include nominated Senior Responsible Owner and Project Manager. 2003-2006: 115 reviews of >90 projects, total worth > $14 billion.

International Practices Program and Project level

Cracking and pothole repairs

Rutting

Heavy patching

Edge break The Audit Office of NSW 2006.

International Practices Program and Project level

International Practices Program and Project level

END