NUMBER 2

VERSION 1.0

NOV 2012

HOW-TO NOTE
Preparing Mission Orders on Evaluation
Monitoring and Evaluation Series

This How-To Note provides additional guidance for Mission Orders on Evaluation

INTRODUCTION
Mission Orders (MOs) are written instructions that are required when additional Mission-specific procedures are necessary to implement USAID policy. Mission orders do not create new policy and do not duplicate or contradict existing policy, but do cite the relevant USAID policy and the corresponding ADS chapter as the appropriate authority. The objective of this document is to provide Missions with guidance in developing Mission Orders on Evaluation (MOE), and to ensure that they address the relevant issues. ADS 203 requires each Mission to prepare or update its Mission Order on Evaluation (MOE) addressing the contextspecific approaches and expectations regarding evaluation for its Mission. Missions may elect to create stand-alone MOEs. However, given the linkages between monitoring and evaluation, missions may also choose to revise existing MOs on performance monitoring to include evaluation guidance, or create new MOs that cover both topics, as long as there is sufficient distinction between the two functions regarding roles, responsibilities and procedures.

CONTENTS OF A MISSION ORDER ON EVALUATION How-To Notes

are published by the Bureau of Policy, Planning and Learning and provide guidelines and practical advice to USAID staff and partners related to the Program Cycle. This How-To Note supplements USAID ADS Chapter 203 Every Mission Order, including Mission Orders on Evaluation, should contain the following sections distinguished by a standardized numbering system: I. Purpose II. Authority III. Definitions IV. Responsibility V. Procedures VI. Effective Date Core content and issues for consideration for each section is provided below. MOEs do not need to restate the ADS or the Evaluation Policy.

www.usaid.gov

HOW-TO NOTE
I. Purpose: This section of the MOE should define the desired results or effect of the Mission Order. It should also include any "context-specific" information that may influence the content of the MOE (e.g. support of country monitoring and evaluation systems). In general, it should be short. To provide Mission guidelines, procedures and recommendations related to project and program evaluations, should be sufficient for MOEs in many cases. II. Authority: This section should cite or reference the relevant authority upon which the Mission Order is based. For Mission Orders on Evaluation, the Evaluation Policy, and the relevant section of the ADS regarding evaluations, should be cited. For example ADS 200.2, 203.3.2 and USAID Evaluation Policy of January 19, 2011. III. Definitions: All specialized and technical evaluation terms for which a thorough understanding is required to carry out the direction and implement the processes contained within the Mission Order should be defined. These may include definitions of terms such as evaluation, impact evaluation, performance evaluation, assessment, performance monitoring, etc. When available, definitions in the ADS should be used. IV. Responsibility: This section outlines specific duties of Mission staff at all levels as they relate to the operationalization and management of evaluation activities within the Mission. USAID policy specifically assigns some evaluation responsibilities to the evaluation point of contact (POC), to mission leadership, to program offices, and technical offices. This section should be consistent with those assignments, which are outlined below as required. This section also provides the opportunity for the mission to assign responsibilities that are not assigned in the

Preparing Preparing Evaluation Evaluation Reports Reports

Evaluation Policy or to specify particular responsibilities within the relevant offices. In general, personnel in the Missions program office are expected to have greater responsibility over evaluations than prior to the Evaluation Policy. Below is a list of key evaluation roles, including required and recommended responsibilities. Mission Leadership Required Responsibilities include: a) Approving exceptions to the Evaluation Policy allowing an office other than the program office to manage evaluation mechanisms; b) Approving exceptions to the Evaluation Policy for ongoing projects with limited flexibility only, allowing an implementing partner to subcontract an impact or performance evaluation due to funds not being set aside during project design for an external evaluation; c) Approving exceptions to the Evaluation Policy (with required PPL concurrence) to not sharing evaluation findings and reports within three months of evaluation completion by submitting them to the Development Experience Clearinghouse; d) Approving exceptions to the Evaluation Policy (with required PPL concurrence) to not having an external evaluation team leader; e) Assigning the evaluation point of contact and alternate. Missions may choose to assign other evaluation responsibilities to mission leadership. For instance, the MOE may designate mission leadership the responsibility to approve the mission evaluation plan or to approve finalization of a draft evaluation report.

HOW-TO NOTE
The Evaluation Point of Contact (POC) The Evaluation POC is the only individual role identified in the Evaluation Policy. The POC typically will be a senior program officer in that units program office. Required Responsibilities include: a) Ensuring Mission compliance with the Evaluation Policy across the breadth of the Missions projects; b) Interacting with regional or technical bureau points of contact and the Bureau of Policy, Planning, and Learning, Office of Learning, Evaluation, and Research (LER). Other responsibilities of the evaluation point of contact may include those listed under the program office responsibilities. The Program Office Table 1 illustrates the required evaluation roles and responsibilities of USAID program offices as described in ADS 203. Missions may choose to assign these responsibilities in the MOE among program office staff as they see fit. In doing so, it is particularly helpful to note which, if any, of these responsibilities will be assigned to the evaluation point of contact. In some missions, the program office may also include an evaluation specialist, or similarly titled staff position, in addition to a more senior program officer who serves as the evaluation point of contact. A program office evaluation specialist is someone with the technical knowledge and expertise to execute evaluation responsibilities. If the program office includes such a staff position, then the MOE should note which of the program office responsibilities are assigned to that individual. Other evaluation responsibilities to consider assigning to program office staff in the MOE include: a) Ensuring that evaluation final reports (or final drafts) and their summaries are submitted within three months of completion to the DEC;

Preparing Evaluation Reports

b) Integrating evaluation findings into decision making about strategies, program priorities, and project design, for instance, by ensuring that evaluation evidence is cited in the CDCS, and that portfolio reviews consider evaluation findings; c) Making the final determination regarding which of the Missions projects are required to be evaluated under the large project and innovative project requirements; d) Ensuring that evaluation teams do not have team members with a conflict of interest that would impair the independence of the evaluation report; e) Making available all relevant information for technical audits of evaluation practices; f) Participating in an agency wide process of developing an evaluation agenda. The Technical Offices Table 1 also illustrates the required evaluation roles and responsibilities of USAID technical offices as described in ADS 203. Missions may choose to assign these responsibilities among technical office staff in the MOE as they see fit. Other evaluation responsibilities that missions may want to assign to technical offices in the MOE include: a) Initiating or leading the drafting of Statements of Work for evaluations within their purview prior to handing off to program offices; b) Providing necessary input and logistical support to evaluation teams during field work to facilitate information gathering; c) Communicating evaluation findings to implementing partners; d) Ensuring that implementing partners collect and maintain relevant monitoring data and documentation that can be accessed for future evaluations; e) Integrating evaluation recommendations into decision making about strategies, program priorities, and project design in cooperation with the Program Office. 3

HOW-TO NOTE TABLE 1: EVALUATION ROLES AND RESPONSIBILITIES

(from ADS 203)

Preparing Evaluation Reports

Program Offices Leadership Training & Learning

Identify an evaluation point of contact

Technical Offices

Invest in training of key staff Actively encourage staff to participate in evaluation community of practice

Planning

Ensure planning for evaluation questions in context of CDCS development Ensure adequacy of Evaluation section of Mission portfolio wide PMP Ensure M&E Plans are incorporated into Project Designs Develop a budget estimate for evaluations Allocate program funds for external evaluations (Goal: three percent of USAID Mission/Offices total program budget) Provide relevant technical support to development of evaluation questions, PMPs and M&E plans

Evaluation Scopes of Work and Evaluation Reports

Ensure that final scopes of work for external evaluations adhere to standards in Section 4 of the Evaluation Policy Manage, in most cases, required external evaluations Organize in-house peer technical reviews to assess quality of evaluation SOWs and draft reports

Provide relevant technical support to ensure that SOWs address standards of the Evaluation Policy

Participate in peer technical reviews

Evaluation Technical Support

Develop contractual mechanisms to access evaluation expertise and support Prepare a Mission Order on evaluation describing context-specific approaches

Reporting & Knowledge Management

Include evaluation reporting and plans in the Performance Plan and Report annex on evaluation Warehouse evaluation data

HOW-TO NOTE
V. Procedures: This section of the Mission Order on Evaluation should consist of the Mission-specific procedures that support ADS 203 and the Evaluation Policy. If the procedures described in the ADS are sufficient, then mission-specific implementing procedures are not permitted. The goal in this section is to minimize internal regulations and increase efficiency while clarifying how the Mission will comply with Agency policy. In writing the Procedures section of the MOE, the relevant sections of the Evaluation Policy to consider are Section 4, Evaluation Practices, pp. 6 -7, and Section 5, Evaluation Requirements, pp. 811. At minimum, the evaluation mission order should contain the mission-specific processes to: a) Identify the projects that will be evaluated. The Evaluation Policy requires large projects and innovative development innovations to be evaluated. The MOE should describe the mission procedure for determining whether a project is large or not and whether a project includes an innovative development intervention. It should also include the process for determining when evaluations are to be conducted of projects that are not required to be evaluated according to agency policy. b) Review scopes of work and draft evaluation reports. Program offices are required to organize peer reviews of evaluation scopes of work and draft evaluation reports. The MOE should describe the procedures for conducting the peer reviews, including, for instance, any requirements on documentation of the peer review process, number and composition of the peer review team, and expectations on the time required to complete the peer review.

Preparing Evaluation Reports

Other mission specific procedures that missions may choose to address in the MOE include: c) Procedures for determining and documenting requests for exemptions to the Evaluation Policy prior to clearance by mission leadership; d) Procedures for planning for evaluations over the life of the CDCS, including what information is to be included in the Evaluation Plan section of the mission performance management plan; e) Procedures for developing and finalizing evaluation SoWs; f) Procedures and exceptions for sharing evaluation designs with country-level stakeholders and implementing partners; g) Procedures for sharing draft evaluation reports with other funders and implementing partners and incorporating statements of difference from USAID or other funders, evaluation team members, or implementing partners, when applicable; h) Procedures for sharing Evaluation Reports widely and in an accessible form with all partners and stakeholders, and with the general public; i) Procedures for responding to and utilizing evaluation findings. VI. Effective Date: This section should identify the date when the Mission Order will become effective. Mission Orders are effective starting on the date the order is signed by the Mission Director.

HOW-TO NOTE

Preparing Evaluation Reports

ADDITIONAL RESOURCES
The following resources provide more information. Some other resources exist but are out-of-date with current USAID guidance. Where information differs, the USAID Evaluation Policy and the USAID ADS (Automated Directives System) take precedence over that in other resources.

Guidance on Preparing Mission Orders (10/01/2003), as referenced in ADS 527. (http:// transition.usaid.gov/policy/ads/500/527sab.pdf). USAID ADS 203 (http://transition.usaid.gov/policy/ads/200/203.pdf). USAID Evaluation Policy, January 2011. (http://pdf.usaid.gov/pdf_docs/PDACQ800.pdf). USAID Evaluation Policy: Answers to Frequently Asked Questions (FAQs) (http://transition.usaid.gov/policy/ evalweb/USAID_Evaluation_Policy_FAQ.pdf).