DRAFT REPORT:19 09 05

ACCOUNTABILITY AND LEARNING IN

THE CO-FINANCING PROGRAMME

The contribution of monitoring and evaluation

MINISTRY OF FOREIGN AFFAIRS, IOB SEPTEMBER 2005

ACCOUNTABILITY AND LEARNING

IN THE CO-FINANCING PROGRAMME:

The contribution of monitoring and evaluation

TABLE OF CONTENTS
1. Main findings and issues 1.1. Main findings 1.2. Issues 2. The Co-Financing Programme 2.1. Background 2.2. Policies and strategies of the Ministry 2.3. The co-financing organisations 2.4. Policies of co-financing organisations 3. Study objectives and approach 3.1. Justification for the study 3.2. Objective of the evaluation and evaluation questions 3.3. Scope and delimitation 3.4. Assessment criteria for monitoring and evaluation 3.5. Approach and methods 3.6. Readers guide 4. Overview of the co-financing aid chain 4.1. Levels in the CFP aid chain 4.2. Relationships between organisations at different levels 5. Monitoring end evaluation in partner organisations 5.1. Overview of partner organisations studied 5.2. Organisation of monitoring and evaluation 5.3. Monitoring instruments 5.3.1. Reporting 5.3.2. Other monitoring instruments 5.4. Evaluation 5.4.1. External evaluation 5.4.2. Internal evaluation 5.5. Organisational feedback systems 5.6. Contributions of CFOs to strengthening monitoring and evaluation in partner organisations 5.7. Use of the information by partner organisations 5.8. Analysis 6. Monitoring and evaluation by CFOs 6.1. Organisation of monitoring and evaluation 6.2. Internal monitoring 6.3. Monitoring of partner organisations 6.3.1. Reporting by partner organisations 6.3.2. Other forms of monitoring 5 5 8 9 9 10 11 12 15 15 16 16 17 18 19 21 21 22 29 29 30 32 32 35 37 37 37 39 42 43 46 51 51 52 53 54 55

6.4. Evaluation 6.4.1. Activity-level evaluations 6.4.2. Programme evaluations 6.5. Aggregation of activity-level results 6.6. Use of feedback information 6.6.1. Accountability 6.6.2. Management decision making 6.6.3. Learning 6.7. Recent developments 6.8. Analysis 7. Monitoring and evaluation by DSI/MY 7.1. DSI/MY: role and responsibilities 7.2. Monitoring systems and instruments 7.2.1. Annual plans and annual reports 7.2.2. Specific requests for information 7.2.3. Regular meetings 7.2.4. Field trips by staff of DSI/MY 7.2.5. Meetings between CFOs and Embassies 7.2.6. Other exchange of information 7.3. Evaluation 7.4. Use of feedback by DSI/MY 7.5. Development of indicators 7.6. Recent developments 7.7. Analysis

59 59 62 62 63 63 64 66 66 68 71 71 71 72 74 75 76 76 77 77 78 79 80 80

APPENDICES:
1. 2. 3. 4. 5. 6. 7. 8. The Policy and Operations Evaluation Department Terms of reference of the study Abbreviations Glossary Partner organisations Co-financing organisations The Ministry (DSI/MY) Relevant literature 83 87 97 99 101 107 135 143

1.

MAIN FINDINGS AND ISSUES

The Co-Financing Programme (CFP) started in the 1960s and has grown into one of the most important aid channels of the Dutch development cooperation effort. Under the 2003-2006 policy, 6 organisations1 receive support through the CFP. The aim of this study is to assess to what extent monitoring and evaluation systems are in place that permit organisations in the CFP aid chain to manage their activities in a more result focused way. Results are achieved at different levels in the CFP aid chain. For the purpose of this study, the results relevant for management are considered to be those achieved at the level of the partner organisations and those achieved at the level of the target groups. For the evaluation all co-financing organisations and the relevant unit of the Ministry are studied. In three selected countries (India, Peru and Tanzania) approximately 10% of the partner organisations are selected for study. The findings of the studies of the partner organisations show a high degree of homogeneity across types and sizes of organisations. Despite the fact that findings are not formally representative, there are no reasons to expect that they do not present a realistic description of monitoring and evaluation in partner organisations. 1.1 A. Main findings PARTNER
ORGANISATIONS HAVE ESTABLISHED MOSTLY ADEQUATE SYSTEMS FOR

MONITORING AND EVALUATION TO PROVIDE INFORMATION FOR THEIR OWN MANAGEMENT USE

(ACCOUNTABILITY, DECISION MAKING AND LEARNING). The monitoring and evaluation systems of partner organisations, comprise a variety of structured and unstructured instruments that generate a considerable amount of information. This information concerns the progress of the implementation of activities, the direct outputs that are achieved, as well as the results that are achieved at the level of the target group. At the operational level, limited reporting and analytical skills regularly constitue a constraint for reporting. Partner organisations use diverse instruments (field visits, other modes of communication) to correct these shortcomings. With regard to information about effects and impact at the level of the target group, a distinction can be made between organisations that regularly conduct impact assessments and surveys, and organisations that collect such information in a less structured, but systematic, way. Although much of this information is unstructured, it contributes to a profound understanding of what happens at the level of the target group and what the activities of the partner organisations mean for them. Unstructured and non-written information are acceptable in the culture of many organisations that operate in developing countries, specifically at grass-root level. Overall, the information available and its scope are suitable for the three purposes of information use: management decision making, specially at the operational level, accountability (both internal accountability to Boards and/or General Assemblies of partner organisations and external accountability to donors), and learning from operational practice. Most partner organisations consider self- and internal evaluations more useful for learning than external evaluations. Partner organisations have institutionalised a series of working routines to make use of the information for their own management and control purposes. Monitoring and
1

The organisations are Cordaid, Hivos, ICCO, Novib, Plan Netherlands and Terre des Hommes.

evaluation feed information into these routines: regular progress meetings for operational management decision making based on monitoring information, as well as internal and external reporting for accountability towards higher levels of management in the partner organisations and external stakeholders. Monitoring and evaluation are frequently well interrelated to planning. Learning is institutionalised usually in the form of evaluation cum planning workshops where experiences from the past period are presented and discussed and incorporated in future plans, or by means of policy/strategy discussions and decision making in General Assemblies of Boards of the organisations. B.
MOSTLY

MONITORING
ADEQUATE

AND ACTIVITY-LEVEL EVALUATION BY CO-FINANCING ORGANISATIONS ARE FOR OPERATIONAL CONTRACT MANAGEMENT OF INDIVIDUAL PARTNER

ORGANISATIONS (ACCOUNTABILITY AND OPERATIONAL DECISION MAKING), LESS FOR MANAGEMENT AT STRATEGIC AND POLICY LEVEL (POLICY DECISION MAKING, LEARNING).

The monitoring and evaluation systems developed over time by the CFOs are geared towards the management of the contractual relationship with the individual partner organisations. It appears to be difficult to relate information concerning the execution and achievements of individual activities to strategic and policy levels. Two constraints are identified: Problems associated with the aggregation of the information about the results achieved (outputs as well as effects/impact) from individual partner organisations. The relevance of dis-aggregated information about results for strategic and policy level management is limited. Much of the information that CFOs obtain from the partner organisations concerns operational aspects and direct outputs, less the effects and impact that are achieved at the level of the partner organisation and the target group. With the available monitoring and evaluation instruments it appears to be difficult to capture and communicate the wealth of unstructured information available among staff of the partner organisation about the results at the level of the target groups. Monitoring and evaluation systems that are specifically suitable for supervision and management of large numbers of independent activities are not (yet) used systematically. Partner consultations and programme evaluations are more suited to exchange or provide information at strategic and policy level. The extent to which the contributions of partner organisations to partner consultations is based on information from monitoring and evaluation, however, could not be assessed. Programme evaluations could provide information at the strategic and policy level if they include structured collection of data about the results achieved. CFOs consider programme evaluations specifically relevant for learning. The assessment of the scope and quality of programme evaluations is done separately by IOB.

C.

THE MINISTRY HAS NOT DEVELOPED A COHERENT PERCEPTION OF ITS ROLE AND TASKS IN CFP THAT CONSTITUTES A BASIS TO DEFINE THE INFORMATION IT REQUIRES FROM MONITORING AND EVALUATION. MONITORING AND EVALUATION FOCUS ON THE PROPER EXECUTION OF THE SUBSIDY AGREEMENTS WITH THE CFOS (ACCOUNTABILITY AND MANAGEMENT DECISION
THE

MAKING).

IT IS NOT GEARED TO SUPPORT MANAGEMENT (POLICY LEVEL DECISION MAKING AND LEARNING).

AT STRATEGIC AND POLICY LEVELS

The feedback information about the execution of CFP supported activities presented in the annual progress report focuses on operational issues (use of funds, numbers of agreements), less on policy level issues, that can be directly related to the overall policy objectives and strategies of the CFP policy framework. In the CFO annual reports, most information about direct outputs, effects and impact on partner organisations and target group is presented as examples. The scope of the information received by DSI/MY is suitable for management of the subsidy agreements with the co-financing organisations, not for policy management. Two issues are identified: The Ministry experiences problems to aggregate the information from the various CFOs to the level of the CFP. Each of the CFOs uses its own sets of priorities, classifications and definitions for policies and strategies. The Ministry also does not make full use of the monitoring and evaluation instruments at its disposal. Visits to the field are relatively rare. The pilot to foster operational co-ordination between embassies and CFOs has not evolved beyond a rather limited exchange of information. DSI/MY does not sufficiently exploit the information available in activity level and programme evaluations. D. INFORMATION
ABOUT RESULTS AT THE LEVEL OF PARTNER ORGANISATIONS AND THE

TARGET GROUPS DOES NOT REACH THE HIGHER LEVEL ORGANISATIONS IN THE

CFP

AID CHAIN.

MANAGEMENT

AT STRATEGIC AND POLICY LEVELS CAN THEREFORE NOT BE SUFFICIENTLY BASED

ON THE RESULTS ACHIEVED.

Communication of information about the achievement of results at the level of the partner organisations and at the level of the target group by monitoring and activitylevel evaluation between levels in the aid chain is insufficient. Management of CFOs and DSI/MY do not have sufficient information about the achievement of results to relate to management at strategic and policy levels. The monitoring and evaluation systems that link the CFOs to their partner organisations and that link the DSI/MY to the CFOs show a number of deficiencies: The systems are not suitable to process and communicate information about results achieved at the level of the partner organisations and target groups. The aggregation of information at two levels, from partner organisations to CFOs and from CFOs to DSI/MY, proves to be problematic. Monitoring instruments that are specifically suitable to supervise large portfolios of individual activities are not (yet) fully operational. The overall mix of monitoring and evaluation instruments emphasises monitoring and activity level evaluation. Programme evaluation at strategy and policy levels is less developed. Although organisations at the different levels in the aid chain make considerable efforts to some of the issues mentioned above, the interconnectedness of the systems at the different levels is insufficient. 1.2 Issues

To enhance monitoring and evaluation in the CFP aid chain, the following issues need to be addressed:

A supervisory concept A coherent supervisory concept is needed to define the roles and tasks of the organisations of the different levels in the aid chain. On this basis a clearer definition of the information expected from monitoring and evaluation can be by developed. This may help the process of selection of information, and help the Ministry to be more selective in its information demands to the CFOs and the CFOs to be more selective in their information demands to partner organisations. This will contribute to avoid waste of resources for the collection and communication of unnecessary information and will enhance the quality of the information provided. This supervisory concept should take into consideration the defining characteristics and constraints of the co-financing programme: the autonomy of the co-financing organisations and their partner organisations, the activity based nature of the programme, embedded in partnerships with a longer time horizon as well as the relatively small scale of activities supported under the co-financing programme. The Ministry should take the lead in developing such this supervisory concept. The mix of evaluation instruments A clearer perception of the supervisory concept should have direct consequences for the evaluations executed by the co-financing organisations. At present considerable effort and resources are used for activity-level evaluations, that generate mainly operational information, useful for accountability, but of limited relevance for learning and management at higher levels in the aid chain. Focussing activity-level evaluations more on results will enhance their contribution to analysis at strategic and policy levels. Execution of more programme evaluations will strengthen the provision of relevant information for the CFOs and the Ministry. The capacity for programme supervision of DSI/MY Earlier studies already indicated that information about results achieved is inadequate. Although some follow-up is given to these recommendations, the present evaluation shows that the situation has not fundamentally changed. The capability of DSI/MY to address these problems and the capacity to properly supervise the CFP have certainly contributed to the lack of progress. A clearer definition of the responsibilities and tasks of DSI/MY provides the a basis for reconsideration of the necessary capabilities and capacities for this unit.

2.
2.1

THE CO-FINANCING PROGRAMME

Background

The Co-Financing Programme (CFP) started in the 1960s, when church related organisations (the present ICCO and Cordaid) requested Government funding for development work. The first allocation for support to non-governmental development projects was made in 1965 (Hal 5 million). In 1968 and 1976 two non-confessional organisations were also admitted to the CFP (Nova and Hives respectively). Essentially this system remained in place for almost 25 years, until 2000, when Plan Netherlands was admitted to the CFP, followed in 2003 by Terre des Homes (The). Initially, funding was project based. The Ministry had to approve each individual project. In 1980, this changed to programme based funding, with a number of policy based conditionalities for the selection of projects. Decision making about projects was left to the CFOs. The modality of programme based funding also included a requirement of regular evaluation. Project and programme evaluations became increasingly frequent. From its beginning, the CFP has enjoyed considerable interest from political parties and Parliament, initially mainly the confessional parties, later also others. The solid and almost unanimous political support for the Co-financing Programme in Parliament lasted, in general terms, till the end of the nineties. Over the years, funding for the CFP increased. When programme-funding was introduced in 1980, 6% of the annual ODAbudget was earmarked for the CFP. This increased to 11% of total ODA-budget in 2003. Nominal budgets increased substantially, as is shown in figure 2.1. Figure 2.1: Evolution of the CFP budget (million )

B udget C -fin. o
4 50 4 00 3 50 3 00 2 50 2 00 1 50 1 00 50 0 16 95 1 75 9 18 90

P rogr .(M U ) il.E R

1 0 99

1 6 99

20 00

2 03 0

Source: Final Report of the Steering Committee evaluation co-financing programme (2002) and Ministry of Foreign Affairs.

In recent years, development co-operation through civil society organisations has become the second largest aid channel (in terms of funding), as is shown in the following table. The CFP is the most important of the programmes included in this category.

Table 2.1: Funding through the different main aid channels (million ). Category 2002 2003 Bilateral development co-operation 1.114 876 Multilateral organisations 899 834 Civil society organisations 627 740 Private sector 137 202 Debt reduction 285 540

2004 691 875 745 208 507

Source: P. Hoebink, van klippen en kapen: de Nederlandse ontwikkelingssamenwerking onder Agnes van Aardenne Internationale Spectator, May 2004 LVIII nr.5.

2.2

Policies and strategies of the Ministry

The 2000 interdepartmental policy review (IBO) of the co-financing programme concluded, among others, that the Ministry had not formulated a clear policy for the CoFinancing Programme. In 2001 the CFP policy 2003-2006 was formulated in consultation with the CFOs2. Under the new policy all Co-Financing Organisations (CFOs) submit a funding request every four years comprising not only objectives and strategy, but also indicators to measure results. The Minister allocates budgets to the individual CFOs after prior assessment of their individual subsidy request including strategic management plans. This marks to some extent the end of the programme funding mode in place since 1980. Key elements of the policy are summarised in the following box. Box 2.1: Key elements of the CFP-policy 2003-2006
Overall objective The overall objective of the CFP is to contribute to structural poverty alleviation in developing countries and countries in Central and Eastern Europe, and to contribute to the realisation of internationally recognised human rights. Strategies The objectives will be achieved following three intervention strategies: direct poverty alleviation, strengthening civil society and lobby and advocacy. Programme management The CFP is a self-governing programme within the general policy and administrative framework. The participating organisations fulfil a number of key criteria. Important for this study is that participating organisations should be ISO-certified, adhere to principles of good donorship, are effective, efficient and operate with a high degree of professionalism. To obtain a contribution from the CFP, participating organisations will formulate a plan for the period 2003-2006 and annual plans. The CFOs should develop a quality management system (in co-operation with the Ministry and other stakeholders). This quality management system should include the following elements: The CFOs are responsible for their internal processes and the results achieved, and report annually to the Ministry following agreed reporting guidelines (narrative and financial). The CFOs are responsible for their own systems for multi-stakeholder accountability. The CFOs have an evaluation system with evaluations conducted at different levels (partner organisations, CFOs and their interrelationships). With regard to the organisations in the target countries, the policy stipulates the following: The primary role of the organisations is to stimulate that poor people can obtain education, can organise themselves to participate in decision making processes to obtain basic services. Organisations should be diverse with roots in different political or social groupings Organisations should be capable of building mutual relationships and alliances with other stakeholders. The role of the CFOs is to identify such organisations and assist them in fulfilling their role. Source: Policy Framework CFP (December 2001)

In the 2003-2006 policy document, policies and strategies are dealt with in very general terms, but it extensively describes the distribution of responsibilities between the
2

Published in December 2001.

10

Ministry and the CFOs. The autonomy of the CFOs is explicitly recognised. The Ministry supervises the programme in general terms, ex-ante through appraisal and approval of the strategic management plans and the annual plans, ex-post by using different monitoring and evaluation instruments. Every 4 years, the Minister will appoint an external committee that advises what organisations qualify for CFP funding and the distribution of the available budget among the participating CFOs. 2.3 The Co-Financing Organisations

Six organisations presently receive funds through the CFP: Cordaid, Hivos, ICCO, Novib, Plan Netherlands (Plan NL) and Terre des Hommes (TdH). In 2003 the distribution of the CFP funds over these organisations was as follows. Figure 2.2: Distribution of CFP funds over co-financing organisations (million )

D istrib tio b d e 2 0 u n ugt 0 3

10 2 10 0 8 0 6 0 4 0 CORDAID NOVIB HIVOS 2 0 ICCO 0 PLAN TdH

Source: Ministry of Foreign Affairs

For each of the CFOs the CFP constitutes an important source of funding, specifically for the four largest recipients. Table 2.2: Proportion of CFP-funds in total budgets of CFOs (percentages).

CFO
Cordaid Hivos ICCO Novib Plan NL Terre des Hommes
3

1990 69 76 66 41 -

1995 88 82 86 59 -

2000 59 88 79 67 4 -

2003 67 91 83 71 34 24

Source: Working Document Steering Committee evaluation co-financing programme and KPMG efficiency audit (2004).

The rapid growth of CFP funding resulted in an increase of the number of partner organisations (POs).

Table 2.3: Number of partner organisations per CFO

3

Due to the merger of Bilance with Caritas Neerlandica and Vastenaktie, CORDAIDs other income increased as well. Consequently the dependence on CFP funding decreased.

11

1995 1999 2003

Cordaid 820 1.288 1.252 4

ICCO 850 839 807 5

NOVIB 717 870 839

HIVOS 550 804 864

Plan NL 446

TdH 255

2.937 3.801 4.061

Sources: Final report Steering Committee evaluation co-financing programme, 2002, and Annual Reports 2003

In the early 1990s, the development debate focused increasingly on quality and efficiency. The CFOs commissioned an Impact Assessment in an attempt to prove that their interventions did make a difference. The outcome of the study was more critical than expected and caused a review of policies and strategies. It resulted in a more business-like approach and further professionalisation of the CFOs. 2.4 Policies of Co-financing Organisations

In the strategic management plans the CFOs subscribe to the general objectives of the CFP policy framework and the three intervention strategies defined. On this basis each of the CFOs has subsequently developed its own sets of policies and priority themes. These are summarised in the following table.
Table 2.4: Overview of policy priorities of co-financing organisations Organisation Summary of policies and priority themes
Cordaid Hivos Five priority themes: urban livelihoods, access to markets, health care, peace and security and HIV/Aids. Cordaid also extends emergency assistance in case of natural or man made emergencies Five sectors: economy, arts and culture, gender/women in development, human rights & HIV/Aids and environment and sustainable development. Recently the sectoral themes economy and environment and sustainable development have been merged into one sectoral theme: sustainable economic development. The priority themes: access to basic services, sustainable & equitable economic development democracy and peace. Five rights: sustainable livelihood, basic social services, peace and security, social and political participation and identity (includes issues of gender and diversity). Five domains: growing-up healthy, learning, habitat, livelihood and building relations, all domains are focused on children. Five developmental elements: human capital, social capital, natural capital, physical capital and financial capital, all elements are focused on children.

ICCO Novib Plan NL TdH

In operational practice a large proportion of the activities of most CFOs concentrates on a few key sectors: health care, education and economic development. Other themes shared by most CFOs are food security, gender, environment and HIV/Aids. Four themes are included by only one CFO: urban livelihoods (Cordaid), arts and culture, ICT (both Hivos), childrens rights (Plan NL and TdH) and sports and games for children (TdH). The following table presents an indication of the expenditures over the principal policy themes (2003). Table 2.5: CFO expenditures per policy priority in 2003 (million ; CFP plus other funding)
4

The figure in the table presents the number of partner organisations in priority sectors. Due to the mergers CEBEMO/Vastenaktie (Bilance, 1999) and BILANCE with MEMISA and Caritas Neerlandica (CORDAID, 2003) the total number of partner organisations had increased substantially (to 1.975 in 2003), but this includes partner organisations for humanitarian emergency activities and others temporary partner ships. According to CORDAIDs Strategic Plan 2003-2006 , the total number of CORDAIDs partner organisations should be reduced to 860 at the end of 2006. 5 The total number of partners, according to ICCOs Annual Report 2003, was 969, but 162 only were granted a so-called mini-funding. 6 This is composed of 40 Plan country offices and 4 regional Plan organisations

12

Cordaid Urban livelihoods Healthcare HIV/AIDS Peace and conflict Access to markets Others Novib Sustainable livelihood Basic social services Livelihood and security Political and social participation Identity gender and diversity PLAN NL (CFP ONLY) Childrens rights and gender Food security Environment Healthcare Education

19.3 38.6 7.4 23.7 39.5 21.4

Hivos Economy and credit Arts and culture Gender, women in development Environment and sustainable development Human rights and HIV/AIDS Others ICCO Sustainable, equitable economic development Access to basic services Democracy and peace building Others

22.6 4.6 11.2 15.0 15.9 4.9

42.3 25.3 8.8 29.6 14.4

30.7 41.9 32.1 15.9

7.4 2.0 0.9 5.6 6.5

Source: annual reports 2003. TdH does not present expenditures in its annual report. Plan NL presents only percentages.

Geographical coverage All CFOs operate in Africa, Asia and Latin America. With the exception of TdH all organisations also work in Europe (Balkans and Caucasus regions). The following table presents the distribution of expenditures for each of the main geographical areas. Table 2.6: Expenditures per continent in 2003 (million and percentages, CFP only)
Cordaid
Africa Asia L. America Europe World-wide Northern countries Total 33.2 17.2 18.9 7.4 8.2 1.1 86 (39) (20) (22) ( 9) ( 9) ( 1) (100)

Hivos
18.9 16.1 18.7 7 6 2.2 (28) (23) (27) (10) ( 9) ( 3)

ICCO
26.3 23.7 18.3 5.7 4.6 18.4 97 (27) (24) (19) ( 6) ( 5) (19)7 (100)

Novib
44,3 34.9 22.1 8.6 8.4 1.9 (37) (29) (18) ( 7) ( 7) ( 2)

Plan NL
11.6 (52) 3.1 (14) 6.5 (29) 0.1 (0) 1.2 (5) 22.4 (100)

TdH
2.3 (24) 5.5 (59) 1.6 (17 ) 9.4 (100)

68.9 (100)

120.3 (100)

Source: annual reports 2003

The activities of the CFOs are distributed over a large number of countries: Cordaid works in 61 countries, Hivos in 32, ICCO in 48, Novib in 64, Plan NL in 39 and TdH in 13. All organisations with the exception of Plan aim at concentrating their activities in a smaller number of countries.

Includes Balkan region and central Asiaand support given to northern organisation for programmes in developing countries.

13

14

3.
3.1

STUDY OBJECTIVES AND APPROACH

Justification for the study

The co-financing programme has for a long time been an important component of the Dutch effort for development co-operation. In 2004, the total support from the budget for development co-operation to the co-financing organisations amounted to more than 410 million. In recent years the programme has been subject of a number of studies. In 2000, the interdepartmental policy review (IBO) of the co-financing programme observed that information about the results that are achieved is insufficient. Supervision by the Ministry focuses mainly on ex-post accountability. The study concludes that the Minister cannot adequately assume the ministerial responsibility for the co-financing programme8. In 2002, the synthesis report of a series of evaluations of the co-financing programme was published9. A key finding of this study is that the co-financing organisations (CFOs) have progressively become more focused on the achievements of results, but the study is critical about the way information about results is communicated, and about the capacity of both the CFOs, as well as their partner organisations, to use this information for the purpose of learning from their experiences. The study reports that the CFOs are critical about the quality of evaluation systems used by their partners10. It is observed that many partners appeared to have problems in translating the new requirements of monitoring and evaluation into their actual practice. The report also states the models have been mainly grafted onto the donors accounting requirements. For its part, the Ministry has not formulated clear priorities and criteria for monitoring and evaluation of the co-financing programme11. The Steering Committee recommends to develop a coherent framework for supervision of the CFP. The report of the Commission Box (2002)12, that advised the Minister on the allocation of funds to the six co-financing organisations, concludes that the relationship between objectives and results achieved is not clearly established in the co-financing aid chain. All these studies commented on the need to focus the activities of the organisations at the different levels in the co-financing aid chain on the achievement of results, and to provide higher levels in the aid chain with information about the actual achievement of results. None of the studies, however, clearly defines what it considers to be the results that management should focus on. In the 2003-2006 CFP policy IOB is assigned the role to inspect and assess the quality of the programme, on the basis of its own evaluation studies, as well as on the basis of the evaluation system of the CFOs. In order to do so, IOB is engaged in a number of evaluation studies. This study of monitoring and evaluation systems is one of these.

3.2
8 9

Objective of the evaluation and evaluation questions

IBO/MFP, 2000. Final report Steering Committee evaluation co-financing programme, November 2002 10 Ibid p. 40. 11 Ibid p. 65 12 Breed uitgemeten. Advies van de Commissie MFP-Breed inzake toetreding en toewijzing 2003 2006. September 2002.

15

As formulated in the terms of reference, the objective of the study is to determine to what extent the organisations involved (Ministry [DSI/MY], CFOs and partner organisations) have operationalised monitoring and evaluation systems and whether these systems contribute to the strengthening of results-oriented management. The questions that the study intends to answer are as follows: How are data collected? What M&E systems and instruments do the organisations possess and how do they safeguard the quality of information provided by such systems? How are the data and information used? How is the information on inputs, activities, outputs and results used within the organisation? How is M&E embedded in the organisation? How does the information flow through the organisation, who is responsible for collecting and verifying the information, who are the intended users and the actual users of the information? 3.3 Scope and delimitation

In the OECD/DAC glossary of terms monitoring and evaluaton are defined as follows13: Monitoring is a continuous function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. and: Evaluation is the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation lessons learned into the decision-making process of both recipients and donors. These definitions clearly demarcate monitoring resp. evaluation with regard to their scope and depth of analysis. This demarcation is maintained throughout this evaluation. For monitoring, the study focuses on all instruments and practical routines that are intentionally designed or used in organisations to provide management regularly with all types of information: including indicators, but also information that is not captured in indicators. With regard to evaluation the study includes all assessment of effects and impact and various types of internal and self-evaluations by partner organisations as well as evaluations of individual activities or relationships with partner organisations that are executed under the responsibility of the CFOs or the partner organisations themselves. Apart from activity evaluations, programme evaluations are executed by the co-financing organisations individually, or co-ordinated under the MBN. The assessment of the quality of these programme evaluations is the subject of a separate IOB activity. Result focused management of the co-financing programme implies a focus on different levels of results. From the perspective of the overall aid-chain management, the results achieved at the level of the target group (including their socio-economic environment)
13

Source: OECD/DAC Glossary of Evaluation and Result Based Management Terms (2004).

16

as well as the results achieved at the level of the partner organisations are considered to be the key results that management at the different levels should focus on. Consistent with the definition presented by the OECD/DAC, results include outputs, outcomes and impact14 achieved at these two levels. The study of monitoring and evaluation in partner organisations was executed early 2004, and presents the situation that existed at that moment. The study of the cofinancing organisations was done in phases in 2003 until September 2004. With regard to their formal reporting to the Ministry the 2002 and 2003 annual reports were analysed. The description of monitoring and evaluation at the level of the Ministry reflects the situation of September 2004. In both the co-financing organisations and the Ministry considerable activity is currently taking place to strengthen monitoring and evaluation. In order to be as relevant as possible an overview of recent developments has been included in the chapters on the co-financing organisations and the Ministry (chapters 6 and 7). They are, however, not subject to assessment. 3.4 Assessment criteria for monitoring and evaluation

Inventory and assessment The ToR define the study foremost as an inventory of systems and of the use of the information they provide. The criteria used for the assessment are based on commitments made by the CFOs with regard to monitoring and evaluation, the terms and conditions included in the contractual agreements between the Ministry and the CFOs, and the general requirements for monitoring and evaluation of the Dutch government. The concern for monitoring and evaluation to support result focused management in the CFP aid-chain exists already for some time. In the course of time the co-financing organisations have committed themselves to strengthen monitoring and evaluation, to serve as an adequate instrument for policy support15, and to provide realistic information about the problems and results of non-governmental development co-operation. CFOs also committed themselves to strengthen management and evaluation in their partner organisations, specifically self-evaluation, reporting and monitoring systems16. The CFP policy framework17 stipulates that CFOs: Must be able convincingly to illustrate the contribution they have made to structural poverty reduction by means of pre-defined, measurable result-based indicators. It also indicates that the Ministry will supervise the programme a.o. with the help of the evaluation system of the CFOs. The results of these evaluations will be submitted to the Ministry. The subsidy agreements between the Ministry and the co-financing organisations includes the following with regard to monitoring and evaluation: The work plan must, to the extent possible, define objectives, results and effects in measurable indicators. The report of the execution of the activities
14 15 16 17

See Glossary appendix 4. Fifty years of Dutch Development Cooperation (p.171). Follow-up of the Impact Study of 1990. Policy Framework for the Co-financing Programme, English translation AVT01/BZ63094A.

17

must reflect the extent to which the work plan has been executed on the basis of an overview of activities and the results achieved18. The appendix of the subsidy agreement between the Ministry and the CFOs presents the format for the annual plans. It also states that: The annual plan will to the extent possible present measurable indicators. These are preferably SMART (Specific, Measurable, Agreed upon, Realistic, Time bound). In all cases the result indicators must be defined and verifiable. The annual report follows the format and contents of the annual plan.19 The guidelines for monitoring and evaluation for the national Government (RPE) formulates a number of quality criteria for the information provided by monitoring and evaluation. These criteria concern the validity, reliability and usefulness of the information provided. On the basis of these various sources, the following criteria have been used for the assessment of the monitoring and evaluation systems at the various levels20: Is the collection of information regular and standardised Can the information be transferred to other stakeholders Is the information relevant for the intended users. Is the information valid, reliable and verifiable. 3.5 Approach and methods:

The study includes the different levels in the co-financing programme: the responsible unit in the Ministry (DSI/MY), the six co-financing organisations as well as at the partner organisations. The study is executed in three phases. Preparatory phase o Desk study. o Selection of locations for field study. Three countries were selected for detailed study of partner organisations (India, Peru and Tanzania), and in each of these countries two geographical clusters. o Establishment of a data base of partner organisations. o Formulation of terms of reference for the study of partner organisations by national consultants. o Visits to the locations of the field study to inform partner organisations and select national consultants. Field study o Detailed study of 60 partner organisations. o Validation of the studies of national consultants through visits to approx. 70% of the partner organisations studied. o Interviews with CFOs and DSI/MY. o Continued study of relevant documents Reporting Representativeness For the workshops a total of 189 partner organisations (32% of the total number in the three countries selected) were invited, and 154 of these actually participated in the
18 19 20

Subsidieregeling Ministerie van Buitenlandse Zaken, Staatscourant 1998 no. 249, page 26. Appendix 2 of the subsidy agreement CFP 2003-2006. This formulation of the criteria for assessment emerged from the first meeting of the reference group.

18

workshops. The total number of 60 partner organisations studied in detail amounts to more than 10% of the total number of partner organisations in the three countries. The selection of partner organisations was done in consultation with the CFOs concerned. Despite the substantial number of partner organisations studied, the findings cannot claim to be formally representative for the total number of more than 4500 partner organisations worldwide. The study at the level of the partner organisations, however, does present an overview of the status of monitoring and evaluation in the different types and sizes of partner organisations. Execution of the study The study was executed from 2003 to 2005. The study of partner organisations by the national consultants provided a wealth of detailed and relevant information. Despite the very detailed terms of reference, however, and the discussion and instructions with the national consultants during missions to the three countries involved, it emerged that each of the six consultants had a specific interpretation of the assignment. This limited to some extent the consolidation and analysis of the findings. 3.6 Readers guide

The study deals with a large number of organisations, that have each developed their own vocabulary and terminology. There is a variety of terms in use for the funding and support of partner organisations (projects, contracts, programmes, relationships), which often have a specific meaning in an organisation. In this report the term activity is used to include the different modalities of funding and support. In chapter 4 the complex organisation of the co-financing aid-chain is presented. Chapters 5, 6 and 7 present a description and analysis of monitoring and evaluation (systems, use of information and organisational aspects) of the partner organisations, co-financing organisations and the Ministry (DSI/MY) respectively.

19

20

1 4. OVERVIEW OF THE CO-FINANCING AID CHAIN

4.1 Levels in the CFP aid chain

From an organisational perspective, the Co-Financing Programme has a complex structure. It comprises a chain of autonomous organisations, each with their own responsibilities, tasks and stakeholders to consider. The following levels can be identified: The Ministry of Foreign Affairs. The 6 Co-Financing Organisations. The partner organisations. Target group Often partner organisations are intermediary organisations that channel the funds to implementing organisations, adding a fifth organisational level to the aid chain. Ministry of Foreign Affairs Within the Ministry of Foreign Affairs (Directorate General for International Cooperation), the Civil Society Organisations Division (DSI/MY) is the unit responsible for co-operation with civil society organisations, including the co-financing programme (CFP). Co-Financing Organisations At present 6 co-financing organisations are entitled to funding under the CFP. These organisations are autonomous non-governmental organisations: Cordaid resulted at the end of 1999 from a merger of two roman catholic development agencies (Mensen in Nood/Caritas Netherlands and Memisa Medicus Mundi) with Bilance, itself the result of the merger of the Dutch Bishops Lenten Campaign (Vastenactie) and Cebemo, the organisation that in 1969 was founded as the exclusive roman catholic channel for the cofinancing funds. Cordaids mission is to support activities in favor of poor, marginalised groups and to foster social and economic justice irrespective of age, sex, sexual inclination, origin, belief or political persuasion. Improving living conditions and strengthening society go hand in hand. HIVOS is a development organisation operating on the basis of humanistic values. It was founded in 1968 by three organisations from the humanistic movement in the Netherlands. Hivos mission is to contribute to the realisation of a free, fair and sustainable world where citizens men and women- do have equal access to resources, opportunities and markets and are enabled to participate as equals in decision-making processes that are decisive for their lives, ways of living together and their future. ICCO was founded in 1964 by the Protestant organisations as their channel for co-financing activities. In 2000 the organisation merged with Dienst over Grenzen (Service Abroad) and Stichting Oecumenische Hulp (Dutch Interchurch Aid), one of its original founding fathers. NOVIB was founded in 1956 by a number of individuals active in universities and science, churches, media and other civil society organisations. Novib is a non-denominational channel and joined the Co21

financing Programme at the end of 1968. NOVIB joint the Oxfam alliance in 1995 PLAN Netherlands is a member of the international federation Plan International established in London. In total Plan International has 15 member organisations in 15 countries in Europe, North America and East Asia, which are responsible for fundraising among sponsors and governments. Plan is working in 45 programme countries mainly through Plan country offices. Plan NL was admitted to the CFP at the end of 1999. Terre des Hommes was founded in 1960 during the independence war in Algeria. From the very beginning the organisation focussed its activities on supporting children in emergency situations and on exercising political pressure to improve childrens position. The Foundation Terre des Hommes Netherlands was founded in 1966 when also an International Federation Terre des Hommes was established on the basis of an international charter. Terre des Hommes was admitted to the CFP per January 2003.

Partner Organisations CFOs achieve their aim through co-operation with independent private organisations, mainly in developing countries, the so-called partner organisations. The policy framework does not specify a preference for a particular type of organisation, but mostly the term NGO is used. Partner organisations are of a different size and type and include21: Grassroots (community-based) organisation: The members and the governing body of the organisation are from the target group itself; the organisation's aim usually is to improve members' socio-economic and/or political position in society. Intermediary (development) organisation: An organisation that does not originate directly from the target group itself and is geared to guiding and supporting development activities by grassroot organisations. Service organisation: An organisation with specific know-how that provides (technical) support and advice to improve the quality of development activities and assists in the implementation of such activities. These organisations usually assist other development oriented organisations or networks that lack the necessary know-how. Network organisation: An alliance of organisations generally centred on a specific theme or interest. Networks are as a rule geared primarily to the exchange of information and the co-ordination of activities. International networks mostly perform a lobbying function. 4.2 Relationships between organisations at different levels

Autonomy and dependence The autonomy of the co-financing organisations and their partner organisations is one of the defining characteristics of the co-financing aid-chain. The chain is composed of organisations that are autonomous, but at the same time strongly interrelated and to a considerable extent mutually dependent. All CFOs, prior to their inclusion in the cofinancing programme, were legally established as autonomous organisations according
21

Met het oog op kwaliteit. GOM, maart 1995

22

to Dutch law. They have their own mission and objectives and formulate and implement their own policies. The same applies for their partner organisations in developing countries that, under different modalities, do have corporate personality and are registered under their respective national legislation. The CFOs dispose of considerable networks of non-governmental organisations in developing countries, and have the capacity and skills to select appropriate partner organisation and activities. Moreover, the CFOs are themselves expressions of broad support in Dutch society for development co-operation, and have assigned themselves the role to maintain and expand this support. At the same time, however, CFOs receive a substantial amount of public funds, and the CFOs act as donors to their partner organisations. This brings with it accountability from the recipient to the donor. These capabilities and the significant role of CFOs in Dutch society to a certain extent balance the financial dependence of the CFOs on public funds, thus providing a basis for continued autonomy. The resulting relationship between the Ministry and the cofinancing organisations is in fact a dynamic balance between these different factors. Funding modalities Funding of the activities of the CFOs by the Ministry was initially project based. Since the mid-80s a programme funding model was applied. A predetermined proportion of the budget for development co-operation was reserved for the co-financing programme, and the distribution of the available resources was largely left to the CFOs themselves. Changes in the laws governing the provision of subsidies related to European regulation and the follow-up of the recommendation of the IBO/MFP led to a redefinition of the role of the Ministry. This resulted in the policy framework 2003 2006 that regulated the relationship between the Ministry and the CFOs. The Ministry became explicitly responsible for the allocation of funds over the different CFOs, and it made the allocation of resources dependent on a number of criteria, inter alia the extent to which results to be achieved were clearly stated in the multi-annual management plans. During the first years of the co-financing programme, support to partner organisations was project based. Progressively the perspective on the CFO PO relationship evolved. During the second half of the eighties some CFOs experimented with core funding of some major partner organisations, mostly referred to as strategic partners. Gradually the number of strategic partners expanded. With these organisations the CFOs initiated a regular policy dialogue. The next step was the organisational or institutional approach where the focus of the (pre-contract) assessment is more on the objectives, strategies and capacity of potential partner organisations than on their specific programmes and/or projects. Partnership and donorship The relationship between the CFOs and organisations in the South is mostly referred to as partnership. This stands for the humanitarian, moral, political, ideological or spiritual solidarity between NGOs in the North and the South that joined together to pursue a common cause of social change22. Partnerships often started on the basis of religious or philosophical/political concurrence between the CFOs and their partners. Such partnerships are unequal in terms of financial and organisational resources, but the
22

Fowler, Alan. The Virtuous Spiral: A guide to sustainability for NGOs in International Development. 2000

23

religious or philosophical/political concurrence is a strong binding factor, and is often perceived to restore, to a certain extent, the balance in the relationship. During the first phase of the co-financing programme partner selection followed more the line of what could be called traditional partners: CEBEMO/CORDAID in the roman catholic domain, ICCO among the protestant denominations and NOVIB and HIVOS elsewhere, including the Muslim world. During the eighties and nineties this traditional division faded, even to the extent that it is not uncommon that two or more CFOs support the same partner organisation. At present the term partnership is used very loosely and it seems to have lost some of its meaning23. Often relationships lack the explicit binding factor and are based more on common formal social objectives for example with regard to poverty alleviation. Also the number of civil society organisations in many developing countries expands rapidly, introducing a degree of competition for external funding, consequently weakening the position of partner organisations in the South. Some24 argue that, given the differences in power, partnership in the sense of equality and reciprocity between donors and recipients is an illusion. At the same time it is argued that the illusion of partnership is a vital element in the relationship between NGOs in the North and the South. With the admission of Plan NL and later, and to a lesser extent, TdH other modalities and relationships were introduced in the CFP. Plan NL used to execute its activities through the Plan country offices in different countries. In 2003 a new strategy was formulated that recognised the importance of working with local partner organisations. TdH is more project focused. As recommended by the Steering Committee, all CFOs25, have formulated partner policies, codes of conduct and principles of good donorship. The partner policies cover the principles of partner selection and the kind of relationships the CFO want to enter with a PO. In general, a distinction is made in the type of relationship, ranging from a project relationship to strategic partnership, with different intensity and length of the relationships. The codes of conduct and principles of good donorship formulate more in-depth how the CFO and the partner organisation work together. In essence, these documents attempt to reconcile issues such as shared visions and interests, mutual respect for autonomy on the one hand with transparency and accountability on the other. Functional relationships In functional terms, organisations at the various levels play a different role in the overall CFP aid chain, and provide a different mix of products (essentially funds and services) to the organisations at the next lower level. An overview of the roles of the organisations involved and the funds and their products for the next level organisations is presented in the following figure.

Figure 4.1: Overview of roles and relationships in the CFP aid chain.
23

The use of the term partnership between NGOs has been discussed frequently during recent years, not only by Alan Fowler, but Fons van der Velden, Lau Schulpen en Paul Hoebink (in: Hulp in Ontwikkeling, Assen 2001), Biem Lap, etc. as well. 24 In : Grenzen aan de Hulp. Beleid en effecten van ontwikkelingssamenwerking. Amsterdam, 1997. 25 TdH was not a CFO at the time when the Steering Committee executed the CFP evaluations.

24

Ministry (DSI/MY)

Roles and internal processes Policy formulation for overall CFP Allocation of funds for CFP in budget for development co-operation Reporting to Minister, Parliament and other constituencies Funds and products delivered Policy framework with overall objectives and strategies Overall programme funding Roles and internal processes Formulation of sector and regional policies. Formulation of strategies for partner selection, partner relationships and intervention strategies. CFOs Partner selection. Approval of activities and commitment of funds. Identification, formulation approval and execution of other interventions Supervision and control. Reporting to donor and constituencies Funds and products delivered Sector, regional and country policies Technical assistance Equipment and facilities Activity, programme and organisational funding Roles and internal processes Formulation of intervention strategies at local, regional or national levels POs Identification, formulation and approval of activities and other interventions. Execution of activities and other interventions Supervision and control Reporting to CFOs and other constituencies Funds and products delivered Knowledge and information Skills Productive resources, including funding Organisation Influence on environmental factors Roles and internal processes Target Group Participation in societal processes Productive processes

CFOs are increasingly redefining their responsibilities, and consequently the domains for which they consider themselves accountable. Achieving impact at the level of the intended target groups is the overall aim of the development intervention, but first and foremost the responsibility of the partner organisations. In a procedural sense the relationships between the organisations is governed by the systems of planning, monitoring and evaluation (PME systems) of the organisations at different levels. Two systems can be clearly identified: PME system governing the relationship between the Ministry and the CFOs. This comprises a plan component (the policy framework, the multi-annual management plans and the annual plans) as well as a feedback component (annual reports, programme evaluations). PME system governing the relationship between the CFOs and their partner organisations. These system also include the relationship between partner organisations and the envisaged beneficiaries. This PME system also comprises a planning component (activity plans, agreements and annual operational plans) and a feedback component (annual reports, activity evaluations, field visits and other contacts and exchange of information). This study focuses on the feedback components of both these PME systems. Monitoring and evaluation in the CFP

25

The complex organisational structure of the CFP leads to a complex structure of monitoring and evaluation. Organisations at each of the levels in the aid chain operate their own M&E systems, often independently. At the same time these systems at different levels are, or should be, to a certain extent interlocked. Fig 4.1 provides a schematic overview of the overall feedback system and the feedback instruments that organisations at different levels use to obtain the necessary information. The figure also provides the rationale for the structure of the remainder of this report. In the following chapters M&E at each of the levels will be described. First in chapter 5 the M&E at the level of partner organisations will be presented. Chapter 6 focuses on the M&E by CFOs, and chapter 7 on the Ministry/DSI/MY.

26

Figure 4.1: Information architecture CFP

Accountability - Policy formulation - Contract management

Chapter 7

Answers to Parliamentary questions

Annual report

Information for Minister

Appraisal information

DSI/MY
IOB evaluations

Visits to partner organisations

Annual reports

Policy discussions with individual CFOs

MBN platform meetings with Ministry

Specific Information requests

Separate IOB assessment

Programme evaluations

CFO
Chapter 6
Visits to POs

Progress reports

e-mail/tel. contacts

PO evaluations

PO
Surveys and Interviews

Chapter 5

Implementing POs organisaties

Intermediary POs
Field visits

Internal reporting

Workshops / seminars

Field visits

Impact studies

Progress reports implementing organisations

Workshops / seminars

Evaluations

Implementing organisations

Internal reporting

Workshops / seinars

Field visits

Internal evaluations & impact studies

Implementation process Target group 27

28

5.
5.1

MONITORING AND EVALUATION IN POS

Overview of partner organisations studied

In the three countries a total of 60 partner organisations is studied. The distribution of the partner organisations over de CFOs is summarised in table 5.1. Table 5.1: Relationships of partner organisations studied with CFOs
Relationships with Dutch CFOs * India 4 4 5 5 2 2 22 Peru 7 4 7 4 1 3 26 Tanzania 7 5 4 5 1 1 23 Total 18 13 16 14 4 6 71 *) Some partner organisations have relationships with more than one of the Dutch CFOs26. Cordaid Hivos ICCO Novib Plan NL TdH Total number organisations 20 20 20 60

Partner organisations are of different size and type, and include intermediary organisations, network and membership organisations and service organisations. The following tables provides an overview of the types and size of the 60 partner organisations studied. More detailed information of the partner organisations is presented in appendix 5. Table 5.2: Type, turnover and staff of partner organisations studied
India Type of partner organisations Intermediary 12 Service provider 6 Network 1 Grassroot 1 Turnover of partner organisations in 2002 < 300,000 (US $) 10 300,000 1,000,000 (US $) 5 > 1,000,000 (US $) 5 Number of staff of partner organisations 1-20 2 21-60 9 > 61 9 Peru 14 4 2 0 3 12 5 5 11 4 Tanzania 10 4 1 5 10 6 4 10 7 3

Table 5.3: Turnover per type of organisation studied

Type of orgs. Turnover 2002 < 300,000 300,000-1,000,000 > 1,000,000 Intermediary organisations 10 15 11 Service providers 8 5 1 Network organisations 1 2 1 Grassroot organisations 4 1 1

Often partner organisations have several donors, each funding a different activity or programme. In fewer occasions partner organisations have donors that fund the entire programme of the organisation (organisational or institutional funding). Also of the partner organisations studied the majority had (funding) relationships with several

26

In Tanzania: Kuleana (Hivos, ICCO), CSSC (ICCO, Cordaid), TGNP (Cordaid, Hivos), in Peru: Cedia (Hivos, Novib) KALLPA (Cordaid, ICCO, TdH), Arariwa (Cordaid, ICCO), Cedep Ayllu (Hivos, Cordaid), IDESI (Novib, ICCO) en in India: Seva Mandir (ICCO, Plan), CWS (Novib, Hivos).

29

donors. Only 3 of the 60 partner organisations 27 had no other donor than the Dutch CFO. One third of the POs studied received funds from 5 to 6 donors. Another third of the 60 POs studied received funds from 7 or more donors. Donors included a broad range of (inter-)national (non-)governmental organisations. Table 5.4: Numbers of donors of partner organisations
Number donors Number POs of of 1-2 10 3-4 11 5-6 20 7-8 6 9-10 5 11-12 3 More than 12 5

Source: Detailed studies POs

5.2

Organisation of monitoring and evaluation

Twenty-six of the 60 partner organisations studied have established some kind of monitoring and evaluation unit (sizes varying from a part-time staff member to a team). The mandate of these units depends on the size and structure of the organisation. In a number of organisations a research, co-ordination or planning unit is also responsible for monitoring and evaluation. The monitoring and evaluation units prepare monitoring and evaluation plans, collect and verify data, prepare tools for collecting information, analyse data and provide feed-back to the management team for the necessary review of plans, organisation of evaluation workshops and internal/external evaluations. Table 5.5: M&E unit in partner organisation
Turnover 2002 USD < 300.000 300.000- 1.000.000 > 1.000.000 Total POs Number of POs 23 23 14 60 Yes 8 10 8 26 No 15 13 6 34 Percentage 31 % 44 % 57 % 43 %

Of the organisations that do not have a monitoring and evaluation unit, all report the existence of formal instructions for monitoring and evaluation. In those organisations the responsibility for monitoring and evaluation rests with management at different levels (management team, programme officers, co-ordinators/head of departments). Organisational arrangements, staffing and mandates of the monitoring and evaluation units vary from one partner organisation to the other. In the following box two examples are briefly described.

27

The three partner organisations are Cedesi (partner of TdH), Nafasi (direct project of TdH), SIFFS (partner of Hivos).

30

Box 5.1: Some examples of M&E Units in partner organisations

CECOEDECON (India, intermediary organisation, turnover $678.000) The M&E Unit of CECOEDECON comprises four regular staff responsible for designated program area and projects. Apart from conducting desk reviews and preparing program reports they conduct field visits to carry out physical verification, output, outcome and impact analysis involving community members. The unit has the following mandate: Plan and execute the monitoring of programme activities and projects Train field staff and CBOs in monitoring processes and tools Secure, consolidate and produce monthly reports and field visit observations Conduct six monthly outcome assessment of project activities Conduct/ organise impact evaluation of completed program are and projects. CEDEP AYLLU (Peru, intermediary organisation, turnover $765.000) Initially the responsibility for monitoring and evaluation was assigned to the training unit. Donors, however, increasingly required analysis for project preparation and more information in progress reports. More capacity for planning and analysis was therefore required. First a planning unit was established, later it was converted into a unit for planning and analysis, including monitoring and evaluation. Initially the unit comprised 4 staff, but departures of staff reduced the number to 1. The unit now has 2 staff, a co-ordinator and an expatriate technical assistant. The unit co-operates closely with the two field directors of the organisation. The M&E related tasks of the unit are: Develop instruments (i.e. reporting formats) and training of staff, specifically with regard to concepts and definitions. Verification of the information collected. Feeding the information in an SPSS data base and execute the necessary analysis (i.e. generate a number of standardised quantitative overviews and tables). Storage of the information in accessible formats. The unit forwards the standard overviews and tables to the field directors, who use it to discuss progress within their units and for operational decision making. The co-ordinator of the unit is occasionally present in these meetings. The information provided gives operational staff an understanding about the progress of programmes, the use of funds and the available budgets. For the activities aimed at the development of productive sectors the system is reported to work satisfactory. For other activities such as enhancing citizenship and democracy it proves to be difficult to define appropriate indicators. This is one of the priorities for the coming year. In the organisation all staff is involved in planning, monitoring and evaluation. The unit support the staff. For monitoring the unit promotes standardisation of the collection and processing of the information. Staff interprets the information. This enhances their ownership of the conclusions drawn. It also enhances their motivation to collect data, and it prevents the isolation of the unit as a separate think-tank in the organisation.
Sources: questionnaire partner organisations and interview IOB-team.

Planning in relation to M&E All partner organisations studied used annual plans as the basis of their M&E system. The majority of partner organisations used planning techniques based on, or derived from, the logical framework for the planning of activities. On the basis of the overall activity plan, or the plan for the entire organisation, a hierarchy of operational plans is formulated, ranging from annual plans for the activity or the organisation to sometimes weekly plans for units or even individual staff. Resource persons from partner organisations indicate that contacts with international donors, among them the Dutch CFOs, have contributed to the widespread use of the logical framework for planning. In some organisations the logical framework plan format is directly translated in monitoring matrices, for example in FODRA (India), CECOEDECON (India) and CORDS (Tanzania).

31

Box 5.2: Translation logical framework into monitoring system:

Fodra (India, intermediary organisation, turnover $102.500) Fodra uses the logical framework for the planning of its activities. On this basis annual plans are drafted, which are in turn broken down into monthly activity plans. These monthly activity plans are used as the reference for monitoring the activities of the field staff. CECOEDECON (India, intermediary organisation, turnover $678.000) The Yearly Plan of Operation (YPO) is designed to present a comprehensive overview of the activities to be undertaken in a year along with allocation of time, responsibilities, cost to be incurred for each activity, and a set of indicators. Each programme unit prepares its YPO and the composite Plan of the organisation is then compiled by the M&E Unit and shared with the other units. Apart from the core programmes, a YPO is also worked out for the related community-based organisations. The yearly plan is further specified into monthly and weekly work plans for field staff, that are used to monitor implementation. CORDS (Tanzania, grassroot org., turnover USD 200,000) For monitoring CORDS uses matrices, derived from the logical framework structure. For each of the expected results the planned activities are listed, with the direct outputs expected, as well as the amount of funds required. This format is used for the regular monitoring of the progress of execution of activities.

In many of the organisations indicators are derived from the logical frameworks for the purpose of monitoring and evaluation. These indicators, however, tend to focus on the more tangible aspects of an activity: the use of inputs, the execution of activities and the realisation of direct outputs. The effects and impact on the target groups and aspects of quality are usually not included. Procedures In the partner organisations studied, monitoring is a regular part of the daily work of operational and management staff. All partner organisations have constructed an organisational and procedural system for monitoring and evaluation that is geared towards the collection and internal communication of feedback information towards the different management levels in the organisations (including recording systems, guidelines for reporting and for evaluation). Routines for using the feedback information exist in most organisations: regular meetings at operational and policy level, and, although at lower frequency, internal evaluations or reflection sessions. These provide the main occasions for use of the feedback information. 5.3 Monitoring instruments In partner organisations a diversity of monitoring instruments is found. This includes well structured instruments (such as reporting procedures) and a variety of other monitoring instruments. 5.3.1 Reporting

Two types of reporting can be distinguished: Reporting between organisations, as is the case between implementing organisations and intermediary (partner) organisations that are funded by CFOs. Reporting in the partner organisations themselves, from staff at the operational level to the management levels in the organisations. This occurs both in implementing partner organisations and in intermediary organisations.

32

These types of reporting cannot always be clearly distinguished, as the different modes of operation in an organisation sometimes merge: implementing organisations often work directly with target groups, sometimes even (in)formal organisations, thus growing towards a role of intermediary organisation. On the other hand, intermediary organisations also often directly implement activities themselves. Progress reports from implementing organisations to intermediary organisations Intermediary organisations have agreements with implementing organisations, mostly grassroot organisations. The agreements stipulate the frequency of reporting from the implementing organisation to the intermediary organisation. Reports are usually required several times per year, with an annual report that provides an overview of activities and (possibly) results for the entire year. The most common frequency for narrative reporting is either monthly or quarterly. Financial reporting follows the same frequency, in a few cases more frequent financial reporting is required. Guidelines for the format and contents of the progress reports are less common. Less than half of the intermediary organisations report the existence of guidelines for narrative progress reports (for financial reporting, however, instructions are much more specific). Where reporting guidelines exist they vary from verbally agreed formats to written instructions. The following box presents a number of examples of reporting requirements. Box 5.3: Reporting requirements for implementing organisations
CRS (Tanzania, intermediary organisation, turnover $965,800) CRS Tanzanias contracts with implementing organisations are clear on the necessity for reporting on results. Projects have a clear monitoring and evaluation plan including objectives, indicators, collection frequency (monthly, quarterly, yearly), data source, who is responsible for collecting and analysing information, who will use the information as well as the critical assumptions. NAFASI/TdH (Tanzania, intermediary organisation, turnover $52.300) Contractual arrangements clearly define responsibilities for reporting and guidelines are provided. Quarterly financial and half yearly progress reports have to be produced. The logical framework of the project is the basis for reporting. Sruti (India, intermediary organisation, turnover $146.100) The Fellows of Sruti are the implementing organisations. They are required to provide six monthly progress reports. The Fellows supplement field-visit information by sending a narrative account of the activities being undertaking at their end. The Fellows are expected to report on: case studies, issues/ impact of work and problems faced. No training on reporting style and standards has been organised. The Fellows are not very punctual in submitting the progress reports as only 50 % do it on time. Plan Peru (intermediary organisation, turnover $2.840.000) Plan Peru has contracts with 5 local organisations. All the contracts with implementing organisations stipulate the content and frequency of the reports. The narrative report has to include the number of clients, movements in the financial portfolio, cash flow, capacity building processes, etc. The financial report contains a analysis of the portfolio. Source: Reports of detailed studies of partner organisations

During the preparatory workshops (where the majority of participants were representatives of intermediary organisations), resource persons of these organisations considered the quality of the narrative progress reports from the implementing organisations very variable, often inadequate. Frequently mentioned constraints were the limited reporting and analytical skills in many implementing organisations. This results in a limited scope of the reports, focusing on the use of inputs and description of activities, frequently incomplete or with considerable inconsistencies that required checking and further requests for information from the side of the intermediary 33

organisations. The large majority of intermediary organisations therefore do not rely only on progress reports. These are usually complemented by intensive contacts during the year, by means of field visits, regular contact by telephone and sometimes e-mail and contacts during meetings, workshops and other events. Internal reporting All partner organisations studied had a hierarchy of internal reports, sometimes starting from daily or weekly reporting by implementing staff, up to annual reports. The reports are generated both for internal and external purposes, the general rule being that daily, weekly and monthly reports are exclusively for internal management purposes and the (bi-)annual reports predominantly for monitoring by the boards of the partner organisations and external use. Table 5.6: Reporting in partner organisations studied
Number of > Monthly POs 20 5 20 5 20 5 Frequency of internal reporting Monthly Quarterly Half-yearly 16 13 13 4 12 12 17 18 14 Annually 20 20 20

India Peru Tanzania

Source : detailed studies of partner organisations The internal reporting is often based on a system of registration of inputs, activities and direct outputs. These registration systems are often well structured and elaborate. Box 5.4: Registration system.
Guaman Poma de Ayala (Peru, intermediary organisation, turnover USD 1.700.000) The organisation uses 8 formal standard institutional forms (fichas) for all its programmes and projects, which together record the experiences of all teams. The forms are: Monthly Programming Form Form to report on Participation in Training Events (for workers) Form to report on Participation in Work Co-ordination Bodies Register Form (for trainers) Evaluation Form (for participants) Form to report on Participation in External Consultative Bodies (for workers) Register form for advisory services (for advisors) Form for cost calculation. In addition to these 8 standard forms, specific forms are developed to monitor key-indicators for each programme and project. These fichas, together with the minutes of meetings with the implementing organisations are used as input for the monthly, biannual and annual reports

The scope of the different reports varies, but most reports focus on the use of inputs and the progress of implementation. Six-monthly and annual reports are sometimes expected to include information on outputs and even the effects of interventions on the target groups, but from the detailed studies and interviews it was observed that these progress reports usually do not provide such information. Many organisations are engaged in a process to put more emphasis on the reporting of the outcomes and impact of their interventions. Resource persons indicated that this is very complex, due to the time lag between interventions and the occurrence of outcomes/impact, and the difficulties encountered to collect relevant and reliable information. Often problems are encountered in defining, measuring, aggregating and reporting such information. The limited capacity of staff at the field and lower

34

professional levels in the organisations to measure and report outcome was regularly mentioned as the key constraint. Because management is aware of the generally limited capability to report, there is considerable attention for the quality of the information provided in the different types of reports. Information collected by operational staff is validated by the next higher levels in the organisations. Because operational staff and middle management (and sometimes senior management) are quite close to the operational reality and the target groups, information is received from different resource persons, often through more informal channels. If major inconsistencies appear between the information from different resource persons, this is usually investigated by field visits from officer level staff. Some organisations reported that when activities or achievements reported by field staff are contested by the target groups or other stakeholders, more detailed verification is done by officer-level staff of the organisations. In about half of the organisations visited, follow-up field visits are done by staff that has not been involved in the execution of the interventions. Attitudes of staff, specially at the operational levels, are action oriented. The commitment to the work is usually high, and reporting is seen as a formality that does not directly contribute to the achievement of the organisations mission. Reporting is often not considered as work by middle and lower level staff, but perceived as a burden linked to donor funding. The oral traditions in the societies where partner organisations operate also favour oral exchange of information over reporting. For management in the different organisations processing information is a time-consuming task, as is illustrated in the following box. Box: 5.5: Observation by the Director of MRHP:
. through the overall feedback information system in the organisation a substantial amount of information is generated at the operational levels in the organisation. This information is brought to the attention of the middle level staff of the organisation, but these find it difficult to analyse the information and communicate it to management. Processing of the information was considered a serious burden on the organisation, and therefore it is only summarised. Much of the information value is then lost in the process.

Reporting to target groups In a majority of the partner organisations, target groups are somehow organised (in groups) for the purpose of implementation of activities. Only in a few cases, however, these groups are used for informing them about the progress of the activities that directly concern them, or about the activities of the partner organisation in general. There are no standard practices to present written reports to target groups. Sometimes, they are informed in an ad-hoc manner about activities and achievements of the organisations. 5.3.2 Other monitoring instruments

Apart from the regular reporting, monitoring takes place through a variety of instruments including: Field visits. Regular contact by email and telephone. Various methods to obtain information from target groups. In the operational practice of most organisations a mix of these instruments is often used. 35

Field visits All organisations require their staff to visit the field regularly to collect information about the progress of their activities and about the perceptions of stakeholders, including target groups. Field visits are done by programme staff and supervisory staff (all organisations), management (on specific occasions by most organisations) and administrative staff (by some organisations). Some of the organisations explicitly require all staff (also from headquarters) to regularly visit the field, and a few organisations operating in rural areas have an explicit policy requiring field staff to live close to, or among the target groups, in order to enhance regular informal contact between them and the staff of the organisation. Most partner organisation require reports of field visits. These reports are submitted to supervisors or management, and/or are discussed during the regular internal progress meetings. Most of the information included in the reports concerns the progress of activities and the realisation of direct tangible outputs. During field visits qualitative observations are obtained about outcomes and impact, and the perceptions of target groups and other stakeholders. This information is ad-hoc and impressionistic, but it substantially enhances the understanding of the situation and needs of the target groups and of the effects of the organisations activities on their lives. However, it rarely finds its way into the field visit reports of staff, but is more frequently shared verbally and informally among colleagues and sometimes in progress meetings. Regular contact by email and telephone Communication between intermediary organisations an implementing organisations is often frequent. Contact by email and telephone is a more informal way of monitoring. The distinction between monitoring and building a relationship, however, is a thin one. Contacts are initiated by both sides. Implementing organisations seek contact with the intermediary organisations for example when a problem occurs and assistance is needed. Intermediary organisation value such regular contact with the implementing organisations to keep informed on the progress of activities and about possible problems as soon as these emerge. Various methods to obtain information from target groups Workshops and meetings with target groups and other stakeholders are used by a minority of organisations to collect information from them. Such events may be very informal to structured and formal, with a predefined agenda and programme. An example of an informal workshop to provide information to a lobby organisation is the regular workshops organised by the Tanzania Gender Network Programme, a lobby organisation for gender issues.

36

Box 5.6: Example of a workshop providing monitoring information

TGNP (Tanzania, network organisation, turnover $701,000)

Every Wednesday afternoon TGNP holds an open forum under the mango trees outside its offices. This is a community learning session, which includes common people from the streets, and surrounding communities who walk in and give their views on TGNPs lobby and advocacy work. The forum also discusses the views of people on the various policies, legal frameworks etc, that are in place in Tanzania. After the forum TGNP management discusses the issues that were raised, and decides how TGNP will react to them.
Source: detailed study TGNP and interview with TGNP management

In some partner organisations stakeholders, among them target groups, are eager to volunteer information. Most common are their visits to the offices of organisations that are frequently observed when visiting offices of partner organisations, specially those that work directly with rural communities. In some cases information is received in written form. HakiElimu, for example, receives several thousands of letters from children in the schools where they have been active and from the broader public (even to the extent that processing these letters is a major burden for the organisation). Other organisations, specially those that use mass media receive telephone calls during radio programmes or letters from listeners, for example LHRC. The information thus obtained is unstructured and difficult to systematise. It is likely to be biased but it is often very rich information, that adds to the understanding of individual staff of the organisation about the effects of their activities on target groups and the perceptions of the target groups on the activities of the organisation. 5.4 5.4.1 Evaluation External evaluation

External evaluations (other than the activity evaluations initiated by the CFOs - or other donors - that are discussed in chapter 6) are rare. Among the 60 organisations studied only 2 reported external evaluations of their own activities initiated by the organisation itself. Three intermediary organisations reported that they execute external evaluations of implementing organisations. 5.4.2 Internal evaluation

In almost all organisations some form of internal evaluation is reported. This includes impact studies, internal activity evaluations and self-evaluations often in the form of regular evaluation workshops to discuss and reflect on experiences from the execution of activities. The following table presents an overview of the incidence of different types of internal evaluation among the partner organisations. Table 5.7: Incidence of internal evaluation among partner organisations (number of organisations).
Number of Pos studied India Peru Tanzania 20 20 20 Impact studies 4 4 3 Internal activity Periodic selfevaluation evaluation 10 5 9 15 17 14 Organisations without any type of internal evaluation 1 1 1

37

Impact studies

In India, four organisations execute detailed studies by students and interns or local consultants. These studies generate a wealth of detailed information about the execution of activities and their effects on the target communities. The quality of these studies is judged to be very variable by the resource persons in the organisations concerned, but their costs are low. In partner organisations where micro-studies are reported, synthesis studies are often done to draw more general conclusions that have a broader relevance for the organisation. Box 5.7: micro-studies by Seva Mandir
Seva Mandir (India, intermediary organisation, turnover $1.200.000) Seva Mandir carries out micro-studies by visiting students, interns or volunteers. These micro-studies, while not strictly-speaking evaluations, have helped the organisation in documenting and assessing a number of achievements, processes and impacts on target groups. The studies are looked at carefully and necessary action taken, based on the their recommendations. After completion, the studies are presented in an open forum attended by Seva Mandir staff and other stakeholders. The open forum is also a tool for Seva Mandir to get more detailed information and perceptions from the students / interns / volunteers who undertake micro studies. It is also an opportunity to cross-check the data collected during the field research with stakeholders present. During 2003 and 2004 a total of 51 such studies have been completed.
Source: Detailed study Seva Mandir

In Peru, two partner organisations report the regular execution of surveys among target groups 2 years after completion of a programme in an intervention area. The purpose of such surveys is to assess the lasting effects of the programme. One partner organisation executes a survey upon completion of the intervention programme, and another organisation conducts surveys to assess impacts on a yearly basis. In Tanzania, three organisations report the regular execution of surveys, two at the end of an intervention, one annually. Although a clear pattern does not emerge, the organisations that conduct impact studies are often larger and longer established than other organisations. The majority of the 11 organisations works in the general field of rural and/or economic development. Internal activity evaluation and self evaluation In almost all organisations considerable attention is paid to internal evaluation, in a variety of forms. This includes internal activity evaluations, internal programme review workshops during regular retreats, specific meetings dedicated to review progress and emerging constraints etc. Usually operational staff and middle management present the progress of activities, and sometimes the outcomes/impact of the interventions on target groups to higher management that participates in the exercise. This information is discussed and analysed, and if applicable, lessons are drawn from the experience for future actions. Respondents indicate that documentation of the discussions and lessons learned of these sessions is variable: sometimes elaborate minutes are made of the discussions and conclusions, but usually reporting is minimal or absent. Many organisations use internal activity evaluations for the internal evaluation workshops. These workshops are considered important by staff and management of the partner organisations. Internal evaluation workshops usually precede the formulation of new annual work plans. The findings of the evaluation then feed directly in the planning

38

process. The following box illustrates some of the practices found with partner organisations. Box 5.8: Examples of internal evaluation workshops.
Kivulini (Tanzania, membership organisation, turnover $148.200) Kivulini organises 6 monthly seminars attended by Kivulini staff and community members to discuss progress. The discussion feeds into the operational decision making in the organisation, and is shared with donors and target groups. The department of Capacity Building is responsible for preparing these workshops. It aims to build conceptual and practical skills for successful implementation. IDRO (Tanzania, intermediary organisation, turnover $1.176.700) Internal reviews are done during heads of departments meetings held every 2 months. A major evaluation workshop is organised annually to assess the performance of the implementing organisations. Monitoring reports are presented and discussed during this workshop. Fedina (India, intermediary organisation, turnover $381.600) Fedina organises regular review sessions. It is the only way to assess achievements of general objectives and direct results. These internal reviews are also instruments to check what the real problems and difficulties are, and to discuss ways to correct mistakes mid-course without waiting for the end of the projects. The aim is also to improve the cohesiveness of each local group and of Fedina in general. Respondents indicate, however, that these reviews would be more effective if they were better prepared. SACCS (India, intermediary organisation, turnover $669.400) The current practice of conducting half-yearly evaluation workshops within the organisation reflects the spirit of interactive participation and openness in the organisation. During these workshops, an assessment is made of the achievements and shortcomings and a future plan of action is framed. A critical analysis is done comparing the reports presented by the activists and the reports made by the project staff during their visits. This also helps strengthening the perception about the work that remains to be done and the effectiveness of the strategies. MYRADA (India, Intermediary organisation, turnover $5.400.000) Internal review workshops are used to assess how the work is going (targets versus achievements, achievement of expected effects, quality of work, content of activities), what changes are required in programmes and strategies and what new requirements are emerging from the field. Such sessions also provide inputs for new documents and case studies, reporting, and training programmes. Cedep (Peru, intermediary organisation, turnover $684.000) Every year an internal workshop of three days is held with the participation of all staff members (execution completed Annual Plan and preparation new Annual Plan). This process of reflection and internal evaluation is perceived to be very rich in information and very useful. IDEAS (Peru, intermediary organisation, turnover $870.000) Once a year during three days, IDEAS conducts an internal workshop where projects are evaluated, the performance of staff is assessed, and policies and operational experiences are discussed. The outcome of the workshop is documented by the executive director who presents the report to the Asemblea de Socios. It is used for institutional learning and strategic decision making.
Source: Questionnaires partner organisations and interviews IOB-team

5.5

Organisational feedback systems

Partner organisations combine the different instruments mentioned in the previous paragraphs into an overall organisational monitoring and evaluation system. This system exists not only in larger organisations, but also in the smaller and less sophisticated partner organisations. Two examples of such an organisational system are presented.

39

Table 5.8: Overview of monitoring and evaluation system MRHP (Tanzania, intermediary organisation, turnover $98.100)
M&E instruments
External evaluation by CFO

Description
Evaluation by external evaluators done in 1993 and 1998. Next external evaluation envisaged for 2004.

Purpose

Refers to:
Evaluated the project on the basis of the project document that is part of the agreement with ICCO

Self Evaluations

End of year progress and financial report Annual reports

Quarterly reports (departments) Quarterly reports (operational) Monthly staff meetings Weekly MT meetings Informal contacts with target group

Accountability: to CFO (external) and management and Board (internal). Decide on possible continuation of co-operation between PO and CFO Formulate lessons learned for possible next phase of partner ship. Annual (from 2001 onwards) Management decision making internal participatory and learning. evaluations done at village, divisional and organisation level Report written in prescribed Mainly accountability: to CFO format for donors relating and Board. achievements to project proposal. Reports the achievements Report by management and results compared to plan presented to the General and explains variances; Assembly of the organisation for reviews organisational discussion of progress and policies. policies. Consolidation of quarterly Review of progress and key operational reports at issues by heads of departments departmental level. and management as well as Board Report on activities executed Review of progress by heads of to heads of departments departments. Discusses operational issues: Review of progress of progress of activities, implementation at departmental problems encountered, level external relations. Discusses operational issues: Review of progress by team coprogress of activities, ordinators. problems encountered, external relations. A variety of informal contact with groups or individuals of the target group during field trips and group meetings

Annual plan

Project proposal

Annual plan

Annual plans and Quarterly plans Quarterly plans Quarterly plans

Not specified

40

Table 5.9: Overview of monitoring and evaluation system of RAID (India, service provider, turnover $47,900)
M&E instruments
Joint Evaluation Mission Internal Evaluations

Description
Evaluation mission towards the end of the project, composed of evaluator appointed by TdH and staff of RAID Annual internal evaluation by RAID staff, requested by TdH. Focuses on achievements and qualitative aspects Review made for the Secretary for external reporting (TdH) Reports made by field staff, submitted to supervisor and Director. RAID follows the GoI system to assess the performance of SHGs

Purpose

Refers to:

Taskforces formed for different Evaluated the project on the aspects of the project. basis of the project document, Output: Joint Evaluation Report including the logframe Taskforces established for different aspects of the project. Output: Internal Evaluation Report Review made by Director RAID on the basis of 6 month consolidated reports of field staff. Consolidation of monthly activity reports by field staff Assessment of SHGs with the help of the standard GoI list of indicators. Cross-checking and consolidation of monthly reports of field staff, to be submitted to Director. These reports are discussed with the supervisory staff concerned. Overview of activities undertaken per month Annual plan per project

6 monthly Activity Monitoring Report 6 monthly field staff reports Quarterly assessment of SHGs

Annual plan per project

Monthly plans per sector and annual plan per project. Operational standards set by GoI for SHG to have access to services Monthly plans per sector

Monthly reports by co- Report of activities and ordinators achievements per month per sector, made by co-ordinators

Monthly reports by field staff Weekly staff meetings Reports of field visits educational centres Reports of specific activities Informal contacts with target group

Report of activities and achievements per month by each fieldworker. Meetings held by supervisory staff with their field staff Summary of findings of education supervisor

Monthly plans per sector Monthly plans per sector Operational standards defined in programme description Monthly plans and project plan

Opportunity for field staff to present progress and problems Visits to the education centres, summary made on the basis of notes and observations Completion reports of specific Description of activity and activities (workshops, training) outputs A variety of informal contact with groups or individuals of the target group during ad-hoc discussions, community play, training, field trips and social activities

Source: detailed study RAID and interviews.

Despite variations, the overall set-up of monitoring and evaluation in most organisations shows considerable similarities. These similarities extend across the three countries studied, as well as across the different types and sizes of organisations in the sample. Common characteristics of monitoring and evaluation are: All partner organisations operate a hierarchy of internal reporting often based on recording systems maintained by operational staff (activities, use of inputs, outputs realised). This system of reporting is embedded the line of management and in a hierarchy of progress meetings, where the information is discussed and validated and used for management decision making and accountability. There is considerable attention for different types of internal evaluation. In many organisations regular internal assessment of progress and discussion of lessons learned is nurtured as a key organisational characteristic. In partner organisations operational level staff and middle management are often quite close to the target group and other external stakeholders. This proximity enhances trust and the regular exchange of information between staff of the partner organisation and the target group. In some organisations, promoting proximity to the target group is a deliberate organisational strategy.

41

5.6 Contribution of CFOs to monitoring and evaluation in partner organisations Strengthening monitoring and evaluation systems is increasingly perceived as a significant element of the strengthening of partner organisations. CFOs value effective monitoring and evaluation systems of their partner organisations for a variety of reasons: The effectiveness of management of partner organisations depends partly on adequate monitoring and evaluation systems. The capacity to learn and innovate of partner organisations, and hence their continued relevance, depends partly on adequate monitoring and evaluation. The partner organisations report to CFOs on the basis of their own internal monitoring and evaluation systems. The type and quality of the information that CFOs receive depend, among others, on effective monitoring and evaluation systems of POs. CFOs emphasise that a well-functioning M&E system is primarily of interest of the partner organisation itself. Within partner organisations this perception is increasingly shared, as is shown by the response of the management of Fedina (India): In the past it was felt that the annual and 6-monthly reports were made mainly for the benefit of the CFO. Now it is realised that these reports could especially fulfil an internal function as well, to support review of problems and opportunities and reflections and document experiences. Capacity building depends on both the needs of the individual partner organisation and the kind of relationship with the CFO involved. Altogether 46 out of the 60 partner organisations studied report general support to capacity building by CFOs. This included support in technical/ developmental issues, organisational and management issues, planning and monitoring and evaluation. With regard to monitoring and evaluation it includes training, the financial support to establish a computerised monitoring and evaluation system and developing M&E systems. Box 5.9: Support to monitoring and evaluation by Cordaid
Cordaid noted the difficulty to monitor the food security programme i.e. how to collect data/information, and what tools to use for collecting and analysing data. CORDAID decided to assist the programme by introducing a simplified monitoring and evaluation model including a simple project monitoring procedure and examples of a monitoring system with: (a) Inputs monitoring (b) Daily activity monitoring (c) Result monitoring (d) Indicator based monitoring. The model has proved to be useful not only for the food security programme but also for other programmes. IDRO has extended this model to other programmes. The internalisation of the model will contribute substantially to the understanding of result-based management and its application in practice.
Source: Study Report IDRO (Tanzania, intermediary organisation, turnover $1.100.000)

CFOs reported efforts to strengthen M&E in partner organisations. This varied from support to individual organisations to more structural support. An example of the latter approach is presented in the following box.

42

Box 5.10: Support to monitoring and evaluation by Novib in Peru.

In 2000-2002 the Escuela para el Desarrollo (Lima, Peru), at the request of NOVIB, executed a project to strengthen the PME systems of NOVIBs partner organisations to enhance their role for project management. The project was implemented in 3 phases. In the first phase 13 POs conducted a self diagnosis after their staff had been trained. The self-diagnosis covered four main areas: situation of capacity building (defining results, learning/training, development of techn. systems), organisational culture (focus on results, concern for human resources), development of planning systems (usefulness, techniques, quality of processes), development of M&E systems (usefulness, techniques, quality). During the second phase changes of already existing PME systems were initiated by means of consultancies (10 P0s) taking as a starting point the results and recommendations from the institutional self diagnosis. Workshops were conducted to train staff. During the third phase the improvement of the PME systems of participating POs was completed. The project generated a number of products useful for the strengthening of development NGOs: an innovative conceptual framework for strengthening PME systems based on the P0s own possibilities and limitations; a methodology to diagnose the state of PME systems and the way they are linked to other dimensions of management in the organisations; a set of tools for training in the development of PME systems; a methodology for the design and formulation of PME systems; an approach for the definition of indicators for development projects.

Many donor organisations have contributed to the introduction of the logical framework for planning of activities. The use of the logframe is now common practice among partner organisations. Most partner organisations perceive that this has contributed to enhancing monitoring and evaluation. 5.7 Use of the information by partner organisations Partner organisations use the information collected for accountability, day-to-day management decision making and learning. Accountability Accountability includes both the internal accountability to senior management and the Boards of the organisations, as well as to the donors. Partner organisations consider themselves accountable to donors, the National Governments, target groups and the wider public. For this purpose, partner organisations use the information provided by monitoring and evaluation of their activities. In many of the case studies and during the interviews with resource persons it was observed a substantial part of the monitoring and evaluation efforts of partner organisations are motivated by the accountability requirements towards the CFOs. The majority of partner organisations has several donors. More than half of the POs studied reported that there are no fundamental differences between requirements of the donors in terms of content: most donors require reporting on inputs, progress and results. In cases where donors fund specific activities this defines the scope of the reporting. The difference in requirements can mostly be found in the frequency and format of reporting. The Dutch CFOs require bi-annual or annual reporting, whereas for instance USAID and the EU may require monthly reporting. Most POs are required to use different guidelines and formats for the narrative and financial reports for each of the donors. Only few partner organisations (the local Plan organisations and two other POs (HakiElimu and IDE/I)) use their own reporting formats, which are fully accepted by all their donors. Two partner organisations stated that they use the reporting guidelines of the CFO throughout the organisation, also for the programmes that are not

43

funded by the CFO. The vast majority of the POs studied send different progress reports to each of the donors. Partner organisations consider reporting to donors as a normal part of a donor-recipient relationship. Reporting requirements of the Dutch CFOs are generally perceived to be acceptable and not excessively burdensome when compared to other donors. A frequent observation is that the Dutch CFOs have become more demanding over the last few years, specifically with regard to information about the results that are being achieved. The partner organisations reported that the CFOs have been instrumental in ensuring that proposals for activities and progress reports are more result focused. The guidelines drafted by the CFOs are reported to be more result oriented than those of other donors, who tend to be more interested in the proper use of the funds that are made available. Monitoring systems in partner organisations provide detailed information about the use of inputs, the execution of activities and the outputs realised. This information is based on internal registration systems that are part of the management information and control systems. This information is used for internal accountability and also for reporting to the donor. Not all information provided by monitoring and evaluation, however, is amenable to be reported to the CFOs for accountability. Specifically the information about outcomes and impact collected through direct contact with the target group and observations during field visits often does not find its way into the regular reports to the CFOs. This is mainly due to unknown representativeness and reliability of the information. However, it may be discussed selectively during visits of staff of the donor organisations. Impact information obtained through the internal hierarchy of reports and also by means of impact studies and surveys appears to be more suitable for reporting to CFOs. Approximately one-third of the partner organisations observe that they have no clear perception on how the information provided is used by the CFOs. Some respondents indicate that neither the written reaction to the progress report nor the discussions held during the visits of CFO staff to the partner organisation give an indication to what extent and for what purposes the information is needed. Others assume that the information is necessary for the accountability towards the Ministry. Most partner organisations observe that the feedback received from the CFOs on their narrative progress reports is limited. The boards of the partner organisations play different roles in the leadership of the organisations. As a rule, boards convene quarterly (in exceptional cases monthly). Their main role is to review overall progress, often on the basis of the internal quarterly or half-yearly reports. In about a quarter of the organisations visited, it was reported that boards make field visits as part of their quarterly or half-yearly meetings, in order to gain first-hand information about the progress of activities and the perceptions of target groups. Boards have to approve all reports that are made available to external parties, including the financial and narrative progress reports to donors.

44

Box 5.11: Monitoring by the Boards of partner organisations

SIFFS (India, membership organisation, turnover $1.850.000) SIFFS board meetings are held regularly and its minutes are recorded. The Board discusses all important matters like planning, budgeting, reporting, new contract with donors and important internal issues including staff policy. All activities reviewed in detail every month by Board based on monthly reports RAAA (Peru, network organisation, turnover $337.000) The national board (concejo nacional) of RAAA meets twice a year for tow days. The meetings have an evaluative character. They receive reports from the national co-ordinators (executive reports, technical and institutional development reports). The national board meetings address the following issues: conceptual discussions, planning, monitoring and evaluation.
Source: Reports of detailed studies of partner organisations

In less than half of the organisations studied it was reported that target groups are represented in the boards. Decision making All partner organisations have institutionalised a system of regular meetings from operational level to management team to discuss progress and take decisions. The use of information is facilitated by the fact that activity plans are often based on the use of the logical framework, which provides a direct link between planning and feedback. Quarterly, half-yearly and annual reports are also used by Boards or General Assemblies for decision making on policy or strategy matters. Learning Many partner organisations explicitly consider learning to be an important aspect of their organisational culture. Most have created opportunities to review experiences and draw lessons from their operational practice. A few make it a deliberate policy to enhance the external exposure of their staff by encouraging participation in conferences or seminars. Learning from experience takes place often during internal evaluation workshops and through contacts with other organisations. Respondents indicate that factors that influence learning positively are the committed and positive organisational culture (dedication to the objectives of the organisation) and, in most organisations, the short, informal lines of communication. Negative factors are the sometimes inadequate capacity and capabilities for analysis, the work pressure, the activist orientation of staff and the sometimes frequent changes of staff.

45

Box 5.12: Illustrations of learning in partner organisations.

HCHW (India, intermed. org., turnover $109,000) The organisation started as a project of students from an academy. They knew very little about the practical situation of street children and had to learn from the start. The organisation has made deliberate attempts to maintain that learning culture. Learning is enhanced by monthly meetings, where information is shared among all staff, field visits (all staff, also administrative and management staff is expected to visit the centres for street children regularly) and participation in events (workshops, seminars). Every two months a staff member is asked to give a presentation. MYRADA (India, intermediary organisation, turnover $ 5.400.000) MYRADA is constantly reviewing its own programmes. Staff is frequently exposed to international and national experiences. Office staff is requested to visit the field regularly and attend group meetings to be exposed to the reality of the target groups. MYRADA considers peoples demands a very important vehicle for learning. Every 12 weeks there is an internal workshop to reflect on a specific issue: partly training, partly evaluation. SACCS (India, intermediary organisation, turnover $669.400) SACCS produces guides and booklets with tips and best practices for its field staff and for other NGOs working in the same field. These documents are to a considerable extent based on its own experiences. SACCS is currently drafting a manual based on its experiences with conducting campaigns and doing advocacy work. INCA (Peru, intermediary organisation, turnover $230.000) Inca attaches great importance to internal learning. It distinguishes two important aspects to foster the process of internal learning: To emphasise the need of conscious observation of what is happening to the target groups as well as in the surrounding area, and to interpret this in the light of Incas vision and values. All staff members should be ready to learn from the target group. Incas management promotes such attitudes, regularly stressing their relevance and stimulating active observation. Regular analysis of experiences gained within the organisation and interaction with outsiders (experts, universities, etc.). The analysis should enable the formulation of new policies or strategies. It is important to create opportunities in the organisation in terms of both time and motivation. Incas management establishes contacts with outsiders and takes the lead in the performance of the analysis. The analysis itself is made during internal workshops with the participation of all staff members. CORDS (Tanzania, grassroot organisation, turnover $200.000) CORDS wants to be a learning organisation. This is promoted by: Keep staff informed of developments in the international debates on development. New and relevant developments in thinking are shared within the organisation, sometimes it is translated in Swahili; Staff should consider itself as an agent of change; share information with the target population and using information from them. CEDEP AYLLU (Peru, Intermediary organisation, turnover $765.000). Since 2001 the organisation attempts to systemise experiences in a more structured and collective manner, looking at what they are doing well and wrong. Sometimes experiences are published. The Annual Assembly, with the participation of all teams, is considered the most in-depth exercise of learning and sharing of experiences. A report of all the discussions is made. The final product is the new Annual Plan. Source: meetings with IOB-team

5.8

Analysis

In partner organisations a diversity of instruments exists that together constitute the organisational monitoring and evaluation systems. The following table presents an overview of these instruments.

46

Table 5.10: Overview of monitoring and evaluation instruments in partner organisations

Feedback from
Monitoring Evaluation

Instruments
Registration systems Hierarchy of reports from the operational level up to management and the Boards of the organisations. Often reports use predetermined formats/guidelines. Field visits, including reporting and debriefing to present the findings of the field visit to superiors and colleagues. Workshops and consultations where external stakeholders and operational staff participate. Regular contacts by e-mail and telephone between operational staff and headquarters. Progress meetings at different management levels in the organisation Regular unstructured contacts of staff of different levels in the organisation with stakeholders, including target groups. Evaluation of an activity or several activities by an external evaluator, at the request of the partner organisation. Internal activity evaluations Self evaluation workshops (often internally, sometimes externally facilitated). Impact studies.

Some of these monitoring and evaluation instruments are formally described in guidelines or instructions for operational staff. Others are not in writing, but exist as generally accepted routines, or by explicit oral agreement. The systems cover all aspects of the activities of partner organisations: the use of inputs, the execution of activities, the realisation of direct outputs and the achievement of outcomes and impact. With regard to the quality of the information (validity, reliability and verifiability) provided by monitoring and evaluation, a distinction should be made between operational information (concerning the use of inputs, the execution of activities and the realisation of direct output) and information about the achievement of effects and impact at the level of the target group. Operational information is collected using well defined recording systems, that are subject to systematic internal checks by higher levels in the organisations (through field visits/observation, information from other stakeholders and consistency checks). The information is generally valid and well structured, making it suitable for communication to other stakeholders, including donors. Internal processing of the information is limited to different steps of consolidation or summaries with little analysis done. The information is relevant for operational management control and internal/external accountability: it directly relates to the project plans or annual work plans of partner organisations. For the information about effects and impact a distinction should be made between the organisations where the collection of such information is regular and standardised (11 out of the 60 partner organisations studied) and where it is unstructured. For the 11 partner organisations the information about effects and impact is structured, suitable to be communicated to external stakeholder including donors, and it is often shared with them. The quality of the information, however, is variable. The information is generally valid, but because of the sometimes limited professional skills and independence of the researchers, the reliability of the analysis is uncertain. Most organisations systematically collect information about effects and impact on the target group (i.e. subject to organisational rules to promote contact between staff and the target group), but in an unstructured way: collection of the information is largely unstructured, and selection of respondents not random. 47

This generates a wealth of information about the effects and impact of activities on the target groups and about the perceptions of the target group on the activities of the partner organisation. The information is considered to lead to a deeper understanding by staff. The information is usually available at the level of field staff and middle management, and sometimes senior management. It is usually not in writing, fitting with the oral traditions of the general culture and the organisations. This makes it difficult to communicate to donors in a meaningful way. The interpretation of the information is done by the staff involved. This, and the possible bias of resource persons make the validity and reliability of the information about outcomes and impact uncertain. Partner organisations require information for three main purposes: accountability (internal and to feed the accountability to external stakeholders), management decision making and learning. The relevance of the information provided by monitoring and evaluation for these three purposes varies. The information generated by monitoring instruments about inputs, activities and direct outputs is relevant for internal accountability and for accountability to donors and other external stakeholders. The information collected about outcomes and impact plays a less pronounced role in accountability of operational levels to management, but it often plays a role in the communication with senior management and the Boards of partner organisations. The scope of the information makes it in principle suitable to support strategic decision making by either senior management or the Boards/General Assemblies of the partner organisations. The uncertain validity and reliability of much of this information, do not seem to influence its usefulness as perceived by senior management. Information about outcomes and impact collected through structured studies and surveys feeds to some extent into the communication with donors (see chapter 6), but the outcome and impact information that is collected in an unstructured and ad-hoc manner is usually not shared with them, despite the fact that it is relates to project documents or work plans. For monitoring and evaluation instruments providing operational information (including direct output), an assessment was made for each of the organisations studied. The assessments show a high degree of homogeneity, across the countries studied, the sizes of the organisations and the type of organisations (for detailed breakdown see appendix 5).

48

Table 5.11: Distribution of assessments for evaluation criteria of the 60 partner organisations studied.
Positive Collection of information Regular 56 Standardised 48 Relevance for use Accountability 54 Decision making 56 Learning 49 Can information be transferred to other stakeholders 52 Quality of information Validity 48 Reliability 15 Verifiability 42 NB: * Information about the reliability of information was incomplete Present but not fully satisfactory 3 11 5 2 7 7 8 14 11 Negative No information 0 0

1 1 1 1 2 1 1 1 2

1 2 0 3 30* 5

From the study of partner organisations the following conclusions emerge: Elaborate monitoring and evaluation systems allow management of the partner organisations to exercise relatively tight control over the implementation of activities. Internal reporting about the use of inputs, the execution of activities and the realisation of direct outputs is generally regular and standardised, with a high frequency of reporting. With regard to effects and impact on the target group the collection, analysis and communication of information is regular and standardised in approximately 20% of the organisations. In the remainder it is sometimes systematic (subject to rules in the organisation), but not standardised. Monitoring and evaluation are usually well connected with the processes where use is made of the information. This applies specifically for the operational aspects. Monitoring information feeds directly into regular progress meetings where the information is discussed and decision making takes place. Internal reviews and evaluations are often directly linked to workshops where experiences are analysed, lessons drawn, and new plans formulated. The relatively small size of many organisations, with correspondingly short lines of communication promote the connectedness of data collection and analysis on the one hand and the use of the information on the other. Most partner organisations consider learning from experiences and from experiences of other organisations a core value of the organisation, and have well established routines to learn, partly on the basis of their own experiences, partly based on external information inputs. Organisations show a high level of dedication to their cause. If it appears to be necessary to adjust operations or strategies in the interest of this cause, there is usually great willingness to do so. Specifically the information provided by staff about their own experiences and their interpretations of achievements are used for learning purposes. Resource persons do not perceive the uncertain validity and reliability of this information as a constraint for learning the lessons that the information contain. Accountability to the donors covers mainly inputs, activities and direct outputs. Much of the wealth of unstructured information about the effects and impact of the interventions appears to be unsuitable for standardised and structured communication with the CFOs. This information is difficult to report and interpret.

49

50

6.

MONITORING AND EVALUATION BY CFOS

This chapter describes the monitoring and evaluation by the CFOs. A more detailed presentation of the findings is presented in appendix 6. 6.1 Organisation of monitoring and evaluation

All CFOs, have a project or programme department28 in charge of the funding activities of the organisation including the support and supervision of the partner organisations. These departments are usually subdivided into regional units. In the regional units, teams, composed of at least a programme officer and a financial officer, are responsible for the monitoring and evaluation of the partner organisations. Some of the CFOs supervise their partner organisations through regional offices (Cordaid, Hivos for 2 resp. 3 regions, TdH for all partner organisations). Plan NL differs from the other CFOs because it works through the Plan country offices. The staff of Plan supervises the activities of the Plan country offices. In the future, Plan NL intends to work more with local organisations to implement its activities. Because of this change, the interrelationships between Plan NL, the Plan country offices and the implementing organisations have to be re-established. Programme staff in the regional units have on average 15-30 partner organisations in their partner portfolio. A considerable amount of time is spent on the preparation and approval of new agreements. When taking into consideration the time required for other activities, the time available for supervising the relationship with partner organisations amounts to an estimated 2-4 working days per year, including the yearly visit to partner organisations. All CFOs have management manuals describing the processes and procedures within their organisation (also required for the mandatory ISO certification). These manuals include the principles and procedures of the funding cycle (from first contact with partner organisation to termination of relationship) and CFO management cycle, and describe the procedures for monitoring and evaluation: actions taken, instruments used, responsibility of actions, output of actions. It specifies in what way the feedback information obtained from progress reports, visits and evaluations should be processed. Within CFOs, processes for operational feedback are noticeably more developed and better structured than processes for policy level feedback: ISO certification concerns predominantly the primary process related to the funding and activity cycle. All CFOs have computerised management and financial information systems, which, among others, store key information of partner organisations. With the exception of TdH, CFOs have units responsible for monitoring and evaluation. These units comprise between 1-3 staff with specific responsibility for M&E. In co-financing organisations two (interrelated) monitoring and evaluation systems can be distinguished: The internal monitoring and evaluation of the CFO policy and management cycle. The monitoring and evaluation of the activity cycle of the partner organisations.
28

Project of programme departments: Cordaid: Project component, Hivos: department Programmes and Projects, ICCO: Programme Management Component, Novib: department Projects, Plan: unit Projects and Programmes, TdH: Department Projects and Campaigns.

51

The scope and coverage of monitoring and evaluation change with the perceived roles and responsibilities of CFOs. It is focused on domains in the aid chain for which the CFOs consider themselves responsible. Increasingly, the achievement of effects and impact at the level of the target groups is considered primarily the responsibility of the partner organisations. The example of Cordaid in the following box provides an illustration. Box 6.1: Scope of monitoring and evaluation, Cordaid.
Cordaids M&E system, approved in 2002, is based on the concept of the aid-chain consisting of a series of autonomous organisations. Cordaid has identified four domains: target group, PO, relationship PO-Cordaid and Cordaid. Cordaid bases its M&E system on its own role and responsibilities in the aid chain. Cordaid can directly steer in the domain Cordaid, and therefore be accountable for actions and results in that domain. Steering and accountability in the domains relationship PO Cordaid and PO are less direct. As a consequence, Cordaid has tentatively identified a set of indicators for those three domains. Cordaid perceives that it cannot be directly responsible for the domain target group.

6.2 Internal monitoring The CFO-internal monitoring is based on the hierarchy of plans from the strategic 4year management plan and sectoral plans through the annual plans to the annual regional or country plans. In most CFOs the annual plans result in agreements between the director of the programme department and the head of the regional units or offices specifying portfolio characteristics, target groups, capacity building, inputs and process. In the course of the year, higher management monitors progress of the implementation of the agreements, by means of internal meetings and internal reporting. Most of the indicators that have been defined are input and process indicators, covering aspects such as disbursements per sector or theme, visits to partner organisations, participation in conferences, and selection of new partner organisation, termination of existing partner organisations. For the regional units in the programme department of the CFOs, the annual regional or country plans are the basis of the monitoring. Each year the regional and country plans are operationalised in annual work plans. The work plans state objectives, input and results for the CFO and the partner organisation. Where CFOs operate through regional offices, procedures are usually in place to monitor their performance, through operational audits or other mechanisms. The following box provides an illustration of the supervision of regional offices by TdH. Box 6.2: Monitoring of regional offices by TdH
TdH headquarters monitors the performance and management of the regional offices. The regional offices regularly report to headquarters through reports of field trips by staff of the regional office, regular (monthly) extensive discussions by telephone between headquarters and regional offices, quarterly and annual progress and financial reports and an annual meeting at headquarters. Every regional office receives an operational audit team from headquarters once every two years. The aim of this audit is to assess the administration and management of the project portfolio by the regional office as well as the efficiency of the use of funds. A secondary objective of these operational audits is to verify the achievement of TdH objectives by the portfolio of projects29.

29

Management plan 2003-2006, p.20.

52

6.3 Monitoring of partner organisations The monitoring of partner organisations is based on the activity cycle covering all phases from initial activity proposal to termination of the relationship. During the appraisal of the proposals, key points for monitoring (results, milestones, strengths and weaknesses, threats and opportunities) are identified. The partner organisations of Cordaid are requested to make use, whenever possible, of indicators, that are agreed upon during partner consultations and are based on result areas. To assess results at the level of the partner organisations, Cordaid occasionally uses a scan that includes the following criteria: positioning (the rationale of the organisation), thinking and learning (the capacity to analyse and learn), doing (the capacity to execute relevant activities efficiently), being (the structure of the organisation) and relationships (the role of the organisation in its context). Hivos new agreements with partner organisations (introduced October 2004) include a limited number of manageable and measurable indicators that are identified in consultation with the partner organisation. These indicators are based on the objectives and expected results of an activity and are monitored during the contract period. The monitoring of Hivos focuses on two types of objectives: the objectives of the partner organisation with regard to development on Hivos central policy domains and objectives regarding capacity development of the partner organisation itself. The threats and opportunities identified during the appraisal phase are also monitored. During the activity management cycle, ICCO drafts several memoranda. The kidbemo prepares the decision making for funding. Key points for monitoring, based on the appraisal of the activity and the partner organisation, are defined in the conditions of the agreement. In the period 2001-2003 Novib has developed a Toolbox that is used to identify potential partner organisations and for the appraisal and approval of a grant. It constitutes the basis for the follow-up and review of progress. The expected outcomes as well as the risks, defined by the Toolbox, are the basis for the definition of milestones. These milestones, which are incorporated in the agreement, are used for the monitoring of partner organisation by the programme staff. The Toolbox is used since late 2003-early 2004. Plan NL monitors the activities and programmes, to a large extent on the basis of the information included in the CPME30. The monitoring focuses on the following levels: the hierarchy of objectives of the activity or programme concerned, the objectives on poverty alleviation of Plan NL (including regionalised objectives) and the relationship with the Plan country offices. The monitoring and evaluations of the partner organisations (other than the Plan country offices) is a new concept in Plan NL. Until recently, partner organisations were mainly monitored and evaluated on their activity implementation capacities, and not on the institutional development and civil society building. TdH has introduced the Objective Oriented Project Planning approach, that emphasises the definition of activity objectives and related measurable indicators for monitoring and evaluation. Monitoring by TdH has the specific aims to assess that the executing team is aware of the activity objectives, that execution is according to plan, that activities contribute to the achievement of objectives and that the administrative organisation and expenditures are according to approved budget.

30

CPME: The Corporate Planning, Monitoring and Evaluation System of Plan International.

53

6.3.1 Reporting by partner organisations All CFOs require regular progress reports from their partner organisations and issue partner organisations with guidelines or formats to structure these reports.31 The structuring of the progress reports is intended to enhance the uniformity and quality of the reports and facilitate their processing. Hivos and ICCO state in their guidelines that the reporting guidelines primarily serve needs of management of the partner organisation itself. Hivos does not prescribe a mandatory reporting structure, but allows its partners to develop their own formats or use formats required by other donors, in order to limit the administrative burden on partner organisations. If partner organisations have developed an adequate reporting structure for other donors or internally, all CFOs, with the exception of Plan NL, may (in practice) agree to use this reporting format. The format for the annual report developed by Plan NL, has been standardised for all of Plans organisations. It was reported that this enhanced the quality of the information in the annual reports. Unless otherwise agreed, the partner organisations of Cordaid, Hivos, ICCO and Novib are required to submit an annual narrative progress report, a financial report and an audit report. Plan NL requires an annual narrative progress report and quarterly and annual financial reports. The partner organisation of TdH have to submit a bi-annual monitoring report, a bi-annual financial report and an annual audit report. 32 The partner organisations of regional TdH office in India have to submit bi-annually an activity monitoring report and annually an internal evaluation report. The guidelines for the biannual progress reports of the other regional offices are similar to the guidelines of the internal evaluation report: analysis of the performance during the period, are activities implemented as planned, targets reached and problems that are encountered as well as suggestions for adjustments for next year. An overview of the reporting requirements of the CFOs is presented in table 6.2. Reporting requirements of the different CFOs appear to be largely similar in content. They do differ, however, in the format required, as is shown in appendix 6. Assessment of progress reports An assessment is made of the annual progress reports of the partner organisations studied in the three countries (see appendix 6). It shows that these reports generally comply with the requirements formulated by the CFOs. The reports are generally strong on description of progress and output, but less informative on relating progress or outputs to plan/agreement, explaining any deviations and specifically short on description of effects and impact of the activities on the intended target groups. A general observation is that the more explicit the guidelines or format provided by the CFOs, the more the information presented will comply with the requirements.

31

The various regional offices of TdH use different reporting guidelines. TdH is planning to develop unified reporting guidelines for the entire organisation. 32 An annual audit report is only required for the projects that receive more than 35.000 of TdH. In India an annual audit report is required by law for all projects receiving funds abroad.

54

Table 6.1 Overview of assessment of progress reports for CFOs

Characteristic Cordaid Contents Presentation of context Presentation of target group Presentation of organisation External relations, contact stakeholders Description of progress of activities Relating progress to plan or contract Explanation / discussion of deviations Constraints/ remedial action taken Lessons learned Results at level PO Realisation of output Relating output to plan or contract Results at level target group Results at level target group Number of progress reports assessed % satisfactory among partners of Hivos ICCO Novib Plan TdH NL 75 33 67 75 92 25 42 58 67 100 42 8 12 33 22 33 67 89 44 3 67 67 100 44 22 9 57 57 71 100 93 29 21 65 50 100 36 21 14 100 40 60 80 80 80 80 60 100 100 80 20 5 25 25 100 50 75 50 0 100 75 100 75 50 4 total

64 43 50 64 100 79 36 29 29 100 64 43 14

60 40 60 76 91 48 35 57 57 100 52 26 58

Analysis of the annual progress reports shows that the reports of the 11 partner organisations that regularly conduct impact assessments or surveys (see chapter 5) are more informative on results at the level of the target group: assessment of the progress reports of these 11 organisations indicated that 55% of the reports are considered satisfactory with regard to provision of such information. Partner organisations that obtain information about results from more informal sources rarely include this information in their annual progress reports (satisfactory in only 19% of the remaining 49 partner organisations). All of the progress reports analysed are progress reports of the years 2002-2003. Since then, reporting guidelines and contracts have been revised. It is anticipated by the CFOs that reporting on results, milestones and contract conditions will be more explicit over the coming years. Although progress reports are considered important monitoring instruments, often most information in the progress reports has already been communicated between the programme staff and the partner organisation in a more informal way (by e-mail, during visits to partner organisations or visits of partner organisation staff to the CFO, or during partner consultations). 6.3.2 Other forms of monitoring

In addition to the annual progress reports, CFOs maintain regular contacts with their partner organisations. These regular contacts play an important role in cementing the partnership between the CFO and its partner organisations. They also serve as monitoring instruments to track the activities and progress of the partner organisations. CFOs explicitly recognise the important contribution that information collected by direct contact can make for monitoring. Visits CFOs consider visits to their partner organisations important for monitoring. These visits are used to exchange information with staff of partner organisations, discussing and solving operational problems, giving feedback on reports and to maintain contact 55

with the target groups, to obtain an impression of the impact of the activities of the partner organisation. TdH and Plan (country offices) have a high frequency of visits to partner organisations. TdH deliberately clusters its partner organisations geographically around its field offices, but also staff from headquarters visits the partner organisations regularly. In the regions where Hivos has regional offices, contact with partner organisations is relatively intensive. Other CFOs rely mainly on regular missions to visit their partner organisations, the general rule being that partners are visited at least once a year. In practice, visits are more frequent, but by different staff from the CFO headquarters. To intensify its contact with partner organisations, Cordaid has instructed its programme staff to extend their missions considerably. ICCO has increased both the frequency and the intensity of contacts (including visits) with their partners. From these visits reports are drafted. These reports are discussed with middle level management and action is taken if required. Partner consultations All CFOs conduct partner consultations. Approximately, once a year Cordaids regional units organise partner consultations at regional or continental level with strategic partners active in a thematic field. The consultations provide a framework for exchange of ideas and experiences focused on specific themes, with a depth that goes beyond the regular reporting system and the ad-hoc contacts that are usually concerned with day-today management issues. ICCO conducts partner consultations at local, regional and international level. These can have a country or a thematic focus and usually concern policy level. It is estimated that approximately one-third of the partner organisations participate occasionally in such consultations. Novib conducts regular consultations with selected partner organisations, partner platforms and different types of meetings. Regional TdH offices regularly organise workshops for staff of partner organisations about various subjects, either thematic or of a more managerial nature. In 2004 Hivos, apart from its regular partner consultations, has conducted 5 regional consultations on result measurement. Other monitoring Regular contact (by email and telephone) takes place between programme staff and partner organisations on a great variety of subjects. The aim of this regular contact is to provide more general support and to build a relationship between the CFO and the partner, but the information obtained also serves the purpose of monitoring. CFOs often receive other information from some of their partners, for example activity evaluation reports executed by other donors, research studies etc. Monitoring by regional offices Some of the CFOs have regional offices in countries where they work. Cordaid has two regional offices in Africa, in Nairobi (Kenya) and Kinshasa (Congo). Hivos has regional offices in Bangalore (India), San Jose (Costa Rica) and Harare (Zimbabwe). TdH has regional offices in Cochabamba (Bolivia), Nairobi (Kenya), Colombo (Sri Lanka) and Jakarta (Indonesia). In some of the countries where it operates, TdH has country representatives working under the responsibility of the regional offices. In principle Novib does not work with regional offices. However, in specific situations Novib deploys a field officer. Recently this has been the case in Afghanistan and in the Great Lakes region. 56

The value added of a regional office is reported to be that it is better informed about the local reality and the social and political context. Programme staff of Hivos and concerned partner organisations state that contacts with partner organisations is more intense because of the field offices. The intensity of visits is higher, also because the CFOs and partner organisations meet at different occasions. Visits are reported to last longer (several days vs. half to one day during missions of headquarter staff). An evaluation of the Cordaid regional office in Nairobi concluded that proximity has evidently facilitated increased visits and more frequent face to face communications between the regional office and partners. The . Partners also put strong emphasis (appreciation) on the visits of programme officers in the last one and half years.. According to the partners, Cordaids presence in the region has made it easier to discuss problems more frequently. and expanded possibilities for consultation in shaping the projects better. The evaluation also notes a potential threat: that too close presence risks that the CFO assumes too much responsibility leaving too little ownership to the partners, a situation has largely not occurred in the Cordaid Nairobi office because of the presence of adequate checks-and-balances.

57

Table 6.2: Overview of the information requirements of the CFOs

Changes in context New developments in organisation Progress Output Explanation deviations Constraints/ remedial action taken Result at level target group Lessons learned Cordaid Context information required Organisational information required Reporting on activities carried out as compared to activities planned Comparison of planned and realised activities and output Not specifically mentioned in requirements. Forecast and planning for next period is asked Interim position achieved in period in relation to objectives, intermediate and final goals Assessment by the organisation responsible, degree to which the objectives have been achieved. Information on networking, relationship of partner with government. Not mentioned in requirements. Hivos Context information required Organisational information required Summarisation of implemented activities Foreseen and unforeseen output. Comparison output with proposal. Explanation of major differences. Constraints and remedial action/ any changes in plan. Not specifically mentioned in requirements. Lesson learnt and followup asked. Information about internal and external evaluation. Relationship with members, changes for target group, donors, local authorities. Attention for gender aspect. ICCO Context information required Organisational information required Analysis of the operational plan Achievement of expected results, analysis of effect of executed activities. Explanation of any deviations in planning and results. Changes in the operational plan Achievement of expected results, analysis of effect of executed activities. Not specifically mentioned in requirements. Changes in relationship with other parties. Attention for gender aspect. Novib Context information required Organisational information required Progress of milestones and contract conditions. Give output in comparison with planned output. Reasons for deviations, name internal and external factors. Anticipated threats. Action plans as results of lesson learned. Achievement of multiannual goals. Look at milestones, indicators of progress. Lessons learnt on activity and organisation level relevant for Novib Not specifically mentioned in requirements. Attention for gender aspect. Plan Context information required No information required of local Plan itself. Information required on partners. Description of main activities Direct output in activity table. Reasons for differences in activities achieved and planned, and outputs. Problems that have impeded progress and remedial action. Assessment of progress toward objectives Lessons learned, changing methodology for similar activities, dissemination of lesson learnt Description of involvement of partner , of relationship, of target group, of children. Attention for gender aspect. TdH Context information required. Organisational information required. Realised activities towards objectives. Achievement of goals. Use of indicators. Explanation of deviations required. Problems and difficulties. Recommendations. Plans for near future. Achievement of goals. Use of indicators. Not specifically mentioned in requirements. Description of involvement of community, of children. Not mentioned in requirements.

External relations/ stakeholders Gender

58

6.4

Evaluation

CFOs conduct activity-level33 and programme evaluations. Activity-level evaluations focus on one activity or partner organisation. Programme evaluations focus on several activities/partner organisations, often in more than one country. The programme evaluations are the subject of a separate IOB assessment. 6.4.1 Activity-level evaluations In principle the CFO is responsible for formulation of the terms of reference and selection of the consultants for activity-level evaluations. In practice, however, this applies only to situations where problems are expected or relationships are difficult. Usually terms of reference and the choice of the evaluators are the subject of discussion between the CFO and the partner organisations concerned. While these evaluations are usually initiated by the CFOs, the partner organisations often take the lead in formulating the terms of reference and finding the evaluator. Cordaid has to approve the terms of reference and the team of evaluators, but leaves much of the management of the evaluation to the partner organisations. Hivos specifies in its agreements that it is responsible for the formulation of the ToR and the selection of the evaluator, and effectively does so, in consultation with the partner organisation. ICCO follows different modalities. Sometimes parts of the responsibility for the management of evaluations are sub-contracted to a local organisation. ICCO programme staff then approves the ToR and the team of evaluators. In other cases ICCO is directly involved in the formulation of the ToR and selection of the consultants, and sometimes the ToR are formulated by the PO or the consultants involved. Novib and the partner organisation were, until recently, jointly responsible for the formulation of the ToR and selection of the consultants. In its new evaluation policy, however, Novib shifts the responsibility for activity-level evaluation to the partner organisation34. The activities of Plan NL are evaluated mid-term and towards termination. The mid-term evaluations are initiated and executed by the Plan country offices. For the evaluations at the end of the activity, the ToR and external consultant must be approved by Plan Netherlands. The activity-level evaluations of TdH are joint evaluations with participation of staff of the partner organisation and TdH, occasionally chaired by an external evaluator, who is also responsible for reporting. All CFOs have formulated rules with regard to evaluation coverage. The following table present the rules for evaluation coverage and the number of evaluations executed in 2002 and 2003.

33

The term activity-level evaluation is used to include all evaluations of all individual funding agreements with partner organisations. CFOs use different terminology for this type of evaluations. 34 See paragraph 6.6 (recent developments)

Table 6.3: Overview of evaluation norms and numbers of evaluations realised.

CFO Rules with coverage. regard to evaluation Number of evaluations (# of POs)

In 2002
Cordaid Hivos ICCO Novib A partner organisation has to be evaluated at least once in two agreement cycles. Yearly at least 5 % of its POs General policy is once per funding cycle. All partner organisations have to evaluated every 35 year. Activities that receive over 400.000 in 3 years or over 250.000 in 1 year have to be evaluated. Evaluation of smaller activities is not mandatory.36 All activities are evaluated mid-term and towards the end. Unknown (1869) 48 (766) 87 (883) 123 (861)

In 2003
104 (1252)35 62 (864) 103 (807) 105 (839)

Plan

TdH

All activities are evaluated once per contract cycle of 3 years. Total number of activity-level evaluations
Sources: Annual reports CFOs and correspondence.

Unknown programmes 44 national regional offices) N/A.

(101 with and Plan

22 programmes 44 national regional offices) 46 (255) 442 (4061)

(183 with and Plan

The table shows that every year a substantial number of activity-level evaluations is done. However, although difficult to determine, CFOs do not seem to fully satisfy the rules they have set themselves. The evaluations rely heavily on information already available (collected as part of the regular monitoring and reporting processes), and on a limited number of individual interviews or group discussions with implementing staff and target groups. This provides impressionistic information rather than a solid basis to assess the contribution of activities to complex societal processes. The scope of most of the activity-level evaluations is therefore on organisational and process aspects, rather than on the results of the activity. The objectives of the activities supported by the CFOs often aim at changes in the socio-economic position of the target group, capacity building of individual partner organisations or more general strengthening of civil society, processes of a very complex and often long term nature. Evaluations are rarely designed to address such complex issues. With two exceptions of the partner organisations studied in detail37, external activity-level evaluations lasted between one and three weeks, with fieldwork accounting for less than half of the time. Respondents indicated that such a short duration gives only little opportunity for the collection of meaningful information about the effects and impact on the target group. An analysis of the evaluations of the partner organisations studied in South India illustrates the scope of the evaluations.

35

The total number of partner organisations include the partners in exit phase, short-term relationships of the humanitarian aid unit, one-time funding and micro-finance. The partner portfolio of Cordaids priority themes consists of 1252 POs in 2003. 36 This norm was common practice until 2005 and has been formalised as of 2005. 37 TGNP and Saipro

60

Table 6.4: Overview of scope of 6 activity evaluations in South India

Year Duration (days) External/joint Topics dealt with in report Partner organisation Organisational history Vision/mission Structure Leadership Strategy Organisational culture Organisational learning Staff training Community Based Organisations Relationship Capacity building Withdrawal/sustainability Implementation Financial Programme Effects/Impact On community On organisation Implications for the future FEDINA 2003 3 External SIFFS 2000 5 External Samskar 2003 10 External Raid 2002 N/A Joint HCHW 2003 4 External QWRIDS 2002 3 Joint

+ + + + +

+ + + + + + + + +

+ + +

+ + +

+ + + + + + + + +

+ + +

+ + + + + +

+ + + +

+ + +

+ +

+ +

+ +

+ +

+ +

+ +

+ +

+ +

+ +

+ +

Note: + indicates mention in report Internal analysis of activity-level evaluations The CFOs (with the exception of TdH) have recently assessed the quality and relevance of their activity-level evaluations. In 2003 Cordaid has analysed 104 evaluations. Comments were made on 17% of the evaluations: deviations from the ToR and little attention for (specifically) outcome related issues.38 ICCO has observed that it had no coherent policy with regard to evaluation, that it does not attain the envisaged evaluation frequency, and that better use could be made of the evaluations that were done. It appeared that the evaluations results were generally in line with the evaluation questions, but little could be said about the methodological quality and relevance of the evaluations. In 2004, Plan NL has made an internal assessment of the quality of the activity-level evaluations executed in 2003. It was noticed that the amount of information on results achieved at target group level was too limited, usually based on a few observations and not on a sound methodological design. It was noticed that the quality of the ToR was not always in compliance with the guidelines. Plan NL was not always sufficiently involved in the preparation of the ToR. Role and perceptions of activity level evaluations The principal aim of activity-level evaluations is to support management of the relationship with the partner organisations. Activity-level evaluations are therefore focused on operational aspects: the (internal) organisation of the partner organisation, the implementation process, progress, direct outputs (sometimes including reach of services) and financial/administrative organisation.

38

Jaarverslag Cordaid 2003

61

The evaluations are clearly perceived to play a role in the communication with the partner organisation about continuation of the relationship. Most partner organisations express their appreciation for the fact that an external expert assess the strong and weaker points in the interventions and the organisation itself, but at the same time it is realised that there is an element of risk in any evaluation. With evaluations of usually short duration, evaluators cannot always grasp the more intricate details of the execution of the activities over a period of several years. Staff of the partner organisation sometimes perceive evaluators to have an excessively critical attitude and tone. Overall, however, the attitude towards evaluation is positive, and there appears to be a genuine appreciation for their potential to contribute to the quality of the activities of the organisation, specifically by management. 6.4.2 Programme evaluation All CFOs, with the exception of TdH39, conduct programme evaluations. This includes evaluations done by CFO individually and a series of evaluations agreed upon in the context of the MBN quality management programme. The CFOs together execute approximately 5-7 programme evaluations annually, although more were envisaged for 2004. Table 6.5: Numbers of programme evaluations reported by CFOs Cordaid Hivos ICCO Novib Plan 2002 2 1 3 0 0 2003 2 1 2 1 1 2004* 4 4 3 1 0
NB: * numbers of programme evaluations envisaged. Source: Voorstudie IOB Evaluatie MFP Wiert Flikkema 15-10-2004

TdH n.a. 0 0

The number of programme evaluations envisaged for 2003-2006 in the context of the MBN quality programme was 7, and has been revised to 5. Compared to the number of activity level evaluations, the number of programme evaluations is limited. CFOs perceive programme evaluations to be specifically relevant for policy formulation, strategic decision making and learning. They are also considered to serve the purpose of accountability to the Ministry. Because of their perceived importance, CFOs have indicated that they intend to increase the number and coverage of programme evaluations. Both the programme evaluations of each of the CFO individually and the MBN programme evaluations are subject of a separate IOB assessment. 6.5 Aggregation of activity level results

All CFOs are aware of the difficulties concerning result measurement and the aggregation of results from partner level to policy level. Over the last few years CFOs have attempted to develop methods to aggregate results of individual activities. Broadly, two different approaches are followed. The first attempts to define standard policy domains and assess the contribution of each individual activity to these policy domains. An internal assessment of the strengths and weaknesses of the ICCO PME
39

TdH executes programme evaluations as of 2005.

62

system40 concludes that the main problem faced is that aggregation of data is either not possible or meaningless. ICCO has started a system to measure the performance of the partner organisations in relation to a set of standard criteria (contribution of activities to direct poverty alleviation and civil society development, gender policy, autonomy and sustainability, economic sustainability, basic services and effective PME system) representing 6 policy priorities to determine the contribution of the partner organisations to these priorities. Novib has formulated the 5 rights (see appendix 6) as its main policy priorities, and is developing indicators for these rights to measure the contribution of individual activities. As of 2003, Plan NL uses the results of activity level evaluations for result measurement. In its Annual Report 2003 Plan NL has made a first effort to make a results report, in an attempt to establish a system of result measurement at the level of the target group and communities. The second approach is based on systems to give a standardised assessment of the achievements of individual projects using of a rating system. Hivos has introduced a new computerised administration system that allows incorporation of result information. To assess the achievement of results by partner organisations a rating system is developed to assess the annual progress reports from 2004 onwards. The rating system assesses the outputs realised and the achievement of objectives of the activity by partner organisation. At termination of the agreement a final rating is given on the basis of all available information (annual reports, evaluations and field visit reports). It is expected that the rating system will allow some degree of aggregation of achievements at the level of Hivos policy priorities. From January 2004 Novib has introduced a Performance Registration System. This system documents the direct results of the partner organisations, and also the outcomes and medium to long term effects. The system includes a rating system. Using the system, Novib can indicate the number of activities that has been successful. Linked to the Performance Registration System, are the Toolbox and the revised guidelines for progress reports by partner organisations as well as activity-level evaluations. By the use of these instruments, Novib intends to react in a more structured manner on the specific goals and milestones of an activity. With the performance registration system Novib expects to be able to report more on results. 6.6 Use of feedback information

Information provided by monitoring and activity level evaluations is used for the purposes of accountability, management decision making and learning. 6.6.1 Accountability

CFOs use the information obtained from partner organisations for internal and external accountability. Internal accountability is usually directly linked to operational decision making for management of the agreement with the partner organisation. Internal accountability CFOs use progress reports to assess whether the activity is being implemented as planned, and expenditures are done according to plan. The emphasis is on quantitative information. The detailed procedures of the CFOs to process progress reports,
40

The pressure to measure: Planning, monitoring and evaluation at ICCO 05-09-2003

63

although different in form, are in essence rather similar. Progress reports are appraised by financial and programme specialists. Programme staff appraises the progress reports in relation to the contractual agreement, previous reports and other decisions that are agreed upon. The financial staff assesses the financial information according to the financial guidelines. This information is then compared. The quality of the information is assessed by verifying the consistency of information with other information received form the partner organisation (visits to partner organisations, other exchanges of information). If necessary, the CFO staff requests additional information. A positive assessment of the progress reports is required before the next instalment can be transferred. The CFOs should react to the partner organisation within three months after reception of the progress reports. Information received from the partner organisations is used for reporting by the unit to higher level management in the CFOs. This includes financial disbursements and information on indicators. The internal reports do not only include information from partner organisations, but also achievements of the unit itself (achievements in terms of staff, development of expertise). External accountability The annual CFO reports drafted for the Ministry of Foreign Affairs give information on the implementation of the annual plan. Mainly information on CFO level is given: implemented activities, disbursement per sector or country, changes in the partner portfolio, information about partner consultations, and also information on policy developments and information on organisational developments etc. TdH and Novib state the number of beneficiaries they have reached with their interventions. Aggregation of information from partner organisations about direct outputs, effects and impact on the target groups does not take place. Examples are presented about the implementation of activities by partner organisations (information on output, and sometimes outcome and impact results for individual activities). These examples, however, are purposely selected and are not representative for the portfolio. Cordaid and Plan also include a summary of the activity-level evaluations in the annual report. All CFOs provide the Ministry with key characteristics of all activities they support (kenschetsen). Plan and Novib (since 2003) include an indication of the results of these activities in their key characteristics. 6.6.2 Management decision making

Operational decision making is closely integrated with the internal accountability described above. A positive assessment of the narrative and financial progress reports is usually required for the disbursement of subsequent instalments. (Programme staff of the CFOs, however, observed that very few progress reports are qualified as unacceptable). At the level of individual agreements Activity-level evaluations are important instruments for management of activities under implementation and support decision making about possible continuation of the activity or the relationship with the partner organisation. They also contribute to learning, specifically at the operational level (programme officer and partner

64

organisation). The activity-level evaluation report is appraised by the CFO, and this appraisal is forwarded to the partner organisation. The lessons learned and the conclusions of activity-level evaluations are considered in the context of the formulation of new activities and the negotiations for continuation of activities. In the contracts of some of the CFOs explicit reference is made to the findings of evaluations and the follow-up that ought to be given. Evaluations are also used to identify specific issues to monitor during the next funding cycle. Portfolio management The regional or country teams/units monitor their portfolios of activities at a higher level. Attention is paid to the political and economic context in a country and some of the activity level information is aggregated (disbursement per sector and/ country and the flow in the partner portfolio - new and old partners). Instruments to support portfolio management are largely informal. Hivos has recently introduced a progress assessment rating system that is considered particularly useful for portfolio management. Novib is developing a Strategic Programme Management system on country / regional programme level in order to be used for strategic portfolio management. Portfolio management is becoming more important as the CFOs are attempting to be more strategic in their choice of new partners. Policy management Use of progress reports for policy development The use of information in progress reports for policy development is limited. Hivos sectoral staff use the narrative progress reports to monitor the implementation of the sectoral policies. For policy adjustments, senior (sector) policy staff gather relevant information from the different regional units. This is done ad-hoc, there is no structured information routine in place to aggregate activity level information to the level of policy domains. Novib has initiated pilot sessions for thematic learning on the basis of the information obtained from its performance registration system. Use of partner consultations for policy development Most of the partner organisations attending the partner consultations are considered strategic partners, usually with a longstanding relationship with the CFO concerned. The consultations provide a framework for exchange of ideas and experiences focused on specific themes, with a depth that goes beyond the regular reporting system and the ad-hoc contacts that are usually concerned with day-to-day management issues. The consultations are used by the CFOs to obtain information and perceptions of key stakeholders on specific themes. CFOs consider this information very useful for policy development. Use of activity-level evaluations for policy development Lessons learned from activity-level evaluations that exceed the level of the individual partner organisation, are shared within the entire organisation and can be used as input for policy development. In some CFOs, this is done structurally, in others more in an ad-hoc way. In Novib a summary of the activity-level evaluation is forwarded to the Quality and Control unit. The Research and Development unit annually analyses all evaluations, to define trends and lessons. These may be used for policy adjustments. Also in Cordaid, ICCO and Plan, the lessons learned that are relevant for the entire

65

organisation are shared. In Cordaid, the lessons learned are forwarded to the Bureau Policy and Projects and to the corresponding thematic groups. Hivos and TdH do not have a mechanism in place to share the lessons learned from activity-level evaluations with the entire organisation. Organisational use is ad-hoc, depending on the relevance of the findings. 6.6.3 Learning

All CFOs are keen to be seen as learning organisations. Some CFOs are developing routines explicitly aimed at enhancing learning from activity level information. Novib uses information from execution (progress reports and activity-level evaluations) to define good (and bad) practice. It is still working on routines to integrate the learning process in its internal policy cycle. In 2004 a pilot to strengthen learning was started41. ICCO also promotes learning in the organisation. The funding cycle is the core of the learning process, based on information collected from the progress reported by partner organisations, evaluations and visits to partner organisations. Broader learning of thematic lessons takes place by means of thematic partner consultations and training. Learning within Cordaid is promoted by organising 5-10 presentations and discussions annually per department (lekgotlas or palavers), introduced early 2003. During these presentations the emphasis is on drawing the lessons from practical experience on the basis of cases. These cases may be individual partner organisations or themes or strategic issues. A first review of the practice of lekgotlas is positive, but reveals that a best practice still has to emerge. Hivos organises regularly thematic meetings for specific themes and broader reflection sessions. In December 2003 the organisation adopted a Knowledge sharing strategy, and systematically organises and analyses feedback within the entire chain including partner organisations and establishes contacts between people and knowledge, in order to enhance institutional learning. The focus in Plan used to be very much on the accountability function of feedback information. This is now changing, and more emphasis is put on learning from experiences. The findings from activity-level and programme evaluations are increasingly being used to support policy formulation for Plan NL, as well as the Plan country offices. TdH considers the joint evaluations that it executes regularly as important learning instruments for itself and for the partner organisation. In addition to these evaluations, TdH stimulates and organises meetings or seminars for exchange of experiences among partner organisations. TdH has not (yet) executed a programme evaluation. 6.7 Recent developments

This study is conducted in the period that the CFOs are developing and implementing changes in their monitoring and evaluation systems and quality systems. In this paragraph a brief overview is presented of the most recent changes in monitoring and evaluation. In the context of the MBN, the CFOs have established a quality system that consists of three elements: peer reviews, minimum quality requirements (INK assessment) and the MBN programme evaluations. For the programme evaluations the MBN quality working group has developed criteria for the terms of reference, the evaluation report
41

Performance Management in Strategic Development Funding, June 2004.

66

and external consultants. A reference group is also established to assess the quality of the evaluations. CFOs are strengthening their own monitoring and evaluation policies and systems. Cordaid is currently systemising and improving the process of recording and sharing lessons learned from activity-level evaluations. It is also formulating best practices for its learning sessions on the basis of actual experiences (lekgotlas). ICCO has revised its evaluations policy. The aim of the new policy is to assure efficient use of resources available for evaluation including the use of the evaluation results. The development of the evaluation policy includes formulation of guidelines, and explanation of the types and role of evaluation in the ICCO PME cycle. The elaboration of a standardised format for terms of reference for evaluations is part of the operationalisation of the evaluation policy as well as training of staff in its application. Since December 2004 Hivos has a policy framework in place for improved result orientation and result assessment. This framework is a follow-up of the partner consultations on result measurement. The framework is aimed at improved result orientation in the partner relationship. Agreement on result indicators is an important element in the new contracts between Hivos and their partners42. An updated evaluation policy will be published in 2005. Hivos aims to introduce an evaluation approach that enhances the possibility to aggregate results achieved by its partners. The introduction of a diversified menu of types of evaluations is envisaged (self assessments, activity-level evaluations and programme evaluation). Hivos intends to put more emphasis on evaluations at programme/thematic level that will be more relevant for Hivos itself. The responsibility for activity-level evaluations should be increasingly assumed by the partner organisations themselves. By the end of 2004 the Novibs new evaluation policy was approved. It distinguishes different types and modalities of evaluation. The responsibility for activity-level evaluations is shifted to the partner organisations as of 2005. The aim of the counterpart-led evaluations is that they will be more result-focussed, contain more information from the field and will be less focussed on organisational and process aspects. The scope of the evaluation, however, is clear: it should adhere to the expected and unexpected outcomes and risks as identified in the Toolbox as well as to the contribution of these outcomes to policy and practice changes. The new evaluation policy includes guidelines for terms of reference for counterparts use. The quality of evaluation reports is formally assessed and evaluation results are taken into account in the performance registration system, that includes a rating system. All programme evaluations are making full use of the reports of activity-level evaluations. Plan NL strives to enhance the quality of the ToR to stimulate evaluators to search more systematically for outcome-information. In the internal study of Plan NL it is recommended that it should always reserve approval of the ToR, that Plan NL needs to be more involved in the decision making process of the evaluators and that the final responsibility of the evaluations remains with Plan NL. Another instrument to enhance the quality of evaluations, used by both Plan NL and Hivos is the consultant data base. In 2003 TdH introduced a standard planning system for all partners in the 4 regions where TdH operates. In 2004 this was followed up by the introduction of an M&E system for individual activities. At the cut-off point of the study, TdH did not execute programme evaluations. However, TdH will start executing programme
42

See also paragraph 6.3

67

evaluations as from 2005. The programme evaluation will be carried out from an accountability perspective. 6.8 Analysis

The monitoring and evaluation systems of CFOs include the following instruments. Table 6.6: Overview of monitoring and evaluation instruments of CFOs
Instruments
Monitoring Annual narrative and financial progress reports (sometimes biannual) from partner organisations to CFOs, usually on the basis of formats defined by CFOs Visits to partner organisations, including reporting and debriefing Workshops and consultations with strategic partners. Regular contacts by e-mail and telephone. Progress meetings at different management levels Activity-level evaluations: evaluation of an activity or a partner organisation by an external evaluator, initiated as part of the agreement between CFO and its partner organisation. Programme evaluations: Evaluation of several activities, or policies/themes, initiated and executed under the responsibility of a CFO. The (5) evaluation studies co-ordinated under the MBN quality system (under execution). Internal evaluations and learning sessions. Reflection workshops / and learning sessions.

Evaluation

The monitoring and evaluation instruments studied provide mainly information about the use of inputs, the progress of activities and the realisation of direct output. Information about effects and impact on partner organisations and the target group is variable. Monitoring and evaluation instruments appear to be suitable to collect and communicate such information to CFOs when it is available with the partner organisations in a structured and written format. Through field visits little information about the effects of interventions on partner organisations and target groups is collected. CFOs consider programme evaluations, that are not included in this study, to be the main instrument to provide information about the results of their activities at the levels of the partner organisations and the target groups.

68

Co-financing organisations follow several strategies to control the quality of the information they obtain from partner organisations through the regular reports. On the one hand, prescribing formats for reporting with a number of specific questions focuses the reports on the information desired by the CFO concerned. Comparison of progress reports from partner organisations to CFOs indicates that the more specific the questions posed in the reporting guidelines, the more specific and informative the information provided. Unequivocal definition of concepts used in these guidelines also enhances the quality of the information obtained. A second strategy used by the CFOs is that they compare information obtained from the different sources and feedback instruments: reports, information and impressions gained during field visits and information obtained from other contacts with partner organisations. Finally, the provision of information from the partner organisations to the CFOs is embedded in an often long term relationship, where trust is an essential element. In such a context, partner organisations have a strong incentive to provide correct information to the CFOs. Together these check-and-balances in the relationship control the validity and reliability of the information communicated from partner organisations to CFOs. Currently CFOs are strengthening tools to enhance the quality of activity level evaluation. For CFOs the main use of the information received from the partner organisations is contract management and portfolio management. The information is also used for the purpose of accountability to the Ministry. The annual reports mostly provide aggregated information on inputs and some process-aspects. Reports include examples of direct outputs and results achieved. These examples reflect exclusively positive experiences. The value of such selected examples for the purpose of accountability is limited. Learning on the basis of monitoring and evaluation is mainly focused on the operational level, with the exception of the partner consultations, which are more clearly related to policy development. From the study of monitoring and evaluation systems in CFOs, the following conclusions can be drawn.

69

The different monitoring and evaluation instruments provide the CFOs with the information they need for the day-to-day supervision and management of the relationship with their partner organisations. The collection of information by the CFOs from their partner organisations follows a well defined rhythm (annual progress reports, annual visits and evaluations according to a defined rule) and is standardised (guidelines and formats for annual reports, field visit reports and terms of reference for activity level evaluations). The prescribed formats follow existing practices, and tend to emphasise the communication of information about the use of inputs, the execution of activities and the realisation of direct outputs. This information directly relates to the project documents and the agreements and is therefore relevant for the CFOs. The link between operational planning and monitoring and evaluation is direct and usually clear. Until recently contract management mostly concerned the management of individual agreements. Increasingly CFOs are developing instruments to manage portfolios, for example rating systems that allow analysis and comparison of activities at portfolio level. The connectedness between the information provided by monitoring and activity-level evaluation is less pronounced. Two factors are important: o The information provided about effects and impact is irregular and not standardised. Monitoring instruments used and the activity level evaluations provide little information about the effects and impact of the activities on the target groups. In the annual progress reports the organisations that systematically collect information about effects and impact of their activities are more informative than partner organisations where the collection of this information is ad-hoc and unstructured. CFOs have not designed monitoring and evaluation instruments that make use of the substantial body of unstructured information about effects and impact available at the level of the partner organisation. o Aggregation of information from partner organisations about effects and impact is difficult. The activities of the co-financing organisations are dispersed, both geographically as well as thematically. Between the level of policies and the operational level there is no structured process of dis-aggregation, from policies into strategies and programmes, and finally into activities. There is little synergy or coherence between different activities. This makes aggregation of results achieved by activities to higher levels programmes or policies very difficult.

CFOs consider programme evaluations to be specifically relevant for learning and for feeding information into policy monitoring and policy formulation. The number of programme evaluations executed so-far is limited when compared to the size of the co-financing programme. The programme evaluations are subject of a separate assessment by IOB.

70

7.
7.1

MONITORING AND EVALUATION BY DSI/MY

DSI/MY: role and responsibilities

Within the Social and Institutional Development Department (DSI), DSI/MY is the unit specifically responsible for co-operation with civil society organisations, including the Co-Financing Programme43. The responsibilities of DSI/MY are described as follows44: Policy development and formulation, including: o monitoring trends, gathering experiences, making available knowledge with regard to the co-operation between government and nongovernmental organisations (NGOs). o Formulating the policy of the government with regard to the co-operation with NGOs Policy implementation: financial programming and supervision of implementation of programme agreements. Quality management: giving advice on co-operation and policies regarding NGOs and contribute to the development of modalities for programme funding. Co-ordination of the contribution of NGOs in bilateral and multilateral organisations, support the role and opportunities for co-operation of NGOs in international platforms. The CFP is, in financial terms, the largest programme supervised by DSI/MY. Organisation of monitoring and evaluation Monitoring and evaluation is the core task of DSI/MY. DSI/MY is composed of a head of the unit, 3 senior policy staff and 3 policy staff. In addition it employs a trainee (numbers as per 2 December 2004). The operational responsibility for supervision of the CFOs is distributed among 5 staff who, like the remaining staff, are involved in other tasks of the unit as well. Because the political nature of the programme, the Director and Deputy Director of DSI are also frequently involved in discussions with the CFOs. 7.2 Monitoring systems and instruments

The annual plans and the annual progress reports submitted by the CFOs constitute the core of the regular monitoring system of DSI/MY. The planning and reporting cycle that DSI/MY uses to supervise the co-financing programme is described in the CFP policy document. The subsidy agreements between the Ministry and the individual CFOs stipulate that CFOs must submit annual plans and annual narrative and (audited) financial reports. The purpose of the annual progress report is to assess if the CFOs achieve the intended results as formulated in both their strategic management plans 2003-2006 and annual plans. The management plans 2003-2006 and the annual plans provide the framework for the monitoring of progress by DSI/MY.
43

Other programmes include funding thematic NGOs, the NCDO for awareness raising, municipalities and trade unions. 44 See Formatieplan 1996 DSI/MY

71

In addition, DSI/MY monitors the co-financing programme through a variety of mechanisms: Specific requests for information necessary for fulfilling external information requirements for the allocation of ODA funds or political discussions or questions in Parliament. Regular meetings between the Minister, staff of DSI/MY and management of the CFOs. Occasional field trips by staff of DSI/MY. Consultations between CFOs and Embassies. Other exchange of information. 7.2.1 Annual plans and annual reports

The annual narrative and financial plans should be submitted to the Ministry before the first of November of each year. In the 2003-2006 policy a format is presented for the annual plans (see appendix 7). In order to make the annual plans and reports more result focused, an amendment on the reporting guidelines was proposed by DSI/MY in August 2004, after consultation with the CFOs. Key elements of this amendment are45: Introduction of a matrix structure in the reporting. Specify results per level: results achieved at the level of the CFO itself, at the level of the partner organisation or the relationship between the CFO and the partner organisation and results at the level of the intended target groups. Finding the right balance between qualitative and quantitative information. Reporting achievements related to the Millennium Development Goals. The annual plans are assessed by DSI/MY, DSI/CU and by FEZ46. The assessment of the plans by DSI is based on the result-focus of the plans47. The observations and recommendations of the evaluations by the Steering Committee (2002) are used as criteria for assessment of the annual plans.48 The CFOs developed plans of action to incorporate the recommendations of the Steering Committee. These include developments with regard to partner policy, concentration and specialisation, learning capacity and civil society building. DSI monitors the internalisation of these aspects in the annual plans. The 2003 and 2004 plans were commented on by DSI/MY (see appendix 7 for a summary of the comments relevant for monitoring and evaluation). After the appropriate changes in the plans have been made by the CFOs, the annual plans were approved. The annual reports should be submitted before the end of the month of April of each year. In the CFP policy document it is stated that the annual reports should follow the outline of the annual plans.

45
46

Letter DSI/MY 421/04 of 3 August 2004. DSI/CU is the unit financial control of DSI. The Financial and Economic Affairs Department (FEZ) assesses only the financial plans 47 The extent to which plans include the sequence Objectives, Results, Activities, Resources (DRAM) and the proper formulation of objectives and results (Specific, Measurable, Agreed upon, Realistic and Time bound SMART). Definitions from the appendix of the CFP policy framework. 48 DSI/MY has prepared an overview (2003) of the observations and recommendations of the evaluation reports of the Steering Committee. This overview is used for both the assessment of the annual plans and the reports.

72

Analysis of annual narrative reports The 2000 and 2001 annual reports of the (then) 5 CFOs were analysed with a focus on poverty alleviation issues49. For the annual reports of 2000 the conclusion of the analysis was that the annual reports are very useful to inform the general public, but less so as an instrument for accountability towards the Ministry. The reports mainly focus on activities: information primarily concerned what was done. The information about the outputs realised and results that were achieved was considered inadequate. There was little background information on the policy and operational choices that were made. The analysis of the 2001 annual reports shows that the reports maintained their focus on activities. The visions and policies on poverty alleviation are somewhat better explained. Information about the choices made, and specifically the results achieved, however, was still considered to be inadequate. This analysis was conducted by DSI/AI, with the report becoming available in August 2002, well after DSI/MY gave its reactions and observations on the annual reports to the CFOs. The observations form the side of DSI/MY are of a different nature: more process oriented, with emphasis on the role of partner organisations in PRSP processes. For the analysis of the annual reports 2003 DSI paid considerable attention to the result-focus of the reports50. DSI has noticed a positive development in the quality of the annual reports 51 compared to previous years: it is observed that the annual reports 2003 (the first submitted on the basis of the present policy framework) adequately present the relevant activities and developments of the CFOs and the results that are met. DSI expects the annual reports of 2004 and the annual plans of 2005 to be even more result focused (formulated in terms of objectives-results-activities-resources with appropriate indicators52), partly because of the new format for the annual plans and reports. The annual narrative reports 2002 and 2003 were assessed on the basis of a number of criteria53. All CFOs have submitted their annual reports 2002 and 2003 in time. The following table presents a summary of the assessment.

49
50

Memorandum from DSI/AB to DSI/MY of 6 August 2002. See also letter to Parliament DSI/MY 459/04, Jaarverslagen 2003, 2-11-2004 51 See also letter to Parliament DSI/MY 459/04 52 Internally labelled DRAM and SMART. 53 See appendix 7. The criteria used reflect the focus of this study.

73

Table 7.1: Assessment of annual progress reports of CFOs

Cordaid 02 03 Format (for 2003 only) In accordance with policy framework CFP In accordance with annual plan 2003 Contents Development/evolution of strategies Internal organisation of CFO Context in North and South Activities supported Execution of activities Lessons learned from experience Explanation of deviations from plan Results At level CFO Disbursements per theme/ portfolio CFO Policies and policy development At PO level Output of activities Aggregated output of activities Results (strengthening of POs) Aggregated results At level target group Results at level target group Aggregated results at level target group n.a.. n.a. 2 3 4 4 3 2 n.a. 2 3 3 4 4 4 3 2 2 Hivos 02 03 n.a. n.a. 2 3 2 3 3 2 n.a. 3 3 3 3 2 4 4 3 2 ICCO 02 03 n.a. n.a. 3 4 4 4 4 3 n.a. 3 4 4 4 4 4 4 3 3 Novib 02 03 n.a. n.a. 4 3 3 4 4 3 n.a. 3 4 4 3 3 4 4 4 3 Plan NL 02 03 n.a. n.a. 4 3 2 4 4 3 n.a. 3 4 4 4 2 4 4 4 2 TdH 03 3 2 2 3 2 3 2 2 2

4 3 2 1 2 1 2 1

4 4 2 1 2 1 2 1

4 3 2 1 1 1 2 1

4 4 3 2 3 1 2 1

4 4 3 1 3 1 1 1

4 4 2 1 3 3 1 1

4 3 2 1 1 1 2 1

4 3 3 1 1 1 2 2

4 4 3 1 2 1 2 1

4 4 3 2 3 1 2 1

3 2 2 1 3 1 1 2

See for assessment format appendix 7. Scores 1: insufficient information, 2: summary account, 3: good information, 4: insightful information.

The table suggests that: The presentation of information about the execution of activities, strategies followed by the CFOs and organisational changes is adequate in most annual reports. Information about lessons learned from the experiences is variable. Information about the results at the level of the CFOs (policies and policy development, disbursement per theme/ region, portfolio management) is adequate. None of the CFOs (with the exception of ICCO) manages to present a coherent overview of the aggregated results achieved at the level of the partner organisations. Results at the level of the target group are mainly presented by way of illustration. Without exception, these illustrations present positive experiences, and they are therefore not representative. For the criteria used, the analysis shows a moderate improvement of the 2003 annual progress reports compared to the 2002 annual progress reports. Although this assessment is in line with the assessment of the annual progress reports by DSI/AI, it indicates that there is still considerable room for improvement with regard to reporting on results. 7.2.2 Specific requests for information

Specific requests for information to the CFOs usually originate from (international) administrative and policy requirements and from political discussions or reports in the media. Administrative and policy requirements for information are based on international and national commitments with regard to ODA expenditures. These

74

include ODA expenditures according to DAC classification codes and the allocation of ODA expenditures in relation to the MDGs. Also the participation of civil society in PRSP in relevant countries, and the role of CFOs in stimulating their partner organisations lead to regular requests for information. Between 1 January 2002 and September 2004, on 8 occasions questions (in writing) have been asked in Parliament related to the CFP. These questions sometimes give rise to requests for additional information from DSI/MY to the CFOs. Table 7.2: Parliamentary questions about the CFP Date Issues raised
28 May 2002 25 October 2002 and 8 November 2002 6 November 2002 On the basis of reports in a national newspaper about termination of the subsidy for Foster Parents Plan, the question was asked whether the Minister intended to terminate this subsidy. Preliminary conclusions of the evaluation report of the Steering Committee also in relation to the conclusions of the Inter Departmental Study (IBO) of 2000. Political activities of the CFOs The contribution of Novib, Cordaid, ICCO to the United Civilians for Peace (that organised the visit to Palestine of Mrs G. Duisenberg). Support of HIVOS to the report of M.J. Faber about Srebrenica Termination of bilateral development co-operation with India; consequences for the CFP. The measurement of the effects of influencing policy in the Netherlands. Support from Novib and Hivos for 4 organisations in Cuba Support from Novib and Hivos for organisations linked to Pres Castro ICCO campaign about made in Israel; labelling of products produced in occupied territories

6 June 2003 and 15 July 2003 8 October 2003 27 August 2003 18 March 2004 and 14 April 2004 18 June 2004

Issues regarding the CFP were also subject to discussions in the general discussions between the Minister and Parliament54. In 2002 the discussions concerned the recommendations of the Inter-Departmental Policy Review (IBO) report 2000, the new policy for the Co-Financing Programme and the reaction of the Minister on the report of the Commission de Boer on Foster Parents Plan Nederland. During the AO on the Co-Financing Programme in 2003, the discussions mainly concerned the recommendations of the Steering Committee Evaluation Co-Financing Programme, the recommendations given by the appraisal commission CFP, and the possibility to merge the Co-financing Programme and the Thematic Co-Financing Programme into one programme. In 2004 no specific AO on the Co-Financing Programme was held. 7.2.3 Regular meetings

In the context of the regular contacts between the Ministry and the CFOs different types of meetings are held: Annual policy consultations: discussions between DSI/MY and the individual CFOs on the basis of the annual reports or annual plans. Regular meetings with the management of the CFOs collectively. Between January 2001 and September 2004 few annual policy consultations have taken place, and for some of these no reporting is available. Most of the observations
54

General Consultations take place between the Minister and Parliament (Algemeen Overleg AO)

75

on the annual plans and reports and the formal clearance are communicated by exchange of letters between DSI/MY and the CFO concerned. The reports of the annual policy consultations that are available deal with financial and programme management matters, the latter in very general terms. In one case the issue of strengthening the result focus of management is explicitly raised. Meetings between DSI/MY and the management of the CFOs collectively, in the GOM/MBN platform55 are held, in principle on a regular basis but their number decreased over the years (8 meetings in 2002, 3 meetings in 2003 and 3 meetings in 2004). Issues discussed mainly concern the relationship between the Ministry and the CFOs, and administrative and management issues. Occasionally more substantive issues are put on the agenda. In the 2004 meetings the formulation of the 2007-2010 policy was an important subject of the discussions. 7.2.4 Field trips by staff of DSI/MY

Staff of DSI/MY executes field trips to monitor activities of CFOs. Since 2001 six fieldtrips were reported. Table 7.3: Fieldtrips executed by DSI/MY staff between 2001 and September 2004 Name officer countries Period CFO concerned
Schulting N. Blokker A. Holleman B.Paardekooper/v.d.Burg N. Blokker A. Holleman Bangladesh Bolivia/Peru Indonesia Uganda Centr. America Salvador/Honduras February 2001 April 2001 June 2001 November 2001 November 2002 November 2003 TdH ICCO PLAN ICCO HIVOS PLAN

Field trip reports are usually copied to the CFOs concerned, and in some cases key findings are discussed with them. The nature of the reports suggests that the general objective of the field trips is to familiarise DSI staff with the (operational) reality in the field rather than to monitor the implementation of CFO policies and activities in a structured way. In principle every staff member of DSI/MY involved in the CFP should make at least one field visit each year. In 2002, 2003 and 2004 the frequency of field visits has been lower. DSI/MY recently expressed its intention to increase the number of field visits to the level envisaged. 7.2.5 Meetings between CFOs and Embassies

In addition to the direct relationship with DSI/MY, CFOs maintain relationships with the Embassies in the countries where they operate. The nature of these relationships varies, and is often dependent on the interaction between the CFO staff responsible for the country concerned and the staff of the Embassy. In 1999 a system of country specific operational agreements was introduced to strengthen co-ordination between bilateral programme activities and CFP-funded activities. With some exceptions, however, the country specific operational agreements are limited to an exchange of information between the parties involved. In the budget
55

TdH is not a member of the MBN. In 2004 the GOM changed names into MBN.

76

discussions in Parliament in December 2002 the Minister agreed to strengthen the system of country specific operational agreements, and a pilot was started during the year 200456. The agreements mainly focus on co-ordination and exchange of information, and do not play an important role for monitoring or evaluation. 7.2.6 Other exchange of information

Besides the above mentioned contacts, staff of DSI/MY maintains regular contact with the CFOs by e-mail and telephone. Also, the CFOs regularly inform DSI/MY about new developments in their organisations. New policy papers are presented, which are usually commented on by DSI/MY and CFOs inform DSI/MY about the development of instruments and the achievement of, for example, the ISO certification. The CFOs are invited to comment on new policies of the Ministry, like the policy paper Mutual Interest, Mutual Responsibilities57. 7.3 Evaluation

Activity-level evaluations DSI/MY does not directly obtain information from the many activity-level evaluations that are conducted by the CFOs as part of their contract management processes. Programme evaluations Programme evaluation by the CFOs individually, or co-ordinated under the MBN is one of the instruments mentioned in the CFP-policy framework. The programme evaluations executed during the policy period 2003-2006 are subject of a separate assessment by IOB. During this period, IOB also conducts 5 evaluations of activities supported under the CFP. Since 1999 two programme wide studies were conducted, the Inter Departmental Policy Review (IBO) and the evaluations by the Steering Committee. These studies were instrumental in the definition of the CFP policy framework. Efficiency audit In compliance with the CFP policy, DSI/MY initiated an external audit of the CFOs to assess the efficiency of the CFP. The audit is executed late 2004. The final conclusions of the audit are positive with regard to operational management of the CFOs: all CFOs have become more professional over the past few years and all are incorporating quality systems, based on the INK-model. Follow-up to the audit is envisaged for 2005.

56 57

Memorandum DSI/MY 509/03 of 2 September 2003 Aan Elkaar Verplicht (2003)

77

7.4

Use of feedback information by DSI/MY

Accountability and contract management Based on the annual plans DSI/MY58 assesses the progress of the planned activities and achievement of results. It assesses the reports on the following aspects: reporting in compliance with contract, results in line with objectives and time plan, bottlenecks and solutions, compliance to agreed conditionalities, need for adjustment of objectives, sufficient risk assessment and reporting consistent with other information communicated. Annual reports of CFOs are circulated (with a covering letter from DSI/MY indicating specific topics of interest) to relevant thematic directorates and embassies in the countries where the organisations are operating. Thematic directorates and embassies are asked to comment on these reports and regularly do so. Overall the observations made by the embassies and the directorates reflect a positive perception on the work of the CFOs. In a number of cases the lack of information about the activities of the CFOs is mentioned. Often the annual narrative reports lead to observations or the request for specification or additional information. When the additional information requests are met, a formal clearance is given to the CFOs. Table 7.2 in appendix 7 presents an overview of observations by directorates of the Ministry and embassies and information requests by DSI/MY. These reactions by DSI/MY are usually sent several months after the annual progress reports are received. Every year DSI appraises the performance of each CFO. The appraisal is summarised in a rating form in the Ministrys computerised management information system. Key questions for appraisal are: Have the inputs been delivered as planned Have the envisaged outputs been achieved Will the objectives be achieved Sustainability of the activity and its results The last rating found in the files was dated 13 March 2003. Since that date, the rating system has not been used systematically. The ratings found for 2002 were positive for all the CFOs. It was noted, however, that the ratings were given, prior to reception of the annual reports from the CFOs, and were therefore not based on recent information. DSI/MY regularly reports on the CFP in its own annual reports. Since the annual report 2001 these reports follow the new budget structure based on 18 policy priorities. The CFP is included in policy priority 10, co-operation with civil society organisations. In the annual report 2001 of DSI, the results of the CFP are summarised in 5 bullets. These bullets refer to the revision of the subsidy system, the successful conduct of policy consultations with CFOs and the formulation of the CFP policy document as well as discussions with the CFOs about the involvement of civil society in PRSP. It also mentions that the annual reports of the CFOs are specifically analysed from a poverty alleviation and gender perspective, and that discussions have taken place with the CFOs. In the annual report 2002 three bullets summarise the results related to the CFP, referring to the subsidy system, policy reactions to the evaluation report of the Steering Committee and other reports and the process of
58

Reports are also analysed by DSI/CU (unit financial control of DSI) , focussing on the managerial efficiency of the CFOs.

78

coming to agreements between CFOs and the Ministry on more country specific programmes. The annual report 2003 focuses on clarifying the role of the Ministry in relation to the CFOs. With regard to results and thematic policies explicit reference is made to the health programmes. The feedback information, among which the annual reports, the efficiency audits and the IOB CFP programme evaluations, is expected to be used as input for the quality assessment of the work of the CFOs that DSI/MY will formulate on behalf of the Ministry at the end of the subsidy period. Management decision making At the level of the Ministry the information available on the CFP focuses on activities and the allocation of resources by the CFOs over their different programmes and priority objectives. Information about the results that are being achieved in terms of strengthening the capacity of POs and certainly in terms of actual sustainable poverty alleviation is very limited, often absent. The supervision of the CFP by the Ministry is therefore mainly geared towards assuring that CFOs act in agreement with the approved management plans, and with the general rules and guidelines defined in the CFP policy framework, which are very procedural in nature. Learning Monitoring leads to little substantive learning by DSI/MY. Policy formulation with regard to the CFP is largely focused on the procedural aspects of the relationship between the Ministry and the CFOs. The objectives and strategies of the policy framework are formulated in very general terms, and are largely derived from the overall policy objectives and at the moment prevailing priorities of the overall Dutch aid effort rather than from the specific context and contribution of the CFOs and their partner organisations. The monitoring information provided by the CFOs in their annual reports is not relevant for (re)consideration of such high-level policy issues. DSI/MY reported, however, that previous programme evaluations by the Steering Committee have resulted in adjustments in CFP strategies and the relationships with the CFOs. 7.5 Development of indicators

A recommendation of the IBO study was that indicators should be developed to track the achievement of results of the CFP. At the initiative of DSI/MY a working group was established in December 2000. From January till June 2001 5 meetings took place with the participation of DSI/MY and the CFOs, as well as the two federations of trade unions. The working group terminated its activities prematurely. Available information suggests that the requirements of DSI/MY did not take into consideration the progress that was made in each of the CFOs. From the side of the CFOs, it appeared that priority was given to the formulation of their new strategic management plans 20032006. 7.6 Recent developments

79

Currently considerable attention is paid by DSI/MY to strengthen monitoring. Annual meetings are again held with the CFOs. After September 2004 (the cut-off point of this evaluation), DSI/MY has had annual policy meetings with all CFOs to discuss their annual reports 2003. Also the number of field visits is increasing. Since September 2004, 3 field visits has been made to partner organisations of Hivos, Cordaid and Novib. Apart from the programme evaluations executed by IOB and under the responsibility of the MBN, DSI has initiated an evaluation of the front offices59 of the CFOs in 2005. 7.7 Analysis

The following table presents an overview of the instruments used by the Ministry to obtain feedback information about the execution of the CFP. Table 7.4: Overview of monitoring and evaluation instruments DSI/MY
Feedback from
Monitoring

Instruments
Annual plans and narrative and financial progress reports . Visits to countries where CFOs operate. Occasional meetings with CFOs. Country specific operational agreements The 5 MBN co-ordinated evaluations (under execution) and 5 IOB evaluations (under execution) in the period 2003-2006. The Ministry could also request the reports of programme evaluations done by the CFOs, but this has not been done in 2003 and 2004. Efficiency audit (2004).

Evaluation

The annual reports of the CFOs comprise a vast amount of factual information about activities and the allocation of resources. The information on the results achieved, however, is disaggregated and anecdotal. Two CFOs include information on the achievement of results in their project descriptions. The amendment to the reporting guidelines issued in August 2004 is intended to enhance the availability of information on results achieved in the annual reports. It is expected that information specifically relevant for policy level will, to some extent, be provided by the MBN programme evaluations and the IOB evaluations. The administrative and financial information provided by the CFOs is externally verified and considered to be reliable and valid thanks also to the existing protocols between the auditing service of the Ministry and those of the CFOs. In order to enhance the reliability and validity of the result information communicated between the CFOs and the Ministry an attempt was made to develop sets of indicators to be included in the annual reporting. This attempt, however, proved to be inconclusive.

59

In accordance with the 2003-2006 policy the CFOs had to set up a front office to process funding applications from small Dutch development cooperation initiatives which before were submitted to the Minister for Development Cooperation. These activities are outside the scope of the regular programme and, in addition to promoting structural poverty reduction and increased ownership in the South, they also have to boost support for these aims in the Netherlands. (definitie genomen uit tekst/vertaling AVT).

80

The Ministry uses the information it receives about the execution of the CFP to regularly inform the Minister and Parliament, and provide specific information when questions are asked in Parliament. The observations from DSI/MY on the annual reports submitted by the CFOs are mainly process oriented with emphasis on specific policy issues related to the then overall Dutch development cooperation policies and priorities. The conclusions of the study at the level of the Ministry are as follows: The collection of information by DSI/MY from the CFOs has a well defined rhythm (annual reports and consultations). Most of the information collected is structured and systematic, and is amenable to communication to other stakeholders. The reports are formatted on the basis of the annual plans. Field visits are few and irregular, specially in view of the size of the CFP. Much of the financial and administrative information is consolidated at the level of the CFOs. Information about outputs, effects and impact at the level of the target groups concerns the level of the individual activities, and often a selected sample of the information by way of illustration in the annual reports. Aggregation of effect and impact information across CFOs to programme or policy/theme level does not take place. DSI/MY does not fully make use of the monitoring and evaluation instruments at its disposal: specifically CFO programme evaluations, field visits and meetings with CFOs. This can, at least partly, be attributed to the limited capacity of DSI/MY. The number of staff to supervise the programme is small when compared to the size and diversity of the CFP, and its relative importance in the overall Dutch aid effort. DSI has not developed a clear concept for the supervision of the CFP and its role vis--vis the CFOs. The information required by DSI/MY remains to a considerable extent unclear, and what information should be provided by monitoring and different types of evaluation. This leads to different interpretations from the individual CFOs as to what information DSI requires and variations in formats, increasing the burden on DSI/MY to process the information.

81

82

Appendix 1 Policy and Operations Evaluation Department (IOB)

83

Appendix 1: the Policy and Operations Evaluation Department

Objectives of IOB IOB's overall objective is to meet the need for independent evaluation in all fields of foreign policy. Specifically, the aim is to fit the results of the evaluations into the knowledge cycle of the Ministry of Foreign Affairs. The reports of the evaluations are used as feedback to improve both policies and their implementation. With the results of previous exercises at their disposal, policymakers can prepare new interventions more purposefully and more effectively. Quality and independence Parliament has always shown great interest in IOB's independent reports. On the basis of the reports sent to the Lower House, the minister gives a policy response. The Permanent Committee on Foreign Affairs then discusses the report and the policy response. The evaluation results are public and are used, for instance, by universities in their teaching. Approach and methodology IOB has a staff of specialists and its own budget. Given the growing complexity of policies and interventions, a multidisciplinary approach is required. This calls for evaluation expertise as well as specialist expertise in a large number of fields. For this reason IOB uses external consultants and specialists, whenever possible from the countries with which the Netherlands co-operates. This fits in with the policy of flexibilization and professionalization which IOB aspires to. For internal quality control, IOB uses reference groups of involved parties and external experts. In order to compare evaluations, relevant studies are 'clustered' by policy theme. This makes it possible to draw main conclusions for policy making on the basis of a series of studies. This generates an important synergy effect, so that greater attention can be paid to the spearheads of policy: poverty reduction within development co-operation, "good governance" (i.e. the functioning of public authorities and civil society organizations in the countries concerned), issues of international order and the promotion of Dutch interests abroad. Major shifts Since IOB was set up in 1977, major shifts have taken place in its approach, areas of attention and responsibilities. In the early years, IOB's activities consisted mainly of separate project evaluations. Around 1985, the focus shifted to the sector and theme levels, and now its work is conducted on the basis of clustered evaluations. In 1996, there was a review of Dutch foreign policy and the Ministry of Foreign Affairs was reorganised. As part of the review, the name of the department was changed from Operations Review Unit (IOV) to IOB, and its sphere of activities was extended to cover all aspects of the Dutch government's foreign policy, in which development co-operation plays an important role. Future-oriented thinking This new package of activities will be further elaborated in the future. IOB also wants to increase the relevance of its evaluations for policy making and to heighten the impact of the results. Outside the field of development co-operation, IOB will undertake a growing number of evaluations in other policy areas, such as European integration. Within development cooperation, the aim is to expand co-operation with the evaluation services of other donors. And eventually the recipient countries themselves should evaluate the aid extended to them. This means that they have to acquire the necessary evaluation expertise and to manage the evaluations themselves: in short, evaluations have to become recipient-led. Donors and recipients have already taken first steps towards combined evaluations.

84

During IOB's history, there have been major shifts in the approach and methodology of the evaluations. What has remained unchanged over the years, however, is the strictly independent stance. Coupled with a meticulous approach and professional evaluations, this constitutes IOB's ultimate reason for existence.

85

86

Appendix 2 Terms of Reference of the study

87

IOB study feedback and results-based management in the Co-financing Programme Terms of reference (27-1-2004) 1. Background On 1 January 2003, the new Policy Framework for the Co-Financing Programme (CFP) was enacted. In the section the quality assurance system and the assessment process, it assigned the following role to the Policy and Operations Evaluation Department (IOB): The Policy and Operations Evaluation Department (IOB) will inspect and assess the quality of the work delivered on the basis of the results of the co-financing organisations integrated evaluation system. IOB will also conduct its own evaluations. In January 2002 guidelines for monitoring systems and evaluations in central government organisations (RPER) were published. These guidelines intended to strengthen the evaluation function and ensure that policy information in the public sector meets minimum quality standards. An interministerial policy review (IBO) of the co-financing programme (2000), 60 an independent steering committee CFP evaluation (2002)61 and the CFP advisory committee (2002)62 had indicated that the results obtained with support from the CFP needed to be made more transparent if the Minister was to assume her political responsibility for an effective and efficient CFP. In its final report, the steering committee also refers to a problem specific to the aid chain: on the basis of agreements with the Minister, six co-financing organisations (CFOs) allocate Dutch development funds to partner organisations in developing countries so that they can carry out development activities, thus setting up a chain of responsibilities and reporting. If the RPER goal of results-based management is to be achieved, the various levels in the aid-chain must interrelate adequately to keep track of the achievement of results. For this, monitoring and evaluation play a crucial role. In view of this, it has been decided to focus the first IOB study 63 of the CFP specifically on monitoring and evaluation. 2. Scope of study The study focuses on the role of monitoring and evaluation in the organisations concerned at three levels (Ministry [DSI/MY], CFOs and partner organisations) and the interrelationships between these levels. Monitoring and evaluation are seen as organisational processes aimed at obtaining and utilising feedback. The study therefore considers the following interrelated aspects of monitoring and evaluation (M&E): systems and instruments, the working processes that use the information provided by M&E, and aspects of the organisational structure in which M&E systems are embedded. Study objectives Considering the complexities of the aid chain, the objective of the study is to determine to what extent the organisations involved (Ministry [DSI/MY], CFOs and partner organisations) have operationalised M&E systems and whether these systems contribute to the strengthening of results-oriented management. The study findings can provide a basis for further
60

Report by the Werkgroep Interdepartementaal Beleidsonderzoek Medefinancieringsprogramma (2000). 61 Final report by the Stuurgroep Evaluatie Medefinancieringsprogramma, November 2002. 62 Breed Uitgemeten, an advisory report on accession and allocation in 2003-2006 by the Commissie MFP-Breed, September 2002. 63 See IOB programme for 2003-2007.

88

strengthening M&E in the system as a whole. They can also clarify the extent to which CFOs and DSI/MY execute interventions to strengthen M&E, to fulfil their primary responsibility for the quality of the CFP. Study questions The key question for the study is to assess the extent to which the organisations at various levels in the CFP system have M&E systems and make effective use of the information obtained, enabling them to adopt a results-oriented approach to execute CFP activities. The specific questions are as follows: 1. How are the data collected? What M&E systems and instruments do the organisations have and how do they guarantee the quality of information provided by these systems? Special attention will be given to the diversity of instruments. With regard to monitoring, attention will focus not only on formal instruments (such as reporting formats) but also on less structured instruments such as field trips, progress meetings, participation in steering committees, etc. With regard to evaluation, the focus is on evaluation policy and supporting instruments (guidelines and standards) within the various organisations. Both different types of external and internal evaluations will be studied. 2. How are data used? How is the information on inputs, activities, outputs and results used within the organisation? Both structured working methods (progress meetings, management team reviews), but also at more ad hoc, internal processes (discussions with management, special evaluations/inspections initiated by senior management, etc.) will be studied. Moreover bearing the in mind the aid chain structure what is the effect of M&E in reports and communication with the next level in the CFP aid chain? Is there an explicit link between planning, monitoring and evaluation? How is M&E embedded in the organisation? How does the information flow through the organisation, who is responsible for collecting and checking the information, who is the intended user and who is the actual user of the information? Are responsibilities formally laid down, or have they grown up over time? What incentives/disincentives exist for using feedback information, both at operational and policy level?

3.

3. Study structure The study will be carried out at the level of the responsible department of the Ministry, the six CFOs and the partner organisations. Inventory of the various M&E systems used by the Ministry (DSI/MY) and the six CFOs and of the extent to which the information generated by M&E is used by the organisations concerned and in information exchanges between them. The necessary information will be obtained by studying documents (e.g. M&E policy document, established procedures) and through semi-structured interviews with selected staff, both from the CFOs and from DSI/MY. This will mainly be done by the inspector, the lead consultant and a research assistant.

89

Inventory of the various M&E systems used by the CFOs partner organisations and the extent to which the information generated by M&E is used within the organisations concerned and in information exchanges between them and the CFOs. For this part of the study, three countries in three different continents have been selected in which all six CFOs are active: India, Tanzania and Peru. These countries have long standing relationships with CFOs involving a large number of partner organisations,64 and substantial disbursements65 by the CFOs. For logistical reasons, two clusters (regions in which partner organisations are concentrated) have been identified in each of the three countries. Using database information, an average of 30 partner organisations are selected per cluster and invited to participate in the workshops66. In each cluster 10 partner organisations will be selected for more detailed study. Selection will take place according to various criteria. The main criteria will be proportionality (based on the financial weight of the six CFOs with which the partner organisations work), the size and nature of the organisations, the sectors in which they operate and the internal organisation of M&E. It is envisaged to study in detail 60 partner organisations in the three countries. The partner organisations are selected using a database containing basic data on all organisations with which the various CFOs work in the three countries. A questionnaire was used to ask individual organisations about their approach to M&E, both within their own organisation and in their dealings with the relevant CFO. The necessary information on the partner organisations will be obtained by studying documents (e.g. M&E policy document, established procedures) and interviews with selected partner organisation staff by local consultants. A separate, detailed TOR has been formulated for this study (including an outline of the draft report to be drafted up by each organisation) for local management consultants in the relevant countries to execute this part of the study. Local consultants will be asked to submit their draft findings for comment to both the partner organisation concerned and the IOB team. The partner organisations will be asked to verify the facts supplied in the case study report and to comment on the consultants analysis. The local consultant will then submit the entire commentary by the relevant partner organisation to the IOB team together with his/her comments.

The detailed study constitutes the input for the field study by the IOB team, which will validate, supplement and analyse the data it has received. It will pay specific attention to cases where the findings and analysis of the local consultant conflict with the comments made by the organisation concerned. The IOB team will then finalise the country studies by formulating findings and highlights, both in general terms and, where possible, per CFO.

64

In total, the 6 CFOs work with the following number of partner organisations: India 329, Peru 138, Tanzania 123 (2002 annual reports). 65 Total CFO expenditure is as follows: India 34.6 million (2001) and 39.5 million (2002); Peru 17.9 million in both years; Tanzania 11.4 million (2001) and 10.9 million (2002). Total for 2001: 63.9 million; for 2002: 68.3 million. 66 The workshops have meanwhile taken place as part of the preliminary study. The number of partner organisations taking part was 20 in Dar es Salaam, 27 in Arusha (totalling 47 for Tanzania), 36 in Lima, 14 in Cusco (totalling 50 for Peru), 26 in New Delhi and 31 in Bangalore (totalling 57 for India).

90

Scope and representativeness of study The organisations targeted by the study are at three different levels. They are the Ministry of Foreign Affairs (DSI/MY), the six CFOs (ICCO, Cordaid, Novib, Hivos, Plan and Terre des Hommes) and the CFOs partner organisations. The CFOs support partner organisation activities in 95 countries (excluding regional/global activities).67 For the demarcation of this study at the lowest level, it was decided to investigate CFO partner organisations in Tanzania, Peru and India. CFO expenditure in these countries is substantial, and each CFO has agreements with many partner organisations. In total (2002 annual reports) the six CFOs work with 590 partner organisations in the three countries; in that year CFO expenditure in these countries totalled 58.4 million (all funds). Expenditure, all funds (x 1000)
Cordaid ICCO Novib Hivos Plan TdH Total India 2002 7,518 6,101 6,493 5,405 1,122 2,891 29,530 2001 11,036 7,101 6,951 6,840 128 2,558 34,614 Peru 2002 4,500 3,051 3,144 3,002 903 344 17,944 2001 6,150 2,759 4,951 3,082 659 373 17,974 Tanzania 2002 2001 3,201 2,883 1,634 1,883 2,177 2,154 1,423 1,878 2,228 2,154 273 429 10,936 11,381

In 2002 CFP expenditure68 totalled 357.2 million (including regional and global activities). CFP expenditure per country is 9.2 million for Tanzania, 13.1 million for Peru and 23.5 million for India (total of 45.8 million). Thus, MFP expenditure in these three countries equals 12.8% of total CFP expenditure in 2002. Total number of partner organisations involved in on-going activities in 200269
MFO Cordaid ICCO Novib Hivos Plan Terre des Hommes Total India 110 66 30 59 (17 activities) 47 329 Peru 54 23 26 20 (3 activities) 12 138 Tanzania 46 20 18 21 (6 activities) 12 123

Since its inception in 1965 (when it was set at NLG 5 million), the CFP budget has increased substantially. In 2003 it totalled 409.4 million, distributed as follows among the six CFOs. CFP grants (x million euros)
ICCO 110 Cordaid 107,6 Novib 97,8 Hivos 61,2 Plan 27,8 TdH 5,0

67 68

In 2002; 2002 annual reports. CFP expenditure for all CFOs except Terre des Hommes. Terre des Hommes has only received CFP funding since 1/1/2003. 69 The number of partner organisations comes from the annual reports and project books of the various CFOs. Plan India, Peru and Tanzania work with various CBOs, NGOs and local authorities. As far as ICCO and Cordaid are concerned (since the mergers with SOH, MEMISA and MIN/Caritas) and also Novib, it should be borne in mind that the totals include organisations that were originally funded individually from different sources (e.g. VPO).

91

In the selection of partner organisations, the proportionality of the CFP grants allocated to CFOs and of CFO expenditure per country was taken into consideration. This proportionality was reflected in the selection of partner organisations, for both the workshop and the detailed study. Apart from the distribution of organisations per CFO, partner organisations were selected according to the following criteria: type: intermediary, grassroots, service, network, civil society/membership organisation; sector of activity; size: number of employees, turnover, CFO contribution (small/medium/large organisation); age of the organisation and duration of its relationship with the CFO. Partner organisations invited to the workshop/participated in workshop/selected for study
CFO India Invited to workshop 14 17 13 12 7 5 68 Took part in workshop 13 13 11 11 4 5 57 Selected for study 4 4 (5) 4 4 2 (3) 2 20 Peru Invited to workshop 13 17 12 10 2 5 59 Took part in workshop 13 13 8 9 2 5 50 Selected for study (7) 5 (7) 4 4 (4) 3 1 (3) 2 20 Tanzania Invited Took to part in workworkshop shop 16 11 9 8 11 6 12 9 4 4 9 8 62 47 Select ed for study 6 4 5 3 (5) 1 1 20

Cordaid ICCO Novib Hivos Plan TdH Total

( ) Some partner organisations are partner of more than one (Dutch) CFO.

Type of organisation
Tanzania Invited partners 29 5 12 14 2 62 Selected partners 10 0 4 5 1 20 Peru Invited partners 36 1 15 0 7 59 2 20 Selected partners 14 India Invited partners 31 3 14 6 6 60* Selected partners 12 0 4 1 1 18**

Intermediary Target group/ grassroots organisation Service organisation Membership organisation Network organisation Total

*8 organisations not assigned. **2 organisations not assigned A total of 189 partner organisations (amounting to 32% of a total of 590 partner organisations) were invited to take part in the workshops in the three countries. Sixty partner organisations (10% of the total) were selected for the detailed study. The study does not claim to be representative. The detailed study of M&E in a selection of 10% of partner organisations of all CFOs in the three countries, however, provides an illustration of developments in this field.

92

4. Analysis Evaluation of M&E systems at the three different levels of the CFP will be based on the requirements set out in the guidelines for monitoring and evaluation in central government (RPER). The RPER formulates a number of general quality requirements with which information from monitoring systems and evaluation should comply. These general requirements are:70 validity (of the structure of information systems and of conclusions); reliability and accuracy (of the measurement methods used); utility (of the findings). To answer the questions formulated for the study it will be necessary to determine the extent to which guarantees are built into M&E systems to ensure that the information provided by the systems complies with requirements. With regard to the quality of evaluation studies, detailed criteria were defined in an earlier study of the evaluation function within the Ministry of Foreign Affairs (2002) and in the current study of the quality of evaluations completed in 2002. These criteria partly relate to the quality of information and partly to the quality of the systems themselves. As far as evaluation is concerned, the study will mainly focus on: the process whereby evaluations are set-up; the involvement of stakeholders and the transparency of this process; the independence and expertise of evaluators; internal checks on the validity, reliability and utility of evaluations. The quality requirements specific to monitoring systems are as follows:71 valid demarcation of the object of the system; correct determination of information needs; the information to be supplied must be in line with the intended use; proper system design; clear definitions of terms; comparability over a longer period (stability); adequate selection of units, and reliable and valid measurements of both units and their characteristics; valid analysis and conclusions/recommendations. 5. Planning The study will comprise the following stages: preparatory study, field study and report stage. Preparatory study This stage includes the following activities: establishing a database of partner organisations in the three countries selected; consulting CFOs on the set-up and implementation of the study; appointing a reference group; operationalisation of the study: defining the object of the study, approach, assessment framework and initial planning;
70 71

RPE (March 2002), p.33. RPE (March 2002), p.36.

93

 initial inventory of CFO monitoring and evaluation systems; preparatory missions to the three countries to collect information on the various partner organisations: inventory of informants opinions on partner organisation M&E of results; conducting workshops with partner organisations in geographical clusters in order to: explain the study; note the views of senior staff on M&E; present and explain the questionnaire; interviewing and selecting local consultants; collecting information on partner organisations by means of the questionnaire; completing the selection of partner organisations for a more detailed study on the basis of available information (database and completed questionnaire). The outputs of the preparatory stage are a complete database of partner organisations in the three selected countries and a TOR for the detailed study by local consultants. Field study This stage includes the following activities: a detailed inventory of the M&E systems used by the six CFOs: studying relevant documents and interviewing staff; studying the policy framework and DSI/MYs M&E policy and systems relevant to the CFP; studying selected partner organisations (by local consultants); supervising local consultants and feedback on draft reports per partner organisation; field missions to validate and analyse local consultants study findings and to collect supplementary information (this will be done by the IOB inspector and the lead consultant). The output of this phase are detailed mission reports by the IOB inspector and the lead consultant, reflecting findings and challenges in the three countries concerned, as well as partner organisation reports drafted by the local consultants. Reporting stage Formulation of the synthesis report. 6. Resources and organisation The IOB inspector bears the final responsibility for all aspects of the study, specifically: drafting the final terms of reference; the organisation and supervision of the various sub-studies; the collation and analysis of the study findings; drafting the final report. Within IOB, inspector F. van der Kraaij will comment on draft reports. The IOB inspector will be assisted by a lead consultant to be contracted for the study. This consultant, who will have extensive experience and expertise in the field of M&E at project level and at policy level and considerable experience in carrying out policy evaluations, as

94

well as thorough knowledge of the VBTB-RPER process, will be involved throughout the study. A research assistant will support the preparation of the study. This will includes selecting countries and organisations (and establishing the database on which to do so), and drafting, distributing and processing questionnaires and subsequent information, desk studies, draft reports by local consultants, etc. A reference group will supervise the study. The members are: IOB: Henri Jorritsma (chair) PME: Piet de Lange DSI/AI: Floris Blankenberg external experts (Kees Biekart, Fons v.d. Velden) DSI/MY representative chair of the working group on evaluation and policy of the co-financing platform (MBN). The reference group will meet at least three times on the following occasions: TOR and operationalisation (January 2004) reports of the three country studies (May/June) discussion of draft final/synthesis report (August/September). 7. Budget EUR 49.187 1.500 40.000 40.000 10.000 21.000 161.687 8.013 169.700 101.060 15.000 12.750 5.000 85.000 15.000 29.000 50.000 312.810 15.690 328.500 498.200

Budget for preliminary study: Main consultant cf. contract Internal flights consultant in three countries Costs 3 x 2 workshops with partner organisations 6 local consultants (in 2003) DSA and travel expenses local consultants (in 2003) Junior researcher IOB (5 months) Total preliminary study 5 percent contingencies Total preliminary Budget for main study Main consultant (155 days) Tickets main consultant DSA (25 x 3 x EUR 170 average) Costs reference group Costs local consultants (in 2004) DSA and travel expenses in 2004 Junior researcher (7 months) Translation costs, printing reports Total main study 5 percent contingencies Total main study Total costs study

95

96

Appendix 3 Abbreviations

97

Appendix 3: Abbreviations
APRODEV CBO CCCD CEO CFO CFP CPME CSO DRAM DSI/ MY DSI/AI EU FEZ fte GOM IBO ICT INK IOB ISO M&E MBN MDG MFI NCDO NGO ODA OOPP PME PO PRSP SMART TdH ToR UN R&D Q&C Association of Protestant Development Agencies in Europe Community Based Organisation Child Centred Community Development Chief Executing Officer Co-Financing Organisation Co-Financing Programme Corporate Planning, Monitoring and Evaluation Civil Society Organisation Objectives, Results, Activities, Resources Social and Institutional Development Department, Social Co-operation and Institutional Development Division Social and Institutional Development Department, Poverty Policy and Institutional Development Division European Union Financial and Economic Affairs Department Full time equivalent Gemeenschappelijk Overleg Medefinancieringsprogramma Inter Departmental Policy Review Information and Communication Technology Instituut Nederlandse Kwaliteit (Institute Dutch Quality) (see appendix 4: glossary) Policy and Operations Evaluation Department International Organisation for Standardisation (see appendix 4: glossary) Monitoring and Evaluation Medefinanciering Breed Netwerk Millennium Development Goals Micro Finance Institute Nationale Commissie voor Internationale Samenwerking en Duurzame Ontwikkeling Non-Governmental Organisation Official Development Assistance Objective Oriented Project Planning Planning, Monitoring and Evaluation Partner Organisation Poverty Reduction Strategy Paper Specificity, Measurable, Agreed upon, Realistic and Time bound Terre des Hommes Terms of Reference United Nations Research and Development Quality and Control

98

Appendix 4 Glossary

99

Appendix 4: Glossary
Accountability Obligation to demonstrate that work has been conducted in compliance with agreed rules and standards or to report fairly and accurately on performance results vis--vis mandated roles and/or plans. This may require a careful, even legally defensible, demonstration that the work is consistent with the contract terms Accountability in development may refer to the obligations of partners to act according to clearly defined responsibilities, roles and performance expectations, often with respect to the prudent use of resources. Effect Intended or unintended change due, directly or indirectly, to a intervention. Evaluation The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation lessons learned into the decision-making process of both recipients and donors. Feedback Transmission of findings generated through the evaluation process to parties for whom it is relevant and useful so as to facilitated learning. This may involve the collection and dissemination of findings, conclusions, recommendation and lessons from experience Input The financial, human and material resources used for the development intervention. Impact Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Monitoring A continuous function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an on-going development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Outcome The likely or achieved short-term and medium-term effects of an interventions outputs. Output The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes. Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. Result based A management strategy focusing on performance and achievement of outputs, outcomes and management impacts Source: DAC Glossary INK- model The INK-model is a management model of the Dutch Institute for Quality management. The management model is geared towards learning, and improving the organisation in 9 areas of attention: 5 organisational areas (Leadership, Strategy and Policy, Management of Employees, Management of Resources, Management of Processes) and 4 result areas (Employees, Clients and Suppliers, Civil Society and Board and Financiers). ISO 9001:2000 Norm of the International Standard Organisation for quality management systems. The ISO norm 9001:2000 norm specifies requirements for the quality management system, by which an organisation can prove its capability to deliver products that comply to the requirements of the client and to prevailing the regulations that are enforced. Source: www.ink.nl and www.iso.org

100

Appendix 5 Partner Organisations

101

Appendix 5: Partner Organisations of the CFOs

5.1 Acronyms of organisations for detailed study Acronym Full name India
Butterflies CECOEDECON
CHC CWS FEDINA

FODRA
HCHW

IDE (I)
MMP/Samata MYRADA

Navdanya
QWRIDS RAID

RUPCHA SACCS
SAMSKAR

Seva Mandir
SIFFS

SRUTI Vishaka

Butterflies Centre for Community Economics and Development Consultants Society Community Health Cell Centre for World Solidarity Federation for Educational Innovations in Asia Fountain of Development Research and Action Hyderabad Council for Human Welfare International Development Enterprises India Mines, Minerals and People Myrada Navdanya Trust Quarry Workers and Rural Integrated Development Society Rural Association for Integrated Development Rajasthan Uttar Pradesh Catholic Health Association South Asian Coalition on Child Servitude Samskar Seva Mandir South Indian Federation of Fishermen Societies Society for Rural Urban and Tribal Initiative Vishaka Group for Womens Education and Research Asociacin Arariwa para la Promocin Tcnico Cultural Andina Centro Andino de Educacin y Promocin Jos Mara Arguedas Centro de Capacitacin Agroindustrial Jess Obrero Centro de Estudios Cristianos y Capacitacin Popular Centro de Estudios para el Desarrollo y la Participacin Centro para el Desarrollo de los Pueblos Ayllu Centro de Desarrollo de la Salud Integral Centro para el Desarrollo del Indgena Amaznico Centro de Desarrollo e Investigacin de la Selva Alta Centro de Investigacin, Educacin y Desarrollo - Puno Asociacin de Promocin y Desarrollo el Taller Centro de Educacin y Comunicacin Guaman Poma de Ayala Instituto para una Alternativa Agraria Centro de Investigacin, Documentacin, Educacin, Asesora y Servicios Asociacin Nacional de Institutos para el Desarrollo del Sector Informal Asociacin Iniciativa Comunal de los Andes Instituto Peruano de Paternidad Responsable Asociacin Kallpa para la Promocin Integral de la Salud y el Desarrollo Plan Per Red de Accin en Alternativas al Uso de Agroqumicas Agricultural Development Programme Mbozi Trust Fund Catholic Relief Services Christian Social Services Commission Community Research and Development Foundation for International Community Assistance HakiElimu Organisation of Orkonerei Pastoralist Advancement Ileje Rural Development Trust Integrated Development and Relief Office

Peru
ARARIWA CADEP CCAIJO Cecycap CEDEP Cedep Ayllu CEDESI CEDIA CEDISA CIED Puno El Taller Guaman Poma IAA IDEAS IDESI Nacional INCA INPPARES KALLPA PLAN Per RAAA

Tanzania
ADP Mbozi CRS CSSC CORDS Finca HakiElimu Ilamaratak Lorkonerei IRDT IDRO

102

Kinnapa Kivulini Kuleana LHRC MRHP Nafasi Plan SAIPRO SHDEPHA+ TDFT TGNP

Kinnapa Development Programme Kivulini Kuleana Legal and Human Rights Centre Mwanza Rural Housing Programme Nafasi Sports Project Plan Tanzania Same Agriculture Improvement Programme Trust Fund Service Health and Development for People Living Positively with HIV/AIDS Tabora Development Foundation Trust Tanzania Gender networking Programme

5.2 Characteristics of partner organisations in studied in detail

Cluster New Delhi, Turnover 2002 India (x1000 USD) Butterflies 152.6 CECOEDECON 678.2 FODRA 102.5 IDEI 1,822.9 Navdanya Trust 318.4 RUPCHA 45.8 SACCS/AVA 669.4 Seva Mandir 1,249.4 SRUTI 146.1 VISHAKA 117.7 Cluster Bangalore, India CHC 89.4 CWS 1,264.7 FEDINA 381.6 HCHW 109.0 Mm&P/ Samata 174.4 MYRADA 5,451.4 Q.W.R.I.D.S. 52.3 RAID 47.9 SAMSKAR 665.1 SIFFS 1,853.5 Cluster Lima, Peru CEDEP 684.0 CEDESI 37.0 CEDIA 300.0 CEDISA 650.0 IDEAS 870.0 IDESI Nacional 1,850.0 INPPARES 1,820.0 KALLPA 728.0 PLAN Peru 2,847.0 RAAA 337.0 Cluster Cusco, Peru ARARIWA 1,400.0 CADEP 453.0 CCAIJO 420.0 Cecycap 400.0 Cedep Ayllu 765.0 CIED Puno 41.0 El Taller 350.0 Guaman Poma (2003) 1,700.0 Total Staff 54 175 45 150 105 5 55 265 11 186 23 44 28 30 28 320 235 44 220 150 38 7 32 30 44 16 582 62 88 15 55 24 44 26 34 10 33 71 Type of organisation Service provider Intermediary organisation Intermediary organisation Service provider Intermediary organisation Service provider Intermediary organisation Intermediary organisation Intermediary organisation Intermediary organisation Service provider Intermediary organisation Intermediary organisation Intermediary organisation Network organisation Intermediary organisation Intermediary organisation Service provider Service provider Grassroot organisation Intermediary organisation Service provider Service provider Intermediary organisation Intermediary organisation Network organisation Intermediary organisation Service provider Intermediary organisation Network organisation Intermediary organisation Service provider Intermediary organisation Intermediary organisation Intermediary organisation Intermediary organisation Intermediary organisation Intermediary organisation CFO ICCO ICCO Cordaid ICCO Hivos Cordaid Novib ICCO, Plan Novib Hivos Cordaid Hivos, Novib Cordaid Novib ICCO Novib TdH TdH Plan Hivos Novib TdH Hivos, Novib Hivos ICCO Novib (ICCO) TdH Cordaid, ICCO, TdH Plan Hivos Cordaid, ICCO ICCO Cordaid Cordaid Cordaid, Hivos ICCO Cordaid Cordaid

103

321.0 INCA 230.0 Cluster Arusha, Tanzania CORDS 200.0 CRS 965.8 IDRO 1,176.7 Ilaramatak Lorkonerei 369.5 Kinnapa 208.2 Kivulini 148.2 Kuleana 784.5 MRHP 98.1 Nafasi 52.3 SAIPRO 99.5 Cluster Dar-es- Salaam, Tanzania ADP Mbozi 180.0 CSSC 1,631.4 Finca 1,900.0 HakiElimu 673.3 IRDT 190.2 LHRC unknown Plan Tan 5,932.0 SHDEPHA + 154.8 TDFT 201.2 TGNP 701.1

IAA

16 21 16 17 42 32 14 18 30 65 3 12 24 33 142 26 15 33 150 16 18 20

Intermediary organisation Intermediary organisation Grassroot organisation Intermediary organisation Intermediary organisation Grassroot organisation Grassroot organisation Grassroot organisation Service provider Intermediary organisation Intermediary organisation Grassroot organisation Service provider Intermediary organisation Intermediary organisation Intermediary organisation Service provider Intermediary organisation Intermediary organisation Service provider Intermediary organisation Network organisation

ICCO Novib Cordaid Cordaid Cordaid Novib Novib Hivos Hivos, ICCO ICCO TdH Novib Cordaid Cordaid, ICCO Hivos Novib Cordaid Novib Plan Hivos ICCO Cordaid, Hivos

5.3 Cross table of 60 partner organisations studied, divided in type of organisation and turnover
Type of organisat. Turnover 2002 < 300,000 Intermediary organisations CIED Puno, Fodra, HCHW, INCA, MRHP, Nafasi, Qwrids, SRUTI, TDFT, Vishaka Ccaijo, Cecoedecon Cecycap, Cedep, Cedep Aylu, Cedisa, CRS, Fedina, HakiElimu, IAA, IDEAS, LHRC, Navdanya Trust , SACCS/ AVA, el Taller Arariwa, CSSC, CWS, Finca, Guaman Poma, IDRO, INPPARES, Myrada, Plan Peru, Plan Tanzania, Seva Mandir Service providers ADP Mbozi, Butterflies, Cedesi, CHC, IRDT, SHDEPHA+, RUPCHA, RAID Cadep, Cedia, Kallpa, Kuleana, Samskar Network organisations mm&P/ Samata Grassroot organisations CORDS, Kinnapa, Kivulini, SAIPRO

300,000-1,000,000

RAAA, TGNP

Ilaramatak Lorkonerei

> 1,000,000

IDEI

IDESI Nacional

SIFFS

104

5.4 Distribution of assessment of operational information including direct outcomes.

Across the countries studied
Number of partner organisations out of: 20 partner organisations 20 partner organisations India Peru + ? + ? Collection of information Regular Standardised Relevance for use Accountability Decision making Learning
Can information be transferred to other stakeholders

20 partner organisations Tanzania + ? 0 0 0 0 0 0 18 18 18 19 18 17 1 1 1 0 0 2 1 1 1 1 1 1 0 0 0 0 1 0

18 13 16 18 14 15

2 7 4 1 5 5

0 0 0 0 0 0

0 0 0 1 1 0

20 17 20 19 17 20

0 3 0 1 2 0

0 0 0 0 1 0

Quality of information Validity 13 6 0 1 17 1 0 Reliability 4 3 0 13 6 4 0 Verifiability 8 9 1 2 17 2 0 + positive, present but not fully satisfactory, - negative, ? no information

2 10 1

18 5 17

1 7 1

1 1 1

0 7 1

Across the sizes (turnover ) of organisations studied

Number of partner organisations out: 23 partner organisations 23 partner organisations < 300.000 300.000 - 1.000.000 + ? + ? Collection of information Regular Standardised Relevance for use Accountability Decision making Learning
Can information be transferred to other stakeholders

14 partner organisations > 1.000.000 + ? 0 0 0 0 0 0 13 13 14 13 12 13 1 1 0 1 2 1 0 0 0 0 0 0 0 0 0 0 0 0

20 18 19 21 18 17

2 4 3 0 2 5

1 1 1 1 1 1

0 0 0 1 2 0

23 17 21 22 19 22

0 6 2 1 3 1

0 0 0 0 1 0

Quality of information Validity 18 3 1 1 17 4 0 Reliability 6 6 1 10 4 6 0 Verifiability 15 4 2 2 16 5 0 + positive, present but not fully satisfactory, - negative, ? no information

2 13 2

13 5 11

1 2 2

0 0 0

0 7 1

Across the types of organisations studied

Number of partner organisations out of: 36 partner organis. 6 partner organ. Intermediary Grass root + ? + ? Collection of information Regular Standardised Relevance for use Accountability Decision making Learning
Can information be transferred to other stakeholders

4 partner organ. Network + ? 4 3 4 4 3 3 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0

14 partner organis. Service + ? 12 8 11 13 10 11 1 5 2 0 2 2 1 1 1 1 1 1 0 0 0 0 1 0

34 31 33 33 30 32

2 5 3 2 4 4

0 0 0 1 1 0

0 0 0 0 1 0

6 6 6 6 6 6

0 0 0 0 0 0

0 0 0 0 0 0

0 0 0 0 0 0

Quality of information Validity 30 6 0 0 6 0 0 0 4 Reliability 10 13 13 0 0 1 0 5 1 Verifiability 24 9 1 2 6 0 0 0 4 + positive, present but not fully satisfactory, - negative, ? no information

0 0 0

0 0 0

0 3 0

8 4 8

2 0 2

1 1 1

3 9 3

105

106

Appendix 6 Co-Financing Organisations

107

6.1 Cordaid
6.1.1 Background Cordaid is organised around two main components: marketing and projects. The project component is sub-divided in 5 regional units and the unit financial innovation and emergency assistance. Each regional unit has several sub-regional teams, composed of a programme officer, a programme assistant, a financial staff member and a financial assistant. These teams are responsible for the entire activity cycle with the partner organisations. In addition, Cordaid has several staff units, among which the unit Quality and Strategy. Cordaid has two regional offices: in Kinshasa and Nairobi. Total staff is 268 (232 fte). Cordaids M&E system, approved in 2002, is based on the concept of the aid-chain consisting of a series of autonomous organisations, each with their own responsibilities and policy priorities. Cordaid bases its M&E system on its own role and responsibilities in this aid chain. It identifies four domains: target group, partner organisations, relationship between Cordaid and the PO, and Cordaid. Accountability in the domain Cordaid is complete. Steering and accountability in the domains relationship PO Cordaid and PO are less direct. Cordaid has tentatively identified a set of indicators for those three domains. Cordaid perceives that it cannot be directly responsible for the domain target group72. Cordaid plans to strengthen its result-focus by following four steps in each of these domains: Defining expected results Achieving these results Measuring, reporting and accounting for these results Learning from the results Some of the instruments, that are defined for the different steps, need to be fine-tuned in the coming years. For example, the System of Story Telling (the collection of experience stories about the processes that occur). Cordaid formulates annual working plans (overall and regional), stating objectives, input and results in the domains PO and Cordaid. For monitoring of these plans, Cordaid has developed input and process indicators, which are monitored throughout the year. The indicators cover aspects such as disbursements per theme, visits to partner organisations, participation in conferences (input indicators); selection of new partner organisation, termination of existing partner organisations (process indicators). Output indicators are not (yet) incorporated in regular progress monitoring. For its internal monitoring, Cordaid has developed a set of planning and monitoring instruments. Expected results are agreed between senior and middle-level management and between middle level- and operational management. These agreements constitute the basis for regular monitoring in internal meetings. The Management Team Projects meets twice monthly to advise the Director Projects about the execution of activities and monitors progress in general. 6.1.2 Monitoring of partner organisations

For the monitoring of the performance of partner organisations Cordaid uses the following instruments.

72

See: monitoring en evaluatiesystematiek dated 24-06-02, p.7.

108

6.1.2.1 Reporting by partner organisations Partner organisations report to Cordaid annually, sometimes bi-annually, both in a narrative and a financial report. Cordaid issues partner organisations with guidelines for these reports and guidelines for external accountants. The reporting guidelines (including a standard outline) are intended to enhance the uniformity and quality of these reports. If partner organisations have developed an adequate reporting structure for other donors or internally, Cordaid may agree to use that reporting format. 6.1.2.2 Other monitoring Cordaid considers visits to partner organisations by its staff as an important element in linking its policy development with the operational reality of the partner organisations. Visits to organisations are geared towards exchanging information with staff of partner organisations and with other stakeholders, discussing and solving operational problems, giving feedback on reports and to obtain an impression of the impact of the activities of the partner organisation. Visits to partner organisations are generally brief, often one half to a maximum of one full day. Recently Cordaid has extended the field visits of its staff to 5 weeks in order to meet their partners more frequently and more extensively at different formal and more informal occasions/ platforms. From all visits mission reports are drafted. Approximately once a year Cordaids regional units organise consultations at regional or continental level with strategic partners. These partner consultations regroup partner organisations active in a thematic field to exchange information. The consultations provide a framework for exchange of ideas and experiences focused on specific themes, with a depth that goes beyond the regular reporting system and the ad-hoc contacts that are usually concerned with day-to-day management issues. 6.1.3 Evaluations

6.1.3.1 Activity-level evaluations Most evaluations that are undertaken are activity-level evaluations. The main aim of activitylevel evaluations is learning, not accountability for the funds used. As a rule, activities are evaluated at least once in two agreement cycles (of usually 3 years). Activity-level evaluations can be initiated both by Cordaid or by the PO. Activity-level evaluation are used to assess the effectiveness and relevance of the execution of the agreed activity. Contractually Cordaid is entitled to formulate the ToR and recruit evaluators. In practice this is only the case when problems are expected with the evaluation. Usually the partner organisation is responsible for the formulation of the ToR, the selection of the evaluators and the management of the evaluation. Cordaid reserves the right to approve the ToR and the evaluators that are being selected by the partner organisation. The evaluation report is appraised by the Cordaid team, and this appraisal is communicated to the partner organisation. Occasionally evaluations result in reconsideration of agreements or the intended continuation of an agreement. Evaluations are important for Cordaid for the management of the agreements. The outcome of the evaluation is discussed with the partner organisation, and lessons learned are distilled. The following table presents the number of activity-level evaluations in 2002 and 2003.

109

Table 6.1: Cordaid activity-level evaluations

Year 2002 2003 Evaluations unknown 104 Total number of partner organisations 1869 1975

Source: annual reports 2002 and 200373

In 2003 Cordaid has analysed the 104 activity-level evaluations using Cordaids forms for evaluation results. On the quality of 17 % of these evaluations comments were made (deviations from ToR and too little attention given to certain subjects, specifically to outcome related issues74). 6.1.3.2 Programme evaluations Cordaid executes (mainly external) programme evaluations, which are an integral part of the policy cycle of themes and policy priorities. They are often aimed at new themes or policy priorities, and concern complex interventions. The unit Quality and Strategy reports the intention to execute two programme evaluations annually, but in the annual reports a lower number is reported. Other external programme evaluations are executed as part of the MBN schedule of programme evaluation. 6.1.4 Use of feedback information

Use of feedback information obtained from monitoring partner organisations The aim of progress reports is to follow execution of the agreements, to allow the unit teams to intervene timely when required. In a formal sense, a positive assessment of the progress reports is required before the next instalment can be transferred. The information that is collected during the execution of the agreement is also used in the context of the formulation and negotiation of a new agreement. Usually key elements of the information comprised in the progress reports has already been communicated between the programme staff and the partner organisation by telephone, e-mail or during the regular visit of programme staff to the partner organisation or during other contacts. Use of feedback information obtained from evaluations Activity-level evaluations are expected to contribute to learning, specifically at the level of the programme officer and the partner organisation75. Cordaid uses activity-level evaluations for management of activities under implementation and support decision making about continuation of the activity or the relationship with the partner organisation. The outcome of activity-level evaluations can be included in the follow-up documents elaborated for the continuation of the relationship with the partner organisation concerned. The lessons learnt which are valuable for the entire organisation, are forwarded to the Bureau Policy and Projects and to the corresponding thematic groups. Summaries of the activity-level evaluations, with conclusions and lessons learned are included in the Cordaid annual report. Internal programme evaluations are expected to contribute to innovation and policy development. These evaluations are specifically designed from this perspective. They therefore usually focus on existing priority themes, emerging themes and priorities as well as technically complex or politically sensitive activities.

73

The total number of partner organisations include the partners in exit phase, short-term relationships of the humanitarian aid unit, one-time funding and micro-finance. The partner portfolio of Cordaids priority themes consists of 1252 POs in 2003. 74 Jaarverslag Cordaid 2003 75 See kwaliteitshandboek Cordaid section 2.5.2.2

110

Learning within Cordaid is promoted by organising 5-10 presentations and discussions annually per department (lekgotlas or palavers), introduced early 2003. During these presentations the emphasis is on drawing the lessons from practical experience on the basis of cases. These cases may be individual partner organisations or themes or strategic issues. A first review of the practice of lekgotlas is positive, but reveals that a best practice still has to emerge.

6.2 Hivos
6.2.1 Background Management of Hivos is composed of its Executive Director and the Director Programmes and Projects. The department Programmes and Projects includes the regional units Africa, Asia and the former Soviet Union and Latin America & the Caribbean. In the Hivos regional departments a distinction is made between programme officers and sectoral specialists. In addition Hivos has three staff bureaux: ICT, Assessment Evaluation and Control (TEC) and Communication. Regional offices are established in Zimbabwe, India and Costa Rica, another regional office is established in Indonesia in January 2005. Total Hivos staff amounts to 111 fte (78 at headquarters and 38 at regional offices), or 132 staff. Hivos M&E system For its internal monitoring, Hivos uses (continental) plans and internal contracts, where results at the levels of Hivos and the partner organisations are specified. This is used as the basis for internal monitoring. The M&E system follows the activity cycle. It specifically focuses on objectives of the partner organisation with regard to development of the central policy domains Civil Society and Economy and Sustainable Development, and objectives regarding capacity development of the partner organisation itself. During the years 2003 and 2004 the Hivos M&E system was revised and strengthened. Four important elements are: Strengthening monitoring Review of the Hivos evaluation policy Strengthening expertise within Hivos Capacity building among partner organisations 6.2.2 Monitoring systems and instruments

In 2004 Hivos has conducted several regional partner consultations as input for the development of measurable indicators. Hivos staff is trained in the use of these indicators. As from October 2004, for new contracts, a limited number of indicators is identified in consultation with the partner organisation. The set of indicators is based on the objectives and expected results of an activity and will be monitored during the contract period. In addition to these indicators, strengths and weaknesses that are identified during the appraisal of the partner organisation, are monitored. For the monitoring of partner organisations Hivos uses different instruments. 6.2.2.1 Reporting by partner organisations All partner organisations are expected to submit an annual narrative and financial report. Hivos does not impose a reporting format, but it allows partners to develop their own formats

111

or use formats required by other donors, in order to limit the administrative burden on partner organisations. Nevertheless, Hivos has formulated qualitative guidelines for the annual narrative progress and financial report, which are provided as an annex in the General Conditions. 6.2.2.2 Other monitoring Usually contact with partner organisations is intensive during the preparation of an agreement. During the implementation phase organisations are visited by programme staff and senior sector staff as a rule at least once a year, with visits taking between one half and a full day. Staff from the regional offices also visits the organisations on average once a year, but such visits generally tend to be longer (several days) and include visits to field locations. Visits are documented in mission reports. Regular contacts (by e-mail and telephone) take place between Hivos staff and partner organisations on a great variety of subjects. The aim of this regular contact is not only to follow the proper execution of the agreement, but also to provide more general support and build a relationship between Hivos and its partners. 6.2.3 Evaluation

The main aim of Hivos evaluations is learning. In 2003 an internal assessment has been made of the quality of evaluations. The results of this assessment have contributed to an updated evaluation policy, to be introduced in 2005. At present most evaluations are activity-focused. As these are context and organisation specific, aggregation of outcomes is problematic. Hivos aims to introduce an evaluation approach that enhances the possibility to aggregate results achieved by its partners. The introduction of a diversified menu of types of evaluations is envisaged. This includes self assessments and accompanied self assessments in addition to activity-level evaluations at the level of the partner organisation, as well as internal and external programme evaluation and longer term impact assessment at the level of Hivos itself. Hivos intends to put more emphasis on programme level evaluations at programme/thematic level that will be more relevant for Hivos itself. The responsibility for activity-level evaluations should be increasingly assumed by the partner organisations themselves. 6.2.3.1 Activity-level evaluations A activity-level evaluation (mid-term review) can be executed for the following reasons: to assess the organisational quality of the organisation to assess the effectiveness of (the activities of) an organisation to assess if continuation of the co-operation with a partner organisation is desirable. to formulate recommendations for policy development of the partner organisation Hivos policy is to evaluate at least 5% of its partner organisations every year. Only partner organisations that are supported with more than 45,000 per year or partner organisations that are specifically important from a policy point of view are evaluated. The evaluations are done by external, preferably local, experts. To enhance the quality of the evaluations Hivos uses a standard ToR, which can be modified and specified for each evaluation depending on the specific context and purpose of the evaluation. Hivos has specified in its contracts that Hivos is responsible for the formulation of the ToR and the selection of the evaluator. In most cases the ToR is prepared by the programme officer in consultation with the partner organisations. Another instrument used by Hivos since 2001 to enhance and control the quality of the evaluation, is the Hivos

112

Consultants Database. In 2002 and 2003 the number of evaluations executed by Hivos is as follows. Table 6.2: Number of Hivos activity-level evaluations 2002 and 2003
Year 2002 2003 Evaluations 48 62 Total number of partner organisations 766 864

Sources: annual reports 2002 and 2003

Since 1999 Hivos conducts Micro Finance Institute (MFI) assessments on annual basis. MFI assessments are effect measurements at the client level of the MFI. The assessments are an important evaluation instrument for Hivos in the area of micro credit. The succession of similar evaluations over several years enables to extract more general findings, which can be used for policy evaluation and policy development. 6.2.3.2 Financial inspections Hivos reserves the right to execute financial inspections, whenever it is considered appropriate. Annually Hivos executes these inspections with 6 to 8 partner organisations. The inspector assesses the quality and transparency of the financial management and the quality and independence of the external financial auditors. 6.2.3.3 Programme evaluations In the years 2002 till 2004 Hivos has executed several programme evaluations. Hivos also participates in the MBN programme evaluations executed in the period 2003-2006. 6.2.4 Use of feedback information

Use of feedback information from monitoring The programme staff assesses the annual reports in relation to the agreement and if necessary requests additional information. The financial staff processes the financial information in view of the agreement. Programme staff then processes both the narrative and financial reports comparing progress in both fields. Hivos reacts to the partner organisation within three months after reception of the reports. Sectoral staff use the narrative progress reports to monitor the implementation of sectoral policies. For policy adjustments, senior (sector) policy staff gather relevant information from the different regional units. Within the different units, fortnightly meetings take place where progress of activities is discussed, as well as any problems in the relationships with partner organisations. The units report annually to Hivos management. These reports are more focused on the performance of the units themselves. Progress reports of the partner organisations do not play an important role in the reports of the units to management. A new computerised project administration system is introduced that also allows incorporation of result information. Indicators are developed for the different levels at which Hivos and its partners operate. To assess the realisation of outputs and the achievement of objectives by partner organisations a rating system is developed to assess the progress reports from 2003 onwards. At termination of the agreement a final rating is given on the basis of all available information (annual reports, evaluations and trip reports). It is expected that the rating system will allow the aggregation of achievements at the level of Hivos policy priorities.

113

Use of feedback information from evaluation The findings of activity-level evaluations play a role in the decision to continue the relationship with the partner concerned. There is no mechanism in place to share the lessons learned within the entire organisation. For follow-up of programme evaluations an internal process is followed, involving management. Programme evaluations are discussed in all units of Hivos and are used for policy development. Processes to strengthen organisational learning are under development in the organisation. Hivos organises regularly thematic meetings for specific themes and broader reflection sessions.

6.3 ICCO
6.3.1 Background ICCO has four main components: programme management, programme support, central services/management and communication/PR. Total staff amounted to 227 persons, or 189.7 fte (2003). ICCO is considering the establishment of regional offices. The programme management component includes three regional units, and two units for technical assistance and emergency aid respectively. In the regional units programme staff are responsible for the contact with the partner organisations. The programme support unit includes the Policy and Quality unit. ICCOs M&E system ICCO distinguishes between two PME sub-systems: the PME of partner organisations and the ICCO internal PME. The PME of partner organisations concerns the project management cycle of the development intervention. This generates the information that feeds into the ICCO internal PME cycle through proposals, progress reports and evaluations. This information is complemented by information collected during visits by ICCO staff to the organisations concerned. Programme staff regularly rate each activity on a number of criteria, among others progress on thematic criteria. Key information of each individual activity is stored in a central database (IBIS). 6.3.2 Monitoring of the partner organisations

During the different stages of the project management cycle several memoranda are drafted: the intake memorandum, the appraisal memorandum (kidbemo), the progress memorandum and the termination memorandum. The kidbemo prepares decision making for funding (and the accountability for it). Key points for monitoring are formulated in the contract. 6.3.2.1 Reporting by partner organisations All partner organisations are requested to submit an annual narrative and financial progress report to ICCO. The reporting format is based on the logical framework that has been developed for plan formulation. ICCO is lead agency in the effort to harmonise reporting requirements within Aprodev.76

76

Aprodev is a European Cooperation/ Network of 16 like-minded, Protestant-Christian organisations.

114

6.3.2.2 Other monitoring instruments All ICCO programme staff are expected to make a tour to visit partner organisations at least once a year to discuss past achievements and experiences, as well as plans for the future. The visits are expected to give the responsible programme staff a better understanding of the way a partner organisation operates. Staff is expected to prepare a mission report after completion of their visits. In between visits, there is usually regular contact by telephone or e-mail between the programme staff and representatives of the partner organisations. Partner consultations are held at local, regional or international level. The partner consultations can have a country or a thematic approach/ theme and cover policy developments. It is estimated that approximately one-third of the partner organisations participates occasionally in such consultations. 6.3.3 6.3.3.1 Evaluation Final memorandum

At termination, a memorandum is drafted assessing the achievement of expected results, the disbursement of funds and the development of the partner organisation. The memorandum is intended to formulate the required follow-up, including termination or continuation of the agreement, but in practice it is often formulated after such a decision has already been taken. 6.3.3.2 Activity-level evaluations The general policy is to evaluate activities once during their lifetime. All partner organisations have to be evaluated every 3-5 years. The purpose of the evaluation is discussed between ICCO and the partner organisation concerned. ICCO has developed a standardised ToR and guidelines for contracting evaluators. Usually parts of the responsibility for the management of evaluations are sub-contracted to a local organisation. The ICCO programme staff has to approve the ToR and the evaluators that are recruited. Table 6.3: Number of activity-level evaluations in 2002 and 2003
Year 2002 2003 Activity-level evaluations 87 103 Total number of partner organisations 883 969

Source: ICCO annual reports 2002 and 2003

With regard to activity-level evaluations, ICCO has observed that it had no coherent policy with regard to evaluation, that it does not attain the envisaged evaluation frequency, and that better use could be made of the evaluations that were done. It appeared that the evaluations results were generally in line with the evaluation questions, but little could be said about the methodological quality and relevance of the evaluations. The ICCO policy with regard to evaluation is under revision. The aim of the new evaluation policy is to assure efficient use of resources allocated to evaluation and effective use of the evaluation findings for management decision making and broader learning in the organisation. The development of the evaluation policy includes formulation of guidelines, and explanation of the types and role of evaluation in the ICCO PME cycle. The updated evaluation policy is expected to be finalised in 2004. Staff will be introduced to the new policy and trained in its application.

115

6.3.3.3 Programme evaluations ICCO conducts internal studies and thematic evaluations under the responsibility of the unit Policy and Quality. The thematic evaluations provide insights in the progress and results of the policy themes. ICCO also conducts MFI assessments and participates in the MBN programme evaluations. Regional policy is evaluated (internally) at the end of the subsidy period on the basis of the systematic annual review of activities. Key issues are the appraisal of results and their contribution to the three key policy priorities of ICCO. An external mid-term review of the ICCO business plan 2003-2006 has been conducted. 6.3.4 Use of feedback information

Use of feedback information from monitoring The programme staff appraises the progress report in view of the contractual agreement with the partner organisation and other decisions that have been taken. The financial and administrative staff appraise the financial report. After appraisal of the progress report, programme staff draft a progress memorandum (at least once a year). Information from other sources (mission reports and regular contacts) is also used for this memorandum. The memorandum presents an overview of the progress of the implementation of activities, the realisation of outputs and achievement of outcomes and lists agreed follow-up action. The manager of the regional unit should approve the progress memorandum. All teams draft quarterly reports, which include information form the field, obtained form the progress reports. The quality managers integrate these quarterly reports into one report, which is submitted to the management team. In 2002 ICCO has initiated a system to assess the performance of the partner organisations in relation to a set of standard criteria. It developed indicators for 6 policy priorities to determine the contributions of the POs in these fields. The indicators are: relation direct poverty alleviation and civil society development, gender policy, autonomy and sustainability, economic sustainability, basic services and effective PME system. ICCO is conducting a study on the reliability of the internal use of these indicators in order to improve (the use) of the indicators and is developing thematic indicators for the assessment of the three policy themes and their objectives. ICCO intends to integrate these indicators into the financing cycle. Use of feedback information from evaluation Activity-level evaluations are mainly used for communication with the partner organisation concerned and decision making about the continuation of the partner relationship. The funding instruments are being adapted to make more systematic use of evaluation results in the appraisal of new proposals. In the new appraisal memorandum explicit reference is made to the findings of evaluations and the follow-up that ought to be given. ICCO promotes learning in the organisation. The funding cycle is the core of the learning process, with information collected from the progress reported by partner organisations, evaluations and visits to partner organisations directly related to the utilisation of the information about the partner organisation and its context for planning, supervision and decision making with regard to the partner organisation. Broader learning of thematic lessons takes place by means of thematic partner consultations and training.

116

An internal assessment of the strengths and weaknesses of the ICCO PME system 77 concludes that the internal PME system has been successful to relate funding to agreements and to store and retrieve thematic and geographically organised information. The main problem faced is that aggregation of data is either not possible or meaningless.

6.4 Novib
6.4.1 Background Novib has 3 departments: Campaigns, Lobby/ Advocacy and Projects and three Novib-wide Project Groups: Trade Campaign, Corporate Social Responsibility and Humanitarian.. The Projects Department has 9 regional units and a unit financial services. The Management Team Projects (MTP) is composed of the heads of the units. In the units, the primary process (from identification to evaluation) is executed by teams, composed of a secretary, responsible for the administrative aspects of the primary process, a programme officer, responsible for the policy aspects and a financial staff member. The entire team is responsible for the primary process. The final responsibility lies with the bureau head. Supporting units within Novib are Research and Development, Communication and Quality and Control (Q&C). The unit Q&C is responsible for the policy and decision making procedures, including policy evaluation. Novib has 332 staff (including part time staff). In 1995 Novib joined the Oxfam International alliance, in order to work more efficiently and effectively for poverty alleviation. Novibs monitoring system Novib distinguishes three levels of monitoring: Monitoring of the (internal) policy cycle. This is based on the agreements between the heads of the units and the Directors. In these contracts objectives at different levels are specified: beneficiaries, portfolio management, capacity building, input and process. The unit Q&C prepares monthly overviews (mainly concerning financial indicators), four-monthly overviews (reflecting the progress of activities of a unit or department), annual reports (mainly concerning outcomes) and combined 4-monthly / annual reports. These reports are discussed in the units. The MTP and the Director Projects are the main users of these reports. Monitoring of progress of the 5 rights. For the Strategic 4 Year Plan, the 5 rights were divided in funding percentages. Each regional department is monitoring if the total of programmes of the partner organisations corresponds with the allocations per right. The aim of the monitoring is to make the necessary adjustments in the partner portfolio when required. Monitoring of the project- and counterpart cycle. The aim of monitoring activities and partner organisations is to assess whether partner organisations are fulfilling their agreed obligations and whether unexpected circumstances may require action from the side of Novib78. 6.4.2 Monitoring of partner organisations

77 78

The pressure to measure: Planning, monitoring and evaluation at ICCO 05-09-2003 Handboek Kern Processen: Monitoren Projecten

117

In 2001-2003 Novib developed the Toolbox. This is fully in use since late 2003-early 2004. It is used for exploring the possibilities of a relationship, for the appraisal and approval of a grant and constitutes the basis for the follow-up and review of progress. The Toolbox appraisal is a process to support programme staff of Novib in generating a set of judgements about the potential success of a grant. The Toolbox is essentially based on an assessment of the opportunities and risks associated with funding partner organisations. The expected outcomes as well as the risks, defined by the Toolbox, constitute the basis for the definition of milestones. These milestones, which are incorporated in the contract between Novib and its partner, are the basis for the monitoring and review of partner organisations by programme staff. 6.4.2.1 Reporting by partner organisations The agreed reporting by the partner organisation is an important element of monitoring. Partner organisations report to Novib annually, both a narrative and a financial report. In the revised reporting guidelines, approved in 2004, more explicit attention is given to the results a and the effects that are achieved and to the risks, as identified by the Toolbox. 6.4.2.2 Other monitoring Novib staff regularly visits partner organisations (on average once a year). Of all visits trip reports are made and submitted to the Director Projects and the Documentation and Information Centrum. Other instruments used for monitoring include regular consultations with selected partner organisations, partner platforms and different types of meetings. Partner organisations receive the publication Novib network. 6.4.3 Evaluation

An evaluation of the practice of evaluations in 2001 has resulted in the following action to strengthen evaluation in Novib79: Minimum quality requirements for evaluations. This is done in the context of the MBN programme evaluations in 2002 and these requirements are used by Novib as of 2003. Specific guidelines for activity-level evaluations and assessment of activity-level evaluation reports have been developed in 2004. Focus evaluations more on results, less on process and organisational aspects. Development of a coherent evaluation agenda, to maximise the utility and effect of evaluations. 6.4.3.1 Activity-level evaluations Up till 2005, activity-level evaluations were joint exercises, that play an important role in the relationship between Novib and the partner organisation. Learning was the primary aim of activity-level evaluations. The intention to execute an evaluation is formulated in the contract between Novib and the partner organisation. It is preferred that the partner organisation takes the lead in formulating the ToR and selecting the evaluators, and discusses this with Novib. Under the rules of its new Evaluation Policy, as from 2005, the principal responsibility for activity-level evaluations rests with the partner organisations. One of the main reasons for Novib for shifting the responsibility is ownership. The aim is that the evaluations will be more result-focussed, contain more information from the field and will be less focussed on
79

Jaarplan 2003, p 50.

118

organisational and process aspects. The scope of the evaluation, however, is clear: the expected and unexpected outcomes and risks as identified in the Toolbox as well as the contribution of outcomes to policy and practice changes. To support the partner organisations, Novib has developed tools such as guidelines for terms of reference and criteria for consultants. Programme officers are required to assess the quality of each evaluation report. Guidelines have been developed to support this assessment. Activities that receive over 400.000 Euro in 3 years or over 250.000 Euro in 1 year, have to be evaluated. Evaluation of smaller activities is not mandatory, but smaller activities may be subject to country or regional programme evaluation conducted by Novib. All partner organisations (also the ones with smaller activities) can receive financial resources and technical assistance if they wish to conduct an evaluation. Novib aims to execute approximately 100 activity-level evaluations annually. Table 6.4: Number of activity-level evaluations in 2002 and 2003
Year 2002 2003 Activity-level evaluations 123 105 Total number of partner organisations 861 839

Source: Novib annual reports 2002 and 2003

6.4.3.2 Programme evaluations Novib occasionally executes programme evaluations on a thematic or regional issue. It intends to do this in a more structured way, using a multi-annual evaluation agenda and reorienting its own evaluation efforts towards country and regional programme evaluations, which are considered to be more directly relevant to feed into Novib strategies and policies. In 2005 two pilot core country evaluations will be implemented. Novib participates in the MBN programme evaluations that will be executed in the period 2003-2006 and also participates in OI evaluations 6.4.4 Use of feedback information

Use of feedback information from monitoring Novib has a procedure to process progress reports. Progress reports are appraised by the country teams concerned. The aim of the progress report is twofold: on-going assessment of results, to assess the need for adjusting operational plans and accountability for the funds approved in the activity budget. From January 2004 Novib has introduced a Performance Registration System. This system documents the direct results of the POs, and also the outcomes and medium to long term effects. The system includes a rating system using a 4 point scale. Linked to the Performance Registration System, are the Toolbox and the revised guidelines for progress reports by partner organisations. By the use of these instruments, Novib intends to react in a more structured manner on the specific goals and milestones of an activity and/ or partner organisation. By using the system, Novib can indicate the amount of activities that has been successful. Use of feedback information from evaluation Activity-level evaluations are discussed with the partner organisations concerned and its results are taken into account when assessing the counterparts performance (performance registration system) and when doing new appraisals (toolbox). As from 2005 the full version of the evaluation report is made available in the automated system for internal use. Yearly Q

119

&C, together with R&D, makes an analysis of all evaluations, to define trends and lessons. These can lead to policy adjustments. The results of programme evaluations are discussed in the regional units concerned and with the POs involved. The management of the Projects Department, the project advisory committee and other Oxfams are informed about the outcome of programme evaluations. Issues emerging from programme evaluations may be incorporated in activities or policies. Novib aims to enhance learning in the organisation. Through reports, evaluations, audits, inspections, conferences and meetings it receives an enormous amount of information. This information is used to define good (and bad) practice. Novib is still working on approaches to integrate this learning process in its internal policy cycle. In 2004 a pilot to strengthen learning was started80. On country/regional programme level Novib is developing a strategic programme management instrument, based on opportunities and risks identified at country or regional programme level, the partner portfolio can be adjusted past achievements are an important source in identifying these opportunities and risks.

6.5 Plan NL
6.5.1 Background Plan Netherlands (Plan NL) is part of the International Federation of Plan organisations. Plan NL is managed by a National Director. In the directorate Government and Programmes the unit Projects and Programmes is responsible for the execution of activities and for all activities funded by the co-financing programme. At its headquarters Plan NL employs approximately 95 staff (85 fte). Plan NL also employs approximately 200 volunteers. The directorate Government and Programmes has a staff of 29. The mode of implementation of interventions by Plan differs from the other CFOs. Plan NL only implements activities through the Plan country offices, who in most cases execute activities directly with the stakeholders (often local communities directly or community based organisations (CBOs)). The Plan country offices are the direct partners of Plan NL. In some countries, however, this modality is not possible for legal reasons (for example India), where interventions are implemented through local NGOs. In all programmes the Plan country offices are responsible for the interventions that are implemented with the support of Plan NL. The responsibility of Plan NL is limited to the appraisal and supervision of the relationship between the Plan country offices and the implementing organisation. Overall, Plan has two funding modalities, the sponsorship mode and the grant mode. The sponsorship mode applies to the funds obtained from the public and private companies intended to directly support children in developing countries by executing community based activities. The grant mode applies to activities funded with public funds, such as the support from the co-financing programme. M&E system of Plan International The International Federation of Plan organisations has an elaborate Corporate system of Planning, Monitoring and Evaluation, CPME. The CPME comprises procedures, instruments and software for different phases in the project cycle: baseline surveys, planning, monitoring and evaluation. The CPME comprises an elaborate set of indicators for planning, monitoring and evaluation. These indicators are mainly quantitative and focus on the traditional fields of interest for Plan, such as education and health. Indicators for more recent priorities such as civil society building, lobbying and advocacy and rights issues are less well developed. The
80

Performance Management in Strategic Development Funding, June 2004.

120

CPME system has been developed primarily for activities implemented under the sponsorship mode, but is also used for the activities implemented under the grant mode. It is continuously under development, and, with strong support from Plan NL, discussions are on-going to adapt it to the new realities.

121

M&E system of Plan NL Over the years Plan NL has developed some specific instruments for M&E. These are now being harmonised with the corporate CPME system. Plan NL emphasises the monitoring of the performance and development of partner organisations. Plan NL has identified several pilot national Plan offices, which are stimulated to reflect on the possibilities of collecting and producing information in a more structured manner. In the annual plan 2004, Plan NL presents the hierarchy of instruments that is under development. This hierarchy distinguishes 4 levels (Plan NL, Plan country office, partner organisation and community organisation) and different types of results (a.o. output, outcome, lobby partner ships, civil society strengthening and policy). 6.5.2 Monitoring of activities and programmes

6.5.2.1 Reporting The interventions supported by Plan NL are implemented under the full responsibility of the Plan country offices. The operational staff and the organisations involved report directly and exclusively to the Plan country offices. For the CFP funded activities the following reporting procedures generally apply: Quarterly reports: overview of expenditures per output code and per geographical area. Annual narrative and financial report Mid-term review Evaluation towards the end of the project period. The monitoring focuses on the following levels: The hierarchy of objectives of the activity concerned The objectives on poverty alleviation of Plan NL (including the regionalised objectives). The relationship with the partner organisations. Plan NL monitors the activities, to a large extent on the basis of the information included in the CPME. The Plan country offices report annually to Plan NL. For the annual report Plan NL has developed a specific format. As Plan country offices implement activities supported by different Plan organisations in donor countries, this causes substantial duplication of work. Recently the formats have been standardised. It was reported that this resulted in enhanced quality of the information included in the annual reports. 6.5.2.2 Other monitoring In addition to the reporting procedure, Plan monitors its interventions through different studies and reports, including regular peer reviews, workshop reports and minutes / reports of special meetings. As the Plan country offices execute most of their activities directly with the stakeholders, contact with stakeholders is on regular basis. During the year organisations are regularly visited by staff of the Plan country office. In addition staff, of the regional units of Plan NL visit the Plan country offices and the implementing organisations approximately once a year.

122

6.5.3

Evaluation

6.5.3.1 Activity-level evaluations As a rule all activities are evaluated mid-term and towards the end of their existence. The criteria for the evaluation are defined in the CPME. Mid-term evaluations focus on intermediate results, end of activity-level evaluations focus on outcome and purpose 81. The mid-term evaluations are initiated and executed by the Plan country offices. For the end of the activity-level evaluations the terms of reference and external consultant have to be approved by Plan NL. To enhance the quality of the activity-level evaluations, Plan NL has developed a standard terms of reference. The Plan country offices can use that standard as a guideline. Furthermore, Plan NL has developed an assessment system for external consultants. Table 6.5: Number of activity-level evaluations in 2002 and 2003
Year 2002 2003
Source:

Activity-level evaluations unknown 22

Total number of partner organisations 101 programmes with 44 national and regional Plan organisations. 183 programmes with 44 national and regional Plan organisations.

Plan NL intends to use the results of the evaluations for result measurement. The amount of information on outcome-level defines to great extent the utility for result measurement. Given the width of the strategy matrix of Plan NL and the amount of themes that will be reported on, Plan NL expects it to be necessary to obtain a minimum of 10-15 outcome results. In 2004, Plan NL has made an internal assessment of the quality of the activity-level evaluations executed in 2003. It was noticed that the amount of information on outcome-level was too limited, usually based on a
few observations and not on a clear analysis. Also it was noticed that the quality of the ToR were not always in compliance with the guidelines and that Plan NL was not always enough involved in the process of drafting the ToR. In the internal study of Plan NL it is recommended that it should always reserve approval of the ToR, that Plan NL needs to be more involved in the decision making process of the evaluators and that the final responsibility of the evaluations remains with Plan NL. Another instrument to enhance the quality of evaluations is the consultant data base. Plan NL aims to enhance the quality of the ToR to focus the evaluators to search more systematically for outcome-information. 6.5.3.2 Programme evaluations Plan NL conducts different types of programme evaluations. These include evaluations executed under the direct responsibility of Plan NL, participation of Plan NL in the series of MBN programme evaluations, impact studies initiated by Plan NL, but executed in cooperation with the Research and Evaluation Dissemination Unit of Plan International, and audits conducted by independent audit organisations. The purpose of an audit is to evaluate all the programmes within one country (not only financially, but also concerning content). Plan also intends to execute impact studies. A first study has been conducted in Bolivia in 2003. This study describes the change of sixteen impact related indicators over a period of 5 years. Plan NL is reviewing the experience to draw lessons for the future.

81

Update MFP Werkplan 2003, p. 25.

123

6.5.4

Use of feedback information

Use of feedback information from monitoring The culture of Plan NL is focused on accountability, both with regard to the financial aspects as well as with regard to the outputs that are realised. The emphasis is very much on quantitative information. Internally, Plan NL monitors the performance of the unit Projects and Programmes. This includes the achievement of its financial targets (disbursements) as well as the achievements in terms of staff, development of knowledge and expertise. The annual reports of the Plan country offices, or those from the implementing organisations are forwarded to the staff of the regional units of Plan NL. They are used as a basis for the regional annual reports drafted by the units. Use of feedback information from evaluations The findings of evaluations of individual interventions are discussed directly between the staff of the regional unit of Plan NL office and the operational staff of the Plan country office and the staff of the implementing organisation. In some cases the findings are also discussed on regional and international Plan level. The evaluations are used as input for policy development for the entire Plan organisation. Furthermore, as from 2003, Plan NL uses the findings of the evaluations (especially the outcome information) for result measurement. In the 2003 annual report Plan NL has made a first effort to construct a result focused report, a first attempt to establish a system of result measurement at the level of the target group and communities. The outcomes of the impact studies will be used for internal learning and policy development.

6.6 Terre de Hommes

6.6.1 Background Terre des Hommes (TdH) is a relatively small organisation, with a flat organisational structure. It has three operational units: Marketing Communication, Projects and Campaigns and Finance and Administration. The head of the three units together with the Executive Director constitute the Management Team. The 4 regional offices (East Africa, South Asia, Southeast Asia and Latin America) are attached to the unit Projects and Campaigns. TdH headquarters has 29 staff, the 4 regional offices together have approximately 45 staff. The regional representatives are responsible for the support and supervision of the partner organisations (in TdH known as project partners). The philosophy is that support to partner organisations need continuous presence of TdH representatives in the field. TdH M&E system TdH has become part of the CFP only as of January 2003. This has consequences for quality management and the organisation of implementation and supervision. Procedures for these aspects are still under development. With regard to planning, monitoring and evaluation, TdH has had a diverse practice in its regional offices. In 2003 TdH introduced the Objective Oriented Project Planning (OOPP) approach to plan its activities. Partner organisations were introduced to this system through workshops. For the annual regional plans 2004 TdH makes use of logical frameworks. These logical frameworks include information about objectives, themes, envisaged results and related indicators. In the annual plans 2004 the logical frameworks of the 4 regions differ

124

substantially. In 2004, staff of TdH, including the regional representatives, have made an effort to familiarise themselves with the methods to make the logical frameworks of the regional offices consistent. Each of the regional representations has its own approach to supervise their partner organisations. At the moment discussions take place if the approach developed in the South Asia office, where intensive contact is maintained with partner organisations, should become the corporate style. TdH is currently developing a single M&E system for the entire organisation, focussing more on results (output) than on input. The use of the OOPP approach and the logical framework for planning of activities and drafting its regional plans provides a basis for enhanced M&E. 6.6.2 Monitoring

In TdH a distinction is made between the monitoring of the regional offices by TdH headquarters, and the monitoring of partner organisations, mainly by the regional offices and to a limited extent by headquarters. Monitoring of regional offices Headquarters monitors the performance and management of the regional offices. The regional offices regularly report to headquarters: Reports of field trips by staff of the regional office Regular (monthly) extended discussions by telephone between headquarters and regional offices Quarterly progress and financial reports Annual progress and financial reports Annual meeting headquarter staff with regional representatives. Regional representatives must report to headquarters annually, describing (among others) the progress of the activities, general developments in the countries concerned, including the consequences of these developments for the TdH supported activities. Regional representatives meet at headquarters at least once a year, among others to exchange experiences. Every regional office receives an operational audit team from headquarters once every two years. The aim of this audit is to assess the administrative arrangements and management of the portfolio by the regional office as well as the efficiency of the use of funds. A secondary objective of these operational audits is to verify the achievement of TdH objectives by the portfolio of activities82. Monitoring of the partner organisations The key purpose of the monitoring of the partner organisations is described as follows83: Verify that the implementing team is aware of the objectives of the activity Assess whether the activity is being implemented as planned, and expenditures are done according to plan. Assess if the activities contribute to achieving the objectives Assess the effectiveness of the activity and the quality of administrative systems. Support partners with reporting and develop the capacity of partner in different fields. Reporting, internal evaluation and regular visits and other contacts are the key monitoring instruments.
82 83

Management plan 2003-2006, p.20. Management plan 2003-2006, p. 20.

125

6.6.2.1 Reporting In the contract between TdH and its partner organisations the following reporting requirements are stipulated: Project Monitoring Reports: 6 monthly reports by the partner organisation, providing an overview of implementation (related to planned activities and objectives) and a financial report indicating expenditures related to the budget. Audit Report at the end of the financial year (if expenditures exceed 35,000 or if required by law), indicating actual and planned expenditures. This report should be compiled by an external certified accountant. Annual Internal Evaluation Report analysing the performance of the activity during the last twelve months to improve the annual plan for the subsequent year. The Internal Evaluation Report is only included in the contracts of India. Newsletter to inform sponsors about the activities and developments of the activity and the target community. Annually internal self-evaluations are done as part of the regular monitoring of the partner organisations by the regional office. The reports are sent to the regional office and analysed by the relevant staff. If need be, comments are sent to the partner organisation or issues are discussed during the regular field visits. 6.6.2.2 Other monitoring

Partner organisations are supported and supervised by the regional offices. As a rule, each activity must be visited twice yearly by staff of the regional office. The regional representative should visit each activity at least once every two years. During these field visits discussions are held with staff, management and board members of the partner organisations, and contact is made with the beneficiaries. Field locations are visited. Also the financial and administrative systems are reviewed and the legal status of the organisation verified. Attention is given to the quality of the reporting and if need be support is provided. Staff should draft a visit report of every visit. This report is submitted to the regional representative and copied to headquarters. Because of the relative proximity of TdH regional offices to their partners, there are regularly contacts between partner organisations and TdH staff at the regional offices. These discussions mainly concern progress and emerging problems, as well as exchanging views and information. Regional offices regularly organise workshops for staff of partner organisations about various subjects, either thematic or of a more managerial nature. Monitoring and evaluation is sometimes the subject of such workshops. TdH is in a process of clustering its activities, both geographically as well as with regard to subject matter. This will facilitate exchange of information and experiences between partner organisations, thus allowing them to learn from each other. All staff of the unit together make about 5-6 field visits per year. During each of these field trips a number of partner organisations is visited. Field trip reports are made and submitted to the regional offices as well as headquarters.

126

6.6.3

Evaluation

6.6.3.1 Activity-level evaluations The regional offices are responsible for the planning and execution of evaluations. As a rule, all activities are evaluated once per contract cycle of three years (in principle in the 30 th month of the contract period). Evaluations are normally done jointly, with a staff member of the regional office, staff of the partner organisation and in most occasions an external facilitator who bears the final responsibility for the process and the evaluation report. Activities that are directly executed by TdH are always evaluated by external evaluators. The joint evaluations describe and assess the development of the activity and the partner organisations over the contract period. The aim is to formulate lessons learned that can be incorporated in the next phase of support by TdH. The joint evaluations are not used for accountability reasons. Table 6.6: Number of activity-level evaluations in 2002 and 2003
Year 2002 2003 Activity-level evaluations n.a. 46 Total number of partner organisations 252 255

Source: Stichting TdH Nederland Jaarverslag 2003 Medefinancieringsprogramma

6.6.3.2 Programme evaluations At of the cut-off point of the study, TdH did not execute programme evaluations. As it is not a member of MBN, it also does not participate in the programme evaluations initiated by MBN. However, TdH will start executing programme evaluations as from 2005. The programme evaluation will be carried out from an accountability perspective. 6.6.4 Use of feedback information

Use of feedback information from monitoring All monitoring information is used directly for decision making by the regional offices, or sometimes TdH headquarters. A structured use is made of all progress information (progress reporting, annual evaluations and audits) during the annual management review. Use of feedback evaluation from evaluation The conclusions of the joint evaluations executed 6 months before the termination of the project period are an important input for the formulation of the subsequent phase. The lessons learned described in the evaluation are an important element for the appraisal of the next phase of the activity. The follow-up of evaluation conclusions and recommendations is also assured through the regular visits of staff of the regional offices.

127

6.7 Assessment of Progress Reports of partner organisations

Information about Format

Assessment Format of Progress Reports of partner organisations Characteristics 2 Description

Consistence of format Index given Length of report Annexes given Use of language Presentation of context Comparison of the content of the report with information requirements of CFO. Yes/No Length of report according to requirements Yes/No Accessible linguistic usage Information given on local, regional and national social, political and economical situation of PO Significant changes in context Information given on social, political., economic situation of target group Significant changes Information on changes in organisational structure of activities Information on changes of staff, changes in management Information on policy development/ policy changes Organisational development Information on important developments in external relations/ contacts with stakeholders. Description of implementation and progress of the activities Reference given to milestones of plan/ contract Clear description of any delays of activities Attention to any preconditions in plan/ contract Deviations of plan/ contract explained Reasons given for deviations Constraints for the implementation/ progress of activities given. Remedial action taken/ suggested (implementing activities in future in order to stick to original plan or adjusting of plans/ activities) Clear description if direct output is or will be realised. Reference given to plan/ contract Use of previously agreed indicators Clear description if effects are or will be realised Reference given to plan/ contract Use of previously agreed indicators Reflection/ abstraction given of previous year Reference given to evaluations

Context

Presentation of target group Organisation Presentation of organisation

Progress

External relations, contact stakeholders Description of progress Relating actual progress to plan or contract Explanation / discussion of deviations Constraints/ remedial action taken

Results

Realisation of output Relating output to plan or contract Results on target group level Lessons learned

Conclusion

128

Scores: 0 - Cannot be assessed 1 - No information given or very brief; information clearly insufficient to get insight in the aspect concerned 2 - Summary account; gives an idea of aspect concerned, but picture is incomplete 3 - Fairly good information; clear picture of aspect concerned 4 - Very clear/ insightful information (can be concise); leaves little questions A score of 3 or 4 means a satisfactory presentation of the aspect concerned in the progress report.

Assessment of main characteristics of 58 progress reports

Characteristic 1 Unsatisfactory 2 9% 31 % 36 % 26 % 22 % 9% 33 % 35 % 28 % 31 % 40 % 14 % 29 % 50 % 47 % 28 % 9% 22 % 43 % 26 % 3 52 % 47 % 10 % 10 % 29 % 64 % 40 % 12 % 14 % 31 % Satisfactory 4

Format
Consistence of format 0% 9% 24 % 14 % 2% 0% 19 % 31 % 15 % 12 %

Contents
Presentation of context Presentation of target group Presentation of organisation External relations, contact stakeholders Description of progress of activities Relating actual progress to plan or contract Explanation / discussion of deviations Constraints/ remedial action taken Lessons learned

Results
At level PO Realisation of output Relating output to plan or contract At level target group Results at level target group 0% 26 % 26 % 0% 22 % 48 % 28 % 19 % 14 % 72 % 33 % 12 %

129

Titles of 58 progress reports assessed Progress reports India

Partner organisation Butterflies Cecoedecon CHC Name of progress report Annual Report 2002-2003 Programme of Street and working Children Cecoedecon Project Progress Report: Participatory Initiatives for Integrated Rural Development 2003-2006 ( April- September 2003) Annual report of the CHC functional unit of the society for community health awareness, research & action (for the year 2002march 2003) CWS Annual report 2002-2003: Towards solidarity 3 (Novib) CWS Annual report 2002-2003: Sustainable Agriculture (Hivos) Fedina Navachetana Annual report on activities (January December 2002). Fodra Annual Progress Report April 2002- March 2003 : Micro credit and Women empowerment HCHW 2002 Annual report 2003: Quality service with utmost dignity & care No progress report available at time of assessment Project Progress Report (July 2002-June 2003) Navdanya Trust Annual narrative report for the year 2002-2003, Globalisation, Sustainability and Democracy Programme. Grant Report to Plan NLNO (January December 2003) IND0044Jogini Sisters Activity monitoring reports available could not be assessed Activity monitoring reports available could not be assessed Rupcha Annual Report 2002-2003 SACCS/ BBA Narrative report (April 2002-March 2003) No progress report available at time of assessment No progress report available at time of assessment Annual report 2002-2003 SIFFS Progress Report for the year 2002-2003 SRUTI Annual report (April 2002- March 2003) Vishaka Project Report (June 2002 - March 2003) Empowerment of Women through Health Education. CFO ICCO ICCO Cordaid

CWS FEDINA FODRA HCHW IDEI MYRADA Navdanya Trust Plan Jaipur Q.W.R.I.D.S. RAID RUPCHA SACCS/AVA Samata/ MMP SAMSKAR Seva Mandir SIFFS SRUTI VISHAKA

Hivos, Novib Cordaid Cordaid Novib ICCO Novib Hivos Plan TdH TdH Cordaid Novib ICCO Plan ICCO, Plan Hivos Novib Hivos

Progress reports Peru Partner Name of progress report organisation ARARIWA Informe Narrativo periodo enero 2002-diciembre de 2002: seguridad alimentaria en cuatro distritos del valle sagrado de los incas. (Cordaid) CADEP No progress report available at time of assessment CCAIJO Informe abril 2003-octubre 2003 Lucha contra la pobreza en la microcuenca de ccatcamyao Cecycap Informe Programatico 2002-2003 CEDEP Informe anual de actividades ao 2003. Cedep Ayllu Segundo informe 2002 Hivos-Cordaid UE (Hivos and Cordaid) CEDESI Informe del Proyecto Centro Piloto de atencin para madres adolescentes primigestas Julio-Diciembre del 2003 CEDIA Cedia Informe enero diciembre 2002: proyecto: desarrollo sostenible comunitario y manejo de recursos con pueblos indgenas y campesinos de la cuenca del rio chambira. (Hivos) Cedia Informe Enero- Diciembre (Novib) CEDISA Proyecto apoyo al desarrollo regional 2002-2005 informe semestral (enero junio 2003)

CFO Cordaid, ICCO ICCO Cordaid Cordaid Novib Cordaid, Hivos TdH Hivos, Novib

Hivos

130

CIED Puno El Taller Guaman Poma IAA IDEAS IDESI Nacional

INCA

INPPARES KALLPA

PLAN Peru

RAAA

Informe Narrativo periodo 2002 Evaluacin 4to. Periodo marzo 2003 agosto 2003 Evaluacin anual 2003: Programa Salud Preventiva: Fortalecimiento de la seguridad alimentaria en 5 distritos del Cuzco No progress report available at time of assessment Informe narrativo enero- junio 2003, Proyecto PE025131 Incidencia Poltica. 2005 Informe proyecto punte, julio 2003-diciembre 2003 (Novib) Informe 2003, Avance de la Contraparte y de los proyectos Informe Narrativo : proyecto: amigos de la calle (enero - julio 2003) Informe Tecnico anual del proyecto EPS- rural (Cordaid) Informe Evaluativo 2003: Proyecto: Escuelas promotoras de la Salud (ICCO) Informe Narrativo enero junio 2003 Trabajo sin riesgo (TdH) Annual Report Jan.-Dec. 2003 Healthy and Happy Children per0103 Annual Report Jan.-Dec. 2003 Micro Finance for Women with Education per 0302 RAAA informe anual 2002

ICCO Cordaid Cordaid ICCO ICCO Novib, ICCO Novib TdH Cordaid, ICCO, TdH

Plan

Hivos

Progress reports Tanzania Partner Name of progress report organisation ADP Mbozi ADP Mbozi Annual Report 2003 CORDS Cords Annual Report 2002 CRS Progress Report on the Joint Cordaid-CRS Programme in Lake Zone, Tanzania, (October 2001- September 2002). CSSC CSSC Annual Progress Report (January-December 2002): Institutional Capacity Building Project (ICCO) Finca Finca is partner of Hivos Triodos, no progress report available HakiElimu HakiElimu Annual Report 2002 IDRO No progress report available at time of assessment Ilaramatak Ilaramatak Lorkonerei: Annual Progress Report (January December Lorkonerei 2002) IRDT IRDTF Annual Progress Report 2002 Kinnapa Kinnapa bi-annual progress report January - June 2003 Kivulini Kivulini 2002 Annual Report: Mobilising Communities to Prevent Domestic Violence. Kuleana Kuleana Progress Report, Narrative report 2002 (Hivos) LHRC LHRC Annual Report 2002: Building a Human Rights Culture MRHP MRHP Annual Progress Report 2002 Nafasi Nafasi TZ 030 Progress Report January to June 2003 Nafasi TZ 030 Progress Report July to December 2003 Plan Tanzania Plan Tanzania Community Based Health Care & Child Survival, annual report Jan-dec. 2003 Plan Tanzania Vocational Training Project Annual report. Jan-dec 2003 SAIPRO SAIPRO Annual Progress Report (January December 2002) Physical and financial. SHDEPHA + SHDEPHA + Interim Narrative and financial report for a period January 2003- December 2003 in the contract signed between Hivos and SHDEPHA+ (not yet approved by Hivos) TDFT TDFT Report 2002: Poverty as a Development Challenge TGNP TGNP Annual report 2002 (Cordaid and Hivos)

CFO Cordaid Cordaid Cordaid ICCO, Cordaid Hivos Novib Cordaid Novib Cordaid Novib Hivos Hivos, ICCO Novib ICCO TdH Plan

Novib Hivos

ICCO Cordaid, Hivos

131

6.8: Reporting guidelines of the CFOs

Reporting guidelines of Cordaid
Cordaid states in the guidelines that it is important that the lay-out and information in the reporting relates to the original project application. The scope of the reporting should be proportional to the (financial) scope of the project. In general concise reports are to be recommended. Narrative reporting guidelines Reporting format: Context A short up-to-date report should be given on the changes in the context during project implementation. - position/policy of the government, presence in project area, relationship with partner - area and sector: relevant developments in this respect which fall out-side the project intervention - assumptions: possible limiting factors which do not lie within the projects sphere of influence. Objectives (for period covered by report) - report on the interim position achieved in the period covered by the report in relation to the objectives (intermediate goals established in the outset) and final goals. - any changes agreed with Cordaid in the intervening period. Activities and output - Activities carried out (as compared to the activities planned) - Any changes agreed with Cordaid - Comparison of the planned and realised activities and output (summarised in overview) Organisation - New and relevant organisational aspects e.g. development of and changes in the organisation, co-operation, staff changes, management aspect, changes in legal status. Assessment and Forecast - assessment by the organisation responsible (degree to which the objectives have been achieved, achievement figures) - forecast in general terms and financial projection. (Financial reporting)

Reporting guidelines of Hivos

Narrative reporting guidelines The report should be consistent with the proposal referred to in the contract letter, with the relevant work plan as well as the audited annual account. The report should consist of 4 parts: Part a: Context of the organisation (max 2 pages) - Developments in the socio-economic, political and ecological context that were important for the organisation. - Summary of the enabling factors (opportunities) and the constraining factors (threats). Part b: Internal Organisation (max 5 pages) General - Changes in mission and long-term objectives - Major policy decisions Structure - Changes in the organisations formal structure, functions, governing structure and main decision making processes. - Description of the structures and processes during the reporting period. E.g. frequency of boarding meetings.

132

Planning, monitoring and evaluation - The internal planning process function during the reporting period (who was involved, when and in what why) - Description of the way of monitoring the financial and general performance of the programme and use of information. If applicable, use of MIS - Production of reports Human resources - Changes in staff - Training, internships Financial situation - Overview of income sources - Major investments Part c: Programme(s)/activities (10 pages) - Summary of all activities implemented (and explanation of deviation from proposed activities) - Output (foreseen and unforeseen) of each activity. (specify for both men and women) - Use of output by beneficiaries (i.e. effect). Comparison of actual effect with short-term objective. Explanation of difference. (specify for both men and women. - Lesson learnt, follow-up - Summary of internal and external evaluation (if any) The result analysis model can be used that is enclosed in the contract. Part d: Relationships with different stakeholders (max 3 pages) - changes in stakeholders - participation of stakeholders - strategic alliances - relationship with local/ national authorities.

Reporting requirements ICCO

The narrative and financial report should include: - the progress of the expected results, the conditions, the dialogue issues and the monitoring issues, in accordance with the Specific Contract Conditions. In the specific contract conditions the expect results are given, conditions, monitoring issues and points of dialogue are given. - The role of ICCO in relation with the partner. What did the partner think of the relation and the communication with ICCO? At what point, could ICCO support more or differently. Guidelines for Narrative Progress Report The narrative progress report needs to give (concisely) insight in: - Analysis of the execution of the plan van aanpak (project proposal?), including an explanation in case of deviations. - Achievement of the expected results: analysis of the effect of the executed activities, including an explanation in case of deviations - New opportunities or threats in the external context, including a distinction between context the partner can and cannot influence (what are the consequences of this on the achievement of the expected results? Has something changed in the relationship between other actors, through which the strategic role of the organisation within the community has changed). - New developments in the organisation - Any changes in the plan of aanpak, as a result of issues mentioned above. - Gender specificity of the analysis, the results, the execution and the developments.

133

Reporting guidelines Novib

The purpose of the guidelines for progress reports by partner organisations are: Interim assessment of results achieved to allow (if necessary) adjustment of work plans Assure accountability of the use of funds. The following are the key elements of the progress reports from the point of view of Novib: Assessment of progress requires that reference is made to the original plans. Implementation should be linked to funding Reports must indicate the relation of activities to the five rights that Novib pursues. Reports should be concise and to the point. Distinction should be made between progress at the level of the partner organisation and at the level of a project. The narrative progress report (projects or organisations) should present the following: General information: partner organisation, project, reporting period and date. Progress of implementation of plans. Progress of activities, outputs and outcomes Comments on the results achieved Factors that positively or negatively influenced progress Financial information Progress on conditions and agreements Lessons learned Comments on support by Novib Conclusions

Reporting format Plan NL

The format consist of 3 sections: Section 1: basic information Block 1.1: project context Block 1.2: Activities, participation, progress towards objectives Block 1.3: Monitoring and Evaluation Block1.4: Project continuation Section 2: analytical information Block 2.1: Capacity building partnerships Block 2.2 Sustainability Block 2.3 Environmental sustainability Block 2.4: Gender equity Block 2.5: Child Centeredness Block 2.6: Lesson learning and dissemination Section 3: additional information In the guidelines each of the sections is elaborated in detail.

Reporting guidelines TdH

Project Monitoring Report and Project Financial Report - The Project Monitoring Report should be a detailed report on the implementation of the activities based on the project proposal. It should incorporate the financial report, stating in detail the actual expenditure and balances corresponding to the budget of the project proposal. The above mentioned paragraph is taken from the TdH contract format, and is a minimum requirement. Every regional office uses a more explicit format and/ or guidelines.

134

135

136

Appendix 7 DSI/MY

137

Appendix 7: DSI/MY
7.1 Format annual plan / annual narrative report
1. Introduction Background Relation between annual plan and budget / vision on poverty alleviation Stakeholders in formulating annual plan Lessons learned and adaptations compared to previous year 2. Contribution to sustainable poverty alleviation and policy choices Policy development Choices made (objectives, themes, regions, partners, activities etc) Operationalisation of choices made Added value 3. Objectives Objectives and relationship between short term and long term objectives 4. Results to be achieved Results to be achieved and relationship with objectives 5. Partner organisations Choices made Role in policy development Role in knowledge exchange, networks and donor co-ordination Role in implementation Role in monitoring and evaluation 6. Regions and countries 7. Activities Type, importance, phasing and mutual relationships Relationship with envisaged results and objectives 8. Organisational structure 9. Use of resources and financial management 10. Support in the Netherlands 11. Monitoring and evaluation system Lessons learned and follow-up Background on monitoring and evaluation plan Evaluations planned 12. Budget Expected contributions target groups and other partners Background and assumptions about contributions third parties Budget and background revenues and expenditures Subsidy requested

138

7.2 Overview of observations and information requests

Annual report 2001 Cordaid Information requested about tentative experiences with involvement in PRSP processes Financial issues Observations form embassies: generally positive, one embassy observes that the sectors Chosen by Cordaid do not correspond to the priorities of the bilateral programme, and no information about partners is given. [no letter found in DSI archive]

ICCO HIVOS

Information requested about tentative experiences with involvement in PRSP processes additional information requested about stated improvements in appraisal and monitoring of POs Additional financial issues raised Observations from embassies: overall positive, two embassies raise issue of limited contact, one states that most POs are intermediary organisations, few CBOs No observations from thematic directorates NOVIB Request for information about first experiences with involvement of POs in PRSP processes. Observations by embassies: remarks are of general nature, mostly positive. Embassy Rwanda comments on the fact that Novib pays only limited attention to capacity building of its POs. Plan Information requested about tentative experiences with involvement in PRSP processes Observations form embassies: generally positive; one embassy raises issue of co-ordination with national policy on primary education. Annual report 2002 Cordaid Report is very broad. Information at a higher level of aggregation is needed by DSI Not only successes, but also failures. Experiences with PRSP. What are consequences of the shift from project focus to organisation focus. The annual narrative report does not provide information about ICCOs experiences regarding involvement of its partners in PRSP processes. Could some information be given? Use of indicators for sustainable poverty alleviation objectives mentioned in appendix. Has information been collected? The annual narrative report does not review systematically the annual plan 2002. Reporting about PRSP processes insufficient: could HIVOS play a more active role to stimulate partners to participate in the PRSP processes. Observations from embassies: positive reflections on the activities supported by HIVOS, on one occasion the issue of co-ordination with bilateral activities was raised (Kazakhstan). Observations form thematic directorates: positive about the incorporation of gender in the annual report. Observations concern only financial management More explicit relationship between interventions and the planned results for 2002 Overview of involvement of partner organisations (now separate) to be included in the next annual report Observations from embassies: generally positive

ICCO

HIVOS

NOVIB Plan

139

7.3: Overview of observations and information requests Annual report 2003

Cordaid A good overview of Cordaids policy development is given. Cordaid has started several innovative paths. The results are not really methodologically reported. On a considerable amount of the intended results formulated in the annual plan is not reported in the annual report. Additional information is requested on several areas of result. Compliments are given on the openness of evaluation results. It is reported that not all result are met, however the reason for this is not given. Several financial request are made. It is requested if Cordaid is aware and works according to the European regulations for tendering. The observations of the embassies are generally positive. The observations and requests from the thematic directorate DSI/SB concern the HIV/Aids policy of Cordaid. The observations of the thematic directorate DMV/HH are positive. Hivos has a public annual report and an annual report for the CFP. The latter is in line with the Ministrys expectations of reporting on results (resultaatsmeting). For every sector the planned results and the achievements are given, on the level of partner organisations and the level of Hivos. Questions concerning content are asked on the Gender, Woman and development policy, environment and sustainable development policy and the human rights and HIV/Aids policy. Inquiries are made about the results of the partner consultations, and the influence of them on the development of the M&E and value (waarderings-) system. The embassies appreciate Hivos choice in partners and the role of Hivos in the capacity building of the partner organisations. The observations of the embassies are positive. The annual report relates a clear picture of the progress of ICCO in relations to the management plan. However, annual reports mainly relates results, which are SMART, but fails to relate the goals, results, activities and resources with each other (DRAM- system is not used). Annual report is not consistent with annual plan. Questions concerning content are asked on sustainable fair economic development, access to basic facilities, democracy and peace building, HIV/Aids and the public support in the Netherlands. Finance and control: no insight in control capacity and sanction policy. It is requested if ICCO is aware and works according to the European regulations for tendering. The observations of the embassies are generally positive. The annual report relates a clear relation with the annual plan. Results are related to the goals formulated in the annual plan, broken down to both the 5 basic rights and Novibs regions. Insight is given in the bottlenecks concerning the implementation of the annual plan. New instruments are implemented in order to work more result oriented, e.g. the toolbox. Questions concerning content are asked on partner policy, HIV/Aids policy, lobby & advocacy in the Netherlands. It is requested if Novib is aware and works according to the European regulations for tendering. The observations of the embassies vary substantially (positive and negative). The annual report specifies both the execution of the work plan 2003 and the strategic changes process (veranderingsproces) in the long-term. Plan NL is ISO-certified 9001:2000 The report clearly states the comparison between goals, activities and achievements, and the relation between them. In comparison with the year report 2002, the report is more DRAM and SMART. Besides this, Plan has reported the developments in their M&E system. The quality of the evaluations is assessed. The ministry would like to be informed about the follow-up of the streamlining of the project evaluations with result measurement on the different levels in the aid chain. The Minister is pleased with Plan activities on education. The results are formulated on the level of the programmes. It would be valuable if more attention would be given to the sustainability of the results and the transferability and the application on a larger scale as part of the education policy of the government. Plans attention for world wide education and lobby in the framework of lobby & advocacy and capacity building is valued as positive. The observations and information requested by the embassies were in the line of Plans CCCD approach and Plans effort to invest more in lobby and advocacy and civil society development.

Hivos ICCO

Novib

Plan NL

140

TdH

Good overview of necessary development changes within the organisation. The annual report doesnt relate sufficiently enough to the strategy of the business plan 2003-2006. What kind of policy adheres TdH with regard to imposing TdH regional policy on their partners? Does TdH follow guidelines for the average duration of their relationships with their partners? The annual report relates that TdH has invested in explaining the Objective Oriented Project Planning to their partner organisations. Have partner organisation experienced problems with the implementation of OOPP? What are the results of the training concerning the development of an unambiguous process of monitoring and evaluation? The gained results are not sufficiently focussed on the output level. The results are mainly given on activity level. Only the amount of beneficiaries are given. (It is expected that, due to the meetings between TdH and the Ministry concerning result oriented working (resultaatgericht werken) and the new format for the annual plans, the annual plan 2005 will be of a better quality.) In what way give the partner organisations account for their finances and do they do this on time? Extracted from the observations from the embassies, it appears that the embassies are not yet familiar with activities of TdH and their partner organisations.

7.4 Main topics covered during field trips executed by DSI/MY staff since 2000
Name officer countries 4 Brief description

A. Holleman

Indonesia

A. Holleman

Salvador/Honduras Bolivia/Peru

N. Blokker

N. Blokker

Centr. America

B.Paardekooper/v.d .Burg

Uganda

Schulting

Bangladesh

The field trip report of July 2001 for the activities supported by PLAN focussed on its new approach of Child Centred Community Development (CCCD), the shift in approach from provider to facilitator, the involvement of local NGOs in implementation. The field trip report of January 2004 to El Salvador and Honduras for the activities of Plan Netherlands focused on the policy and strategy followed by Plan Netherlands to support educational activities in the two countries. The field trip report of March 2002 to Bolivia and Peru informed about a meeting of the Civil Society Support Platform established by the RNE related tot the PRSP process, a consultation of the RNE with the five CFOs on complementarity and synergy and a visit to some projects supported by ICCO in both countries. The field trip report to Central America of March 2003 addressed the following issues: HIVOS Regional Field Office in San Jose, some projects visited in Guatemala and Nicaragua, some aspects of economic projects supported by HIVOS, donor co-ordination in Nicaragua, HIV/AIDS and environmental NGOs. The field trip report of November 2001 for the activities of both ICCO and PLAN NL focussed on a conference on the Role of NGOs in decentralisation and conflict resolution and a second conference on the Flow of Small Arms and Conflict Prevention (both conferences supported by ICCO), as well as a project in Kampala supported by PLAN. The field trip report to Bangladesh of February 2003 for the activities supported by Terre des Hommes NL addressed the following issues: TdHs project support policy in health and education, the sustainability of some projects implemented by TdH itself, the tensions between GOB and the NGO community, as well as the decision to explore the possibilities of co-operation between RNE and TdH in the framework of a sector wide approach in health.

141

7.5: COMMENTS BY DSI ON ANNUAL PLANS CFOS

COMMENTS ON CORDAID ANNUAL PLAN 2003 Annual plan is a clear elaboration of the management plan 2003-2006. The plan can be more SMART and DRAM formulated. Results are mostly given on Cordaid-level, and results are described in the light of activities or inputs in stead of output or outcomes. Information requested on campaigns planned in the Netherlands in 2003. Lack of connection between objectives of the MFP and disbursements. The annual plan is approved by DSI/MY, but confirmation and explanation is requested on some issues. COMMENTS ON CORDAID ANNUAL PLAN 2004 Positive reaction on annual plan 2004: plan contains a lot of information explaining Cordaids focus for 2004 on the basis of the management plan 2003-2006. Key results are summarised in 6 objectives with matching indicators Important step is the definition of results on 4 levels (target group, partner organisation, relationship Cordaid - partner organisation and Cordaid). The annual plan is approved by DSI/MY COMMENTS ON HIVOS ANNUAL PLAN 2003 Annual plan is not a specific and concrete elaboration of the management plan 2003-2006. Absence of concrete results. As verifiable results are lacking, it is impossible for DSI to assess if intended results are realistic. Results are not formulated SMART enough. Information request concerning development of monitoring and information system. Lack of connection between objectives of the MFP and disbursements. DSI/MY didnt approve the annual plan 2003 in first instance. After the reformulation of chapter 4 activities and results, the annual plan was approved. COMMENTS ON HIVOS ANNUAL PLAN 2004 Absence of lessons learned and changes in comparison to 2003. Where does Hivos stand in relation to the objectives of the management plan 2003-2006? Insufficient appliance of DRAM/SMART system. Better definition of results on output level of organisations is needed. Results are insufficiently SMART formulated. Information request concerning assessment system (waarderingssysteem) of partner organisations. Other comments on additional chapters, besides the chapter 4 results and activities. DSI/MY didnt approve the annual plan 2004 in first instance. After reformulation of chapter 4, the annual plan was approved. COMMENTS ON ICCO ANNUAL PLAN 2003 The annual plan is in compliance with the management plan 2003-2006 and is realistic. Suggestions are given to make the annual plan more result-focused: results and indicators have to be SMART, formulate results also on output-level of the partner organisations. Positive reaction on development of indicators concerning the partner policy. Comments on lack of absence on result indicators for the theme sustainable and fair economic development. Lack of connection between objectives of the MFP and disbursements. Request and comments on budget, as a result of which the annual plan wasnt approved in first instance. DSI/MY didnt approve the annual plan 2003 in first instance. After explanation of the 9 % AKV (operational costs), the annual plan 2003 was approved. Comments on ICCO annual plan 2004 Positive reaction about innovations on ICCOs partner policy and ICCOs activities in the North. Confusion about the results related to objectives on themes and regions in annual plan. Clear picture of result areas and intended results in the regional annual plans. Suggestion to define results in the future on 4 levels. Specific information request about priorities of ambitions in management plan, because of reduction in funds. The annual plan is approved by DSI/MY, but some comments and additional information requests are made on among others the absence of results for some of the objectives. COMMENTS ON NOVIB ANNUAL PLAN 2003 Positive reaction on chapter concerning objectives, activities and results on the basis of the 5 rights: clear picture what Novib wants to achieve in 2003 and why. DSI/MY is interested in the development of the system for result-measurements. Information requests on several operational choices on some themes/activities.

142

 Lack of connection between objectives of the MFP and disbursements. The annual plan is approved by DSI/MY, but confirmation and explanation is requested on some issues. Comments on Novib annual plan 2004 Positive reaction on annual plan: Novibs choices and the relation between objectives, results and activities are clear. No other comments on annual plan are made, only topics for the policy meetings are given (concerning among others the results of Novib of the internal organisation and the management plan 2005-2006). The annual plan is approved by DSI/MY COMMENTS ON PLAN ANNUAL PLAN 2003 Appreciation of connection made between lessons learned from 2001 and 2002 and the annual plan. Absence of consistency between the several regional plans The results can be more SMART formulated. Results are described in the light of activities or inputs in stead of output or outcomes. Feedback requested of the experiences concerning the development of the monitoring and evaluation of the partner organisations (indicators on quality of partner organisations and relation Plan country office and partner organisation) Lack of connection between objectives of the MFP and disbursements. The annual plan is approved by DSI/MY, but information and explanation is requested on some issues. Comments on Plan annual plan 2004 The annual plan is in compliance with the management plan 2003-2006 and with the Stappenplan Appreciation for the inclusion of learning points for 2004 DRAM system is used as basis for annual plan. Some of the results, however, can be formulated more SMART. Feedback requests on several operational choices on some themes/activities. Some financial information requests. The annual plan is approved by DSI/MY, but information and explanation is requested on some issues. COMMENTS ON TDH ANNUAL PLAN 2003 No clear connection between annual plan and management plan 2003-2006. Absence of overview of concrete targets and results. No result indicators are given. Results need to be SMART formulated. Information request on some thematic choices. Lack of connection between objectives of the MFP and disbursements. Some financial information requests are made. DSI/MY didnt approve the annual plan 2003 in first instance. After additional information on targets and results in 2003 and explanation on financial aspects, the annual plan 2003 was approved. Comments on TdH annual plan 2004 Objectives are better formulated in the annual plan 2004 than in the annual plan 2003. Information requested on the implementation of the OOPP system. The connection between local activities and central policies have to become more explicit in future. Absence of specification of amount of activities per region. No clear relation between input of resources and the regions. Positive reaction on the use of logframes, however the quality of the logframes differ for each region. For the next annual plan, some improvements in the logframes need to be made. The annual plan 2004 is approved by DSI/MY, but information is requested on some issues. It is also stated that for the annual plan 2005 some considerable improvements have to be made.

143

144

Appendix 8 Relevant Literature

145

Appendix 8: Relevant literature

GENERAL
Commissie Box. (2003). Breed uitgemeten. Eindrapport van de Adviescommissie Medefinancieringsprogramma. Commisse Box. (2002). Breed uitgemeten. Advies van de commissie Medefinancieringsprogramma-breed inzake toetreding en toewijzing van middelen 20032006. Anheier, Helmut and Themudo, Nuno. Organisational forms of Global Civil Society: Implications of Going Global. Custers, Geert. (2000 juli/ augustus). Ontwikkelingssamenwerking getoetst. Gebrekkige aandacht in beleid en debat voor resultaten. Internationale Spectator. OECD/DAC. (2002). Glossary of key terms in evaluation and results based management. GOM. (1995). Met het oog op kwaliteit. GOM.(2002). MFP-evaluatie 2003-2006. GOM Werkgroep Kwaliteit.(2004). GOM Kwaliteitshuis 2003-2006. Hoebink, Paul. (2002 April). Van wervelwind tot nachtkaars? Vier jaar Eveline Herfkens op Ontwikkelingssamenwerking. International Spectator. IAC/Wageningen.(2003). Report for Plan Netherlands on Impact Assessment. Wageningen. IBO Beleidsonderzoek Medefinancieringsprogramma (2000). Eindrapport Interdepartementaal beleidsonderzoek Medefinancieringsprogramma.

MFP-Breed Netwerk. (2004). Plan van aanpak en kwaliteitscriteria tweede fase MFOevaluaties 2003-2006: november 2004-juni 2006.
KPMG. (2004). Rapportage Doelmatigheidsaudit Medefinancieringsorganisatie. Lieten, Kristoffel & Velden, van de Fons. (1997). Grenzen aan de hulp, Beleid en effecten van ontwikkelingssamenwerking. Amsterdam: het Spinhuis. Mancuso Brehm, Vicky. (2004). Autonomy or Dependence? North-South NGO Partnerships. INTRAC Briefing Paper no. 6. Nekkers, J.A. & Malcontent, P.A.M. (Red.). (2000). De geschiedenis van vijftig jaar Nederlandse ontwikkelingssamenwerking 1949-1999. Sdu Uitgevers, Den Haag. Schulpen, Lau & Hoebink, Paul. (..). Ontwikkelingssamenwerking via particuliere ontwikkelingsorganisaties de MFOs in perspectief. Particulier Kanaal pp 159-191. Schulpen, Lau (red.). (2001). Hulp in ontwikkeling, Bouwstenen voor de toekomst van internationale samenwerking. Assen: Koninklijke van Gorcum. Steering Committee for the Evaluation of the Co-financing Programme. (2002). Final Report.

146

Stuurgroep Impactstudie Medefinancieringsprogramma (1991). Betekening van het medefinancieringsprogramma: een verkenning. James Paul.(2000).NGOs and Global Policy-Making. Global Policy Forum. Toche, Eduardo. (2003). ONG, enemigos imaginados. Desco, Lima.

MINISTRY OF FOREIGN AFFAIRS

Annual plan and reports DSI/MY

Annual report/ plan 2001-2002 Annual report/ plan 2002-2003 Annual report/ plan 2003-2004 Policy papers and documents Beleidskader Medefinancieringsprogramma-breed (MFP-Breed).(2001). Civil society en structurele armoedebestrijding. (2000) Formatieplan 1995- DSI/MY.(1995). Leliveld, A.H.M. Overzicht van kritische punten en aanbevelingen uit de evaluatierapporten van de Stuurgroep Evaluatie Medefinancieringsprogramma.(2003). Mutual Interest, Mutual Responsibilities.(2003).

CORDAID
Annual reports and plans

Cordaid Jaarplan 2003 Cordaid Jaarplan 2004

Cordaid Jaarverslag 2001, including CD-rom projects 2001

Cordaid Jaarverslag 2002, including CD-rom projects 2002 Cordaid Jaarverslag 2003, including CD-rom projects 2003 Cordaid strategie 2003-2006 Cordaid Subsidieaanvraag (10-10-2000)
Policy papers and documents

Code of Conduct (08-09-2003) Cordaid notitie partnerfocus (juli 2001) Lerende organisatie, een trektocht: ervaring met Lekgotla in 2003 genventariseerd (03-03-2004) Lerende organisatie, bijlage: richtvragen voor Lekgotla (24-03-2004) Samenwerkingsbeleid/Partnerbeleid (version 4, sept. 2003) Werken aan goed donorschap (2nd version 9-10-2003) Procedures and instruments Contouren Cordaids systematiek voor Planning, Budgetting, Monitoring en Evaluatie (2811-2003) Cordaid rapportage procedure Cordaid reporting guidelines financial 147

Monitoring en evaluatie op Cordaid niveau (4-12-2003) Monitoring- en Evaluatie systematiek (24-6-2002) On Result Based Management, Logframe Analyses, Performance Indicators and Performance Measurement (2nd version, 21-10-2003) Reporting Guidelines Cordaid

HIVOS
Annual reports and plans

Bedrijfsplan Hivos 2003 2006 Hivos Jaarplan 2003 Hivos Jaarplan 2004 Hivos Jaarverslag 2001 Hivos Jaarverslag 2002 Hivos Jaarverslag 2003 Hivos Partnerorganisaties 2002 Hivos Partnerorganisaties 2003 (Kwantitatieve) MFP-verantwoording 2001 (Kwantitatieve) MFP-verantwoording 2002 MFP-verantwoording 2003 Partnerboeken Hivos Afrika, Azie, Latijns Amerika en Z,O. Europa 2002
Policy Papers and Documents

Hivos Landenanalyse Peru 2003-2006 Hivos Partner Policy (14-12-2004)

Procedures and instruments

Hivos General Conditions (mei 2002) Hivos Monitoring & Evaluatiesysteem, Update beschrijving M&E systeem Hivos (oktober 2002) Hoofdstuk 16: Financile verantwoordingen, Rode Map IV Verantwoordingen Organisation Assessment (BO/ Beoordeling Organisatie.) Concept: final version (1712-01) Outline of Hivos future result assessment system, Planningsweek 2002/ resultaatmeting (4-10-2002) Update on Hivos process of improvement of the result assessment system. (23-112003) Hivos Policy Framework for improved result orientation and result assessment (2005)

ICCO
Annual reports and plans

ICCO Bedrijfsplan periode 2003-2006 ICCO Jaarplan 2003 ICCO Jaarplan 2004 ICCO Jaarverslag 2001

148

ICCO Jaarverslag 2002 ICCO Jaarverslag 2003 Projectboeken 2002 Afrika en Midden-Oosten Azi, Europa en Oceanie, LatijnsAmerika en Wereldwijd, Versterking van het draagvlak
Policy papers and documents

(Be)wegen tot gerechtigheid ICCO beleidsplan 2001-2005 (2000) Building Bridges in the PME (August 2001) Concept Landenjaarplan Tanzania (versie 04-07-03) Follow-up ICCO eindrapport stuurgroep MFP (28-4-2003) Het gebruik van (project)evaluaties, Intern Onderzoek (30 juni 2003) ICCO Beleidsindicatoren (1-9-2003) ICCOs Code of Conduct and Principles of Good Donorship ICCO Evaluatiebeleid voor het primaire proces (21-06-04) ICCO Policy Choices and Implementation for the Period 2003-2006 (june 2003) The pressure to measure, Planning, monitoring and evaluation at ICCO (5-9-2003)
Procedures and instruments

Algemene contractvoorwaarden (version 8-4-2004) Beoordeling Financile eindrapportages (9-10-2003) Beoordeling Financile voortgangsrapportages (9-10-2003) Definities (bij Contractvoorwaarden) Eindmemo (9-10-2003) Financile beoordeling organisatie en financieringsvoorstel (9-10-2003) Gebruikershandleiding nieuwe instrumentarium (14-10-2003) Handboek Handleiding Financieringsproces (2-10-2003) ICCO beleidsindicatoren (version 01-09-2003) ICCO Handboek Financieringen, Hoofdstuk Monitoring Voortgang (version 18-032002) ICCO Guidebook For Appraisal and reporting on savings and credit activities (15-22000) Intake MEMO (1-9-2003) KIDBEMO (1-9-2003) Richtlijnen voor toepassing van KIDBEMO Financieel (9-10-2003) Specifieke contractvoorwaarden (version 8-6-2004) Voorstel ontwikkeling indicatoren resultaatsmeting ICCO, draft. (version 02-01-03) Voortgangsmemo (version 9-10-2003)

NOVIB
Annual reports and plans

Novib Bedrijfsplan 2003-2006, Subsidie aanvraag 2003-2006. (lapsed) Novib Bedrijfsplan 2003-2006: Bijlagen, Subsidie aanvraag 2003-2006. (lapsed) Novib Bedrijfsplan 2005-2006 Novib Jaarplan 2003 Novib Jaarplan 2004

149

Novib Jaarverslag 2002 Novib Jaarverslag 2003 Novib Kenschetsen 2002 (cd-rom) Novib Kenschetsen 2003 Policy papers and documents Capitalizacin de la experiencia sobre sistemas de PME con contrapartes de Novib en Peru. Escuela para el Deasarrollo, Lima, 2003 (cd-rom) Novib and its counterparts (September 2004) Novib Meerjaren Plan 2001-2004. (2000)
Novibs visie op armoede: Bijlage subsidie-aanvraag 2003-2006

Stappenplan 2003-2004; Eindrapport stuurgroep evaluatie medefinancieringsprogramma Towards Global Equity: Oxfam Internationals Strategic Plan, 2001-2004. (2000) Novibs evaluation policy (2004) Procedures and instruments Financieringsovereenkomst (15-07-2004). Handboek processen afdeling Projecten: Inleiding op Handboek en Processen, Handboek Kernprocessen (HPK) 2002-10-17, Handboek Algemene Processen (HAP) 2002-5-31, Handboek Overige Processen (HOP) 2002-11-01. Improving programme and project evaluation practice Novibs Grant-Making Opportunity and Risks Appraisal Methodology. Grantees frequently asked questions and our answers. (2003) Performance Registration System, Instructions and Definitions.(version 28-1-2004) Performance Management in strategic development funding. (16-9-2004). Richtlijnen voor het voortgangsverslag (2-08-2001) Richtlijnen voor Rapporteren (01-07-2004) Toolbox 2004 (Grant-making opportunity and risk assessment)

PLAN NL
Annual reports and plans

Annual Progress Report, West Africa Regional Office, WAR0009, draft, (26-11-2003) Annual Report FGM (Female Genital Mutilation), Plan Mali, Mali Womens Reproductive Health Awareness Programme (December 2002 draft) Bedrijfsplan 2003-2006 Subsidieaanvraag MFP Jaarplan 2004, Final version Jaarplan Medefinancieringsprogramma 2003, MFP werkplan 2003 (december 2002) Jaarplan Medefinancieringsprogramma 2003, update MFP werkplan 2003 (mei 2003) Projectboeken Afrika, Azie, Latijns Amerika, Nederland, Bijlagen jaarverslag 2003 Regional Plans: Subsidy request MFP, 2003-2006 Verwachte resultaten Jaarplan 2003 Afrika

150

Policy papers and documents

Partnership and strategic alliances: a framework for Plan, 2003 Plan Nederland, Notitie over goeddonorschap Report for Plan Netherlands on Impact Assessement, International Agricultural Centre (IAC), Wageningen, May 2003 Procedures and instruments Corporate Planning, Monitoring and Evaluation (CPME) Reference Guide (20-10-2003) Draft Matrix for Plan Netherlands Commitment to fight Child Poverty Kwaliteitshandboek Programmas en Projecten, Lerende organisatie, KH60(version A 2608-2003) Memo. Kwaliteit evaluaties ten behoeve van resultaatmeting Monitoring Sheet Strategische Allianties Monitoring jaarplan 2003, Beleidsunit (24-9-2003) Monitoring jaarplan 2003, Regio Latijns-Amerika (24-9-2003) Output codes Plan Yearly Progress Report Format (draft) (1-10-2002) Plan International Output List (version September 2001) Procedureboek - Identificatiefase primair proces, PR01 versie D (30-01-2003) - Formuleringsfase primair proces, PR02, versie B (11-12-2002) - Beoordelingsfase primair proces, PR03, versie D (30-1-03) - Financieringsfase primair proces, PR04, versie B (11-12-2002) - Uitvoering- en monitoringsfase primair proces, PR05, versie C (11-12-2002) - Evaluatiefase primair proces, PR06, versie F (31-01-2003) - Afsluitingsfase primair proces, PR07, versie D (23-12-2002) - Veldbezoek, PR08, versie D (25-08-2003)

TERRE DES HOMMES

Annual reports and plans

Bedrijfsplan 2003-2006, Stichting Terre des Hommes Nederland Bedrijfsplan 2003-2006: Bijlagen, Stichting Terre des Hommes Nederland Jaarplan 2003 Stichting Terre des Hommes Nederland Jaarplan & Begroting 2004 Terre des Hommes Nederland Jaarverslag TdH 2002 Jaarverslag 2003 Medefinancieringsprogramma Stichting Terre des Hommes Nederland Werkplan en Begroting 2003
Policy papers and documents

Beleidsplan 2001-2005, Stichting Terre des Hommes Nederland

151

Procedures and instruments

Handboek Kwaliteit Terre des Hommes Nederland (nen-en-iso 9001). (1 maart 2004) Handboek Kwaliteit Terre des Hommes Nederland (nen-en-iso 9001: 2000). (1 april 2004) Handboek Operational Audit Regiokantoren, Stichting Terre des Hommes Nederland. (April 2004) Handboek Regiovertegenwoordigers, Stichting Terre des Hommes Nederland. (April 2004)

152