Type of Review: Annual Review Project Title: Girl Hub Ethiopia Date started: September 2012 6 Nov 2013.

Date review undertaken: 31 Oct

Instructions to help complete this template: Before commencing the annual review you should have to hand: the Business Case or earlier project documentation. the Logframe the detailed guidance (How to Note)- Reviewing and Scoring Projects the most recent annual review (where appropriate) and other related monitoring reports key data from ARIES, including the risk rating the separate project scoring calculation sheet (pending access to ARIES)

You should assess and rate the individual outputs using the following rating scale and description. ARIES and the separate project scoring calculation sheet will calculate the overall output score taking account of the weightings and individual outputs scores:

Description Outputs substantially exceeded expectation Outputs moderately exceeded expectation Outputs met expectation Outputs moderately did not meet expectation Outputs substantially did not meet expectation

Scale A++ A+ A B C

Introduction and Context

What support is the UK providing?

Girl Hub is a strategic collaboration between DFID and the Nike Foundation. In Ethiopia, Girl Hub is a DFID team. Although based in a separate office, GHE uses DFID systems and reports to the DFID Ethiopia Head of Office as well as to Girl Hub Global and the Nike Foundation.

Girl Hub Ethiopia had an initial 1m funding of start-up costs approved in November 2011. The main grant, approved by Secretary of State in October 2012, is for 8.8m for 2.5 years. Additional funding of 1.4m is provided by NF, 13.7% of the total project funding. Both parent organisations also provide in kind support. This review covers progress against the logframe for the period October 2012 to October 2013, taking into account the activities carried out during the period from November 2011 to October 2012. The branded social communications element of the programme, Yegna, is implemented through a consortium called Yegna Bet, formed from three companies in response to a competitive process. The consortium was selected through DFID procurement processes. The other elements of the programme are delivered by Girl Hub, which comprises a team of DFID, Girl Hub and Nike Foundation staff.

What are the expected results?

Girl Hub Ethiopia aims to help Ethiopia break the inter-generational cycle of poverty by 2030. In terms of impact, GHE wants to see 3 million girls inspired and their world transformed. This will be accomplished by achieving results in the following four areas that line up with DFID priorities under the Strategic Vision for Girls and Women: Decrease the number of girls suffering from gender-based violence Decrease the number of girls who marry or give birth before the age of 18 Increase the number of girls who have completed primary and transitioned to secondary education Increase the number of girls with control over economic assets

GHE uses a package of social communications products and direct influencing with Ethiopian development and government partners, aimed at improving attitudes and behaviours towards girls. Progress will be measured by an improvement in: girls' perception of themselves (including the way they value themselves, their opinions and their abilities); girls' access to information, supportive social networks and influence over their own life choices; boys', parents' and community members perceptions and behaviours toward girls; and, amount of new partner (government/donor/NGO) resource allocations for girls due to influencing by Girl Hub Ethiopia.

According to the business case the following outputs will be achieved by 2015: Nearly 10 million citizens will be inspired into action by branded social communications products and outreach strategies. This refers to the number of listeners reached by the Yegna radio broadcasts. Additional deliverables are an expected 30,000 girls in Yegna outreach activities, the quality of the Yegna branded offerings, and an expected 35%of the population exposed to Yegna will retain behaviour change messaging. Seven new initiatives targeting adolescent girls will be implemented due to Girl Hub Ethiopias catalytic and convening role An additional target refers to the quality of GHE engagement, communication and advocacy on girls. 12,000 citizens (half of whom are girls) will participate in research and girl-centred participatory M&E to generate real-time, robust evidence and insight for use by communications, systems, investments and policies Other results are the quality of the knowledge products and the number of these disseminated, as well as Girl Hub Ethiopia organisational development achieved and effective.

What is the context in which UK support is provided?

In Ethiopia, adolescent girls aged 10-19 make up 24% of the population. Among these girls, nearly 7 million live on less than US$2 a day and half are married by the age of 15. The majority of girls aged 15-19 have not completed primary education. According to the 2012 UNESCO Global Monitoring Report Ethiopias Female youth illiteracy rate for the age group (15-24) was 62% (7.5 million) in 2006. This was less than the average of 59% for developing countries and Sub-Saharan Africa. Girls survival rate to last grade of primary in 2013 is 52%, much less than the developing country average of 83%, and Sub-Saharan Africa Average of 71% reported on the 2012 GMR. Broadly, social norms do not value or support girls and their safety and mobility are curtailed. Nine out of ten girls need permission before leaving their house. These norms are being transmitted to the next generation. 51% of boys agree that there are instances when a husband is justified in hitting or beating his wife. While development programming focused on gender typically reaches older women and youth work tends to target boys, adolescent girls are left behind.

Section A: Detailed Output Scoring

Output 1: Branded social communications products and outreach strategies are activated and inspiring
Output 1 score and performance description: C GHE has developed and produced a branded package of social communications products including two radio shows, music and outreach activities. Yegna, meaning ours in Amharic, is primarily a radio drama and radio magazine show featuring a band of five girls that aims to inspire and entertain girls and their communities, to be a credible source of information for girls and to role model positive attitudes and behaviours toward girls. The package launched in April 2013. Series 1 of Yegna was launched with a range of activities including direct outreach and billboards aiming to reach both rural and urban audiences. Series 2 of Yegna launched in October 2013, preceded by two promotional concerts held in Bahir Dar and Addis Ababa, each of which attracted audiences of more than 10,000 people. Girls were involved in the design of the brand and current package of products. Since the launch, girls and their communities have been able to feedback to and participate in the Yegna work through listening groups, voicemail and text message services, social media platforms and by speaking with Yegna Ambassadors, a group of university educated young people trained to promote and represent Yegna. The primary outreach programmes are based around Yegna Ambassadors and Yegna Listening Groups. GHE states that there are 157 girl Yegna Ambassadors (girls aged 18-19 years out of 494 Ambassadors total) and 260 girl members of weekly listening groups, with 35 groups in total. A third form of outreach, branded content and curriculum for safe spaces or club programmes has been designed but is yet to be rolled out. Overall, the Yegna brand and products are of very high quality and clearly engage their audience, based on the experience of listener groups who are not compensated financially for their participation. The results of the first representative Yegna listenership study show that the products successfully engage their audience, but that the reach of the products and number of listeners is lower than expected. The social communications model for behaviour change is complicated and impact will take time to measure.

GH E has recently completed its first listenership survey and this has raised important questions about both the limitations of radio as a channel to reach the intended audience numbers. The GH Monitoring and Learning Strategy and individual study methodologies are designed to gather evidence to determine whether the whole range of products is necessary to achieve the intended outcomes. As a result of the first survey findings, Girl Hub Ethiopia is developing a Yegna Reach Strategy. This will review updated evidence, lessons learned and recommend a new mix of channels and products. Decisions will be made using a VfM analysis. The M&L framework is also being refined to strengthen Girl Hubs capacity to ensure that future roll out of Yegna will increase the evidence of what mix of products is needed to reach different audiences at the right dose, what it costs to deliver Yegna to specific audiences at different levels of intensity and what the impact is. The launch of Output One was delayed because of DFIDs new approval procedures implemented in 2012, which lengthened approval times of both the GHE business case and the subsequent Yegna Bet CMA contract. The logframe set targets based on launching in January 2013, however launch was delayed until April and M&E activities, including the listenership survey, were pushed back accordingly. This delay also had some impact on the numbers of listeners reached (see below) and on cost. Marketing and Yegna Bet charges have surpassed indicative expenditure set out in the Business Case for the first year. Additional funds were provided by Nike Foundation. We understand that this decision was approved jointly by the Head of DFID E and General Manager GH London under the joint quarterly governance mechanism, taking risk management into account. We discuss this issue in section 2. Lower than expected listenership figures are due to a combination of the delay to Yegna lauch, inaccurate data and broadcast blackouts. It is difficult to distil out the relative impact of each of these things. In the first year of implementation Yegna aimed to reach an ambitious 25% of the population with access to radios. The absolute number of listeners in the milestone was calculated using evidence from an FCO-funded Audience Survey in Ethiopia which showed that over 70% of the population has access to a radio and that nearly 60% of people listen to the radio every day. GHEs own listenership survey in Sept 2012 has shown that this data is wrong. The assumption made in the FCO study that people listen to the radio communally is wrong and therefore, radio ownership is a better indicator of reach than radio access. According to the DHS 2011, radio ownership is 23%. Secondly, Girl Hubs own real-time feedback mechanisms with communities in Amhara have revealed issues with poor radio coverage and broadcast black spots. This information is not available from the radio stations, and was therefore not available during design. A second radio station (in addition to the one originally planned) has broadcast Yegna in Addis and Amhara, but this has not made up for the short fall. No evidence is available in Ethiopia on the extent and frequency of broadcasting blackouts. Girl Hub has gathered evidence from their own listenership groups and learned that in areas where people own radios and are listening to Yegna, blackouts are affecting listening habits. Thirdly, as noted above, launch was delayed due to new DFID procedures, meaning that the time available for Yegna to reach audience numbers was limited. The targets in the logframe were based on a project start date of Sept 2012. It is too early to see evidence that Yegna outreach activities to date fulfil the intended goal of building girls confidence and capabilities. It is expected that listener groups, clubs, and Yegna Ambassadors will begin to address this issue, and this will require monitoring as planned. The Yegna Ambassadors, although university educated, were selected specifically because they are from the more remote areas, could provide direct outreach to poorer girls and serve as role models from the communities where girls are least likely to achieve access to university. Listener groups are primarily engaged in providing feedback for the Yegna products. There are plans to better mobilise these groups in a similar way to community youth groups through branded teaching materials, as described above. Girl Hub Ethiopia has taken steps to address the risk that the Yegna products dont reach the intended numbers of people through the development of a Yegna Reach Strategy, which will be completed in January 2014. The Reach Strategy will outline how Yegna will reach a revised target number of people, through a mix of channels. The M&L framework will be refined to test the impact and cost of the approach with different communities, including more/less vulnerable groups of girls. Girl Hub will

deliver the strategy within the existing budget envelope and use VfM analysis to make decisions about the mix of channels and Yegna products, in order to balance a) reaching the maximum number of listeners possible with b) testing the theory of change and learning what works. We agree with this approach but recommend that cost comparators are identified and used as part of the approach to testing what works for girls and defining the added value of Yegna. Consideration will be given to maximising the use of the first two series of Yegna before producing more. Consideration also will be given to mobilising further private sector funding, in addition to the US$ 1 million of marketing funds already committed by Ambo. Progress against expected results: All data provided by GHE and self-scored by GHE.
Indicator June 2013 milestone Total: 3,628,566 (estimated 25% of population in broadcast area with access to radio) Progress 407,844 Not Met (preliminary data from Listenership survey currently under review) Clubs not yet started; however 157 girls active as Yegna Ambassadors 251 girl members of weekly listening groups Score as of September 2013 2 - Agree There was a unanimous decision that output one achieved a score of agree. The response from our listener groups has been very positive and indicates that our platform is girl focused, trusted, inclusive, and contextualized. We held back from giving a score of strongly agree as we wont know if its effective until the listenership results debrief and its not yet scalable or sustainable. Not met/ met/ exceeded

# of citizens reached by Yegna radio drama and talk show

# of girls participating in Yegna outreach activities

300 girls in clubs

Not met

Quality of social communications products: relevance, audience engagement and brand loyalty, level of grounding in girls insights and codesign processes, and sustainability

March 2013 QAS Score of 2 - Agree: social comms platform is engaging and relevant for audiences, continued use of girls insights in creative development, business plan implemented

Met

% of exposed population retaining behaviour change messaging from social comms platform

December 2013 25% of population

Progress as of October 2013 On average, 50% of listeners answered Yegna radio drama message retention questions correctly 35% listeners answered Yegna Saat questions correctly (Figures do not include respondents who have only listened to Yegna once. From listenership survey)

Exceeded

This Output scores a C. For two milestones: listenership numbers and clubs, expectations have not been not met. Performance against milestones for message retention and the quality of the Yegna products and brand met or exceeded goals. Recommendations: Update the VfM strategy in the light of the forthcoming recommendations of the Reach Strategy for Yegna. This should enable GHE to evaluate whether or not it has the right mix of marketing and outreach strategies to get the maximum reach with the right product mix at the lowest cost. This needs to be combined with on-going communications & advocacy work to ensure that partners can engage with and understand GHE investment in a branded social communications product. The logframe should be revised, as planned, to reflect the Reach Strategy, including revisions on recommended targets if necessary. The risk of the Yegna project should be reviewed based on lessons learned from recent international and local media publicity, including a review of the extent to which systems for managing response to such publicity proved robust and fit for purpose.

Impact Weighting (%): 40% Revised since last Annual Review? N Risk: Medium Revised since last Annual Review? N

Output 2: Girl Hubs catalysing and convening role is established and effective
Output 2 score and performance description: A GHE is not an operational NGO that directly delivers programming for girls but rather works to catalyse action by others through advocacy on the need to reach girls and address the challenges they face in mainstream development. GHE offers technical assistance for designing girl-centred programmes and services. GHE has shifted its focus from leveraging more resources for girls through scale partners (especially DFID), to using evidence to mobilise a wide range of development and government actors to increase and improve programming for girls in specific impact areas, as set out in the logframe.

GHE has worked through partnerships to influence three programmes to focus on and promote girls interests: Alliance to End Child Marriage, PAGES education project, and Promotion of Basic Services (PBS). For example, in January 2013 and in collaboration with DFID, the Secretariat of the PBS Programme and under the guidance of the State Minister of Finance, Dr Abraham, GHE commissioned a study on Smart Economics to increase the effectiveness of their advocacy work. The work uses a gender and growth diagnostic framework to identify where greater investment in girls, or more efficient coordination of current investments, could yield a growth dividend. The purpose of this work was threefold. Firstly, to respond to a request from the PBS Secretariat of donors to understand more about how a Public Financial Management programme working in 5 service sectors could be more efficient by focusing more on adolescent girls. Secondly, to test the girl effect hypothesis that girls drive growth and poverty reduction, and test the evidence for this in the Ethiopian context for policy dialogue. Thirdly, to build an evidence base for GH Es advocacy strategy and to identify where investment in scale services, integrated with social norm change, would support the GoE ambition to ensure that girls benefit from and contribute to Ethiopias development. The report identifies entry points for GHE influencing work and provides an evidence base for dialogue with the Government of Ethiopia, the PBS programme and interested donor partners to better address girls needs. Although GHE is a DFID team, the structure of GHE leads to interaction with DFID E that is more like a partnership, reinforced by GHEs separate office and staff. GHE sits on the Steering Committee of the Finote Hiwot programme, providing technical assistance to the new Community Security and Justice programme and RMNH programme. GHE also uses the internal communications systems of DFID, Nike Foundation and Nike Inc to raise awareness of GHs work within the parent organisations. GHEs progress on catalysing action for girls is mixed. It is very early to measure the influence of GHE in their various partnerships and on the Ethiopian Government and development community, and this is reflected in the qualitative milestone in the logframe. GHE is doing well to learn from their early experiences and is altering their strategy to increase their impact as the evidence is built. There has been a significant level of engagement with a small number of partners, and GHE has used its own qualitative tracking process to try to understand progress towards outcomes for girls. The team has adopted an innovative portfolio management approach to this component and is using the QAS to adjust levels of investment in different partnerships against perceived returns, in real time. In the example of the PAGES programme, GHE helped secure 10 million of new funding for girls in Afar, and subsequent collaboration with the PAGES consortium has identified where GH E could add most value in implementation. For instance, when Consortium demand for insights work intended to help identify the best places for new schools to be constructed according to girls needs was low, the GH E team reduced their planned inputs in that area and focused instead on meeting Consortium demand to support the design of girls clubs, branded social communications and the M&L framework. The review identified questions about the basis on which GHE engages with partners, DFID E and the Ethiopian Government. This work is largely dependent on the work of the GH Country Director. In May 2013, two girl experts were recruited and have begun to contribute to this output. Partner interviews indicate some lack of clarity about the relationship between DFID and Girl Hub. GHE has begun to develop partnership notes to improve clarity and communications with partners, and will develop a partnership note with DFID that clarifies the relationship based on experience with individual projects. GHE needs to develop clear and persuasive narrative to communicate how it will test & demonstrate whether its advocacy and communications tools, including Girl Insights, Girl-led journeys and Girl Effect University, lead to more and better programming for girls. In addition, Yegna should be a very powerful tool for driving the influencing, catalysing and convening process. Girl Hub has a plan to determine how better integration of work across outputs will harness Yegna as a force to drive progress in Output 2. Progress against expected results:
Indicator March 2013 milestone Progress Not met/ met/ exceeded

Quality of engagement and communications/advocacy products to move DFID E, national govt and global scale partners from interest to commitment to action/delivery

Disagree - influential engagement with DFIDE and small number of other partners; new potential partners seek relationships with GHE (Measured by internal QAS process)

September 2013 3 Disagree GHE is initiating relationships with global and national partners but the relationships are not yet strong enough to influence implementation at scale. Early relationships with the private sector in Ethiopia are in development. March 2013 6 DFID Collaborations: Alliance to End Child Marriage: Alliance to End Child Marriage launched. GH Influence: Providing leadership to the Alliance; convened a Girl Effect University (GEU) workshop with members for strategy setting and workplan development; mobilisation around the National HTP Strategy PAGES education project: PAGES launched. GH influence: Girl-focused technical inputs to concept note, proposal development, M&E framework and inception phase; provision of insights and TA for development of girls clubs and social communications activities; inputs to three design workshops. Promotion of Basic Services (PBS): Smart economics initiative established. GH influence: Ongoing relationship to build girls interests into PBS linked to GTP2, influenced World Bank to agree to host and fund a girl expert to sit within PBS Secretariat for two years; currently developing a workplan to support PBS development of an equity survey to support equity formula for

Met

# new initiatives targeting adolescent girls in Ethiopia due to GHE influence

Exceeded

the block grant. Preliminary workplan developed. DFID Programmes: Finote Hiwot (FH): Programme under implrementation. GH influence: Member of FH Steering Committee; provided human resource inputs into annual review team. Partnership framework developed with DFID E. GH value add to ensure that the programme is girlcentered. RMNH programme: Programme design. GH influence: GHE organised girl-led learning journeys for RMNH design team and provided technical inputs for design; partnership framework agreed with DFID E to formalize the collaboration and set a roadmap for resource allocation/measures of success. Community Security and Justice: Programme design. GHE organized girl-led learning journeys for the CSJ design team and provided technical advice throughout the process to date. We are in the process of formalizing a road map for collaboration/resource allocation and measures of success with DFID E.

Girl Hub identified early in the year that there was a need to tighten this indicator in the logframe. The reviewers have scored this output by considering the evidence that GHs partnerships have has been effective at either a) influencing something new to happen with an exclusive focus on adolescent girls (Aliance, PAGES, Smart Economics) or b) increased the focus on adolescent girls of new or existing programmes (support to better targeting and mainstreaming within DFID programmes). A score of A is warranted because GHE has constructively engaged with partners (including the government), has influenced them toward better focus on girls and is working to improve GHEs own performance on this output. The Smart Economics work is a strong example of a new initiative with an exclusive focus on adolescent girls with good ownership by a number of mainstream or scale partners. We suggest that the indicator of new initiatives targeting adolescent girls in Ethiopia due to GHE influence be clarified in future to better assess GHs impact. GH impact could cover leveraging more investment in girls, changes in the policy or legislative environment, better targeting towards girls, better quality girlcentred programming or better co-ordination and learning across the community of existing actors working on adolescent girls in Ethiopia.

Recommendations: GHE has an opportunity to provide excellent VfM through their advocacy and influencing work but must capitalise on their current strategy process (the end state plan) to develop a more integrated approach to how, why and where GHE should intervene. We envisage that this will be done by the end state plan which has been conducted since the Annual Review, and encourage the team to test whether the current portfolio of partnerships is strategic, as well as use the end state plan to guide future decisions about who to work with and which tools and approaches to use. The link between Output 1 and Output 2 should be more explicit to capitalise on the prominence of the Yegna programme. The logframe should be revised to include a more precise measure of GHE influence. The Country Director should reduce her responsibility for this output in order to increase the capacity of GHE and ensure sustainability of GHE influencing relationships. GHE should consider hiring a senior level girl expert in anticipation that a future CD may not be a DFID SDA.

Impact Weighting (%): 30% Revised since last Annual Review? N Risk: Medium Revised since last Annual Review? N

Output 3: Research and girl-centred participatory M&E (PM&E) generate real-time, robust evidence and insight for use by communications, systems, investments and policies
Output 3 score and performance description: A+ GHE produces girl-centred research at many points in the delivery process. The generation of insights into girls lives has shaped GHE strategy and approach from the start and continues to guide the evolution of the Yegna brand. On-going monitoring is conducted according to the GHE M&L Framework and Yegna M&L Strategy, which set out how evidence will be collected and analysed against the logframe and theory of change. GHE has engaged 6,170 citizens in PM&E activities to date, including 3,460 girls. Over 900 girls were involved specifically in insights, co-creation and storytelling research activities that fed directly into the conceptualization and realization of Yegna. Data is disaggregated by age and sex. Recently, GHE conducted a baseline study on attitudes and behaviours toward girls that involved a community feedback component to share findings back with girls and their communities to help validate, challenge and explain the data. GHE has also contributed Insights work to the PAGES education project. GHE has developed knowledge products to help make the case for social communication work on behaviour change such as the Drivers of Change report and the Value For Money strategy. These have been disseminated within DFID, GH global and NF. GHE has also shared the SenseMaker platform within DFID, INGOs and research organisations. In May 2013, GHE hosted a Girl Learning Symposium that brought together organisations within the Ethiopian girl research and practice community to share each others work (including GHE SenseMaker and girl-centred tools) and build momentum for greater knowledge sharing and collaboration. In addition, we understand that some communications products are used for internal communications for both for DFID and NF, as well as for the Yegna strategy, for example the Yegna music videos, the

10

YegnaPlayer media platform and similar products. It will be important to clarify DFID and NF interests in this respect, and how internal communications products contribute to overall GHE goals. It would be good to clarify what systems are in place or anticipated for monitoring & evaluating the impact of products such as the Afar video on programming. Feedback from the PAGES partners & DFID has been positive. High quality research products like the Smart Economics study show GHEs comparative advantage in producing targeted materials quickly to influence development actors and the Ethiopian government, and it will be important to promote effective dissemination through a range of communication products with different audiences. Many monitoring and research activities are carried out in relation to Yegna, the review team felt that further clarification was needed on whether their purpose is market research measuring the reach and appeal of the Yegna products, or the impact for girls. It is important to show how girls and citizens participation in research processes leads to developing better products and programmes for girls, and test the theory that this will deliver better development outcomes. Currently, the listener groups, mobile interactivity and other ways girls and other citizens interact with and feedback on Yegna products are directed at improving Yegna as a media product. This is appropriate given that the brand only launched at the end of April and it was a priority to assess audience response. These feedback mechanisms should continue to be evolved. We understand that plans are in place for an evaluation of the Girl Hub collaboration by the DFID Policy Division team in early 2015, which will include GHE. It would be good to draw on DFIDs evaluation department at an early stage both to draw on expertise within EvD; for EvD to learn from Girl Hub; and to ensure that through, for example, QA processes, the evaluation is embedded and supported by EvD. Progress against expected results:
Indicator March 2013 milestone Total: 4000 Girls: 2000 Boys: 1000 Men: 500 Women: 500 QAS score of 3 Disagree Progress October 2013 Total: 6170 Girls: 3460 Boys: 1016 Men: 852 Women: 842 September 2013 2.5 Agree/Disagree The score for this output remained the same as last quarter at 2.5 Agree/Disagree. There is good progress being made towards full-fledged research taking place in house, but more work is needed to ensure these are girl-led. 154 girl Yegna Ambassadors were involved in data collection in their communities although this was a oneoff effort rather than part of a longer-term strategy for engaging girls in research, and it did not involve girls in the analysis. October 2013 Total: 14 Media Consumption Study Not met/ met/ exceeded Exceeded

# of citizens participating in PM&E and Insights collection

Quality of knowledge products (content, involvement of girls in production and uptake, demand for girl-driven data)

Exceeded

(Measured by internal QAS process)

Total: 5 # of GH E research, insights and evidence (Anticipated to be:

Exceeded

11

publications disseminated

Social exclusion analysis SenseMaker research pilot report 2 X insights reports Yegna baseline study)

Every Last Girl Strategy (Social exclusion analysis) SenseMaker pilot reports (X2) Yegna Connections Game Testing report Drivers of Change analysis Value for Money strategy Spoken Word Workshop insights report Statistical overview by Helen Derbyshire Girl mapping by Helen Derbyshire Round 3 Afar education insights report Film: Making of Yegna Film: Girls Education in Afar

The GHE performance on M&L is strong so far; a score of A+ is given because goals were exceed. However, we recommend that logframe indicators are strengthened to capture progress towards the output of research and girl-centred participatory M&E (PM&E) generate real-time, robust evidence and insight for use by communications, systems, investments and policies. Indicators that define & capture the quality of research products & the uptake of girl-centred methodologies and policy messages are required. Recommendations: GHE should make sure that it is working with DFID on approaches to measuring behaviour change along with other hard to measure issues making use of DFID experience, knowledge and capacity particularly on measuring the impact of GHE work.

Impact Weighting (%):15% Revised since last Annual Review? N Risk: Low Revised since last Annual Review? N

Output 4: Girl Hub Ethiopia organisational development achieved, effective and sustainable
Output 4 score and performance description: B Starting from scratch in a unique and complicated context, GHE has matured out of the initial start-up phase and begun to set up the required systems to become a sustainable project. It is recognised that this has been complex, in that GHE has had to adapt to the special circumstances of being a DFID project team with close links to NF. This must be recognised as a major achievement, and therefore the reviewers have awarded an A, notwithstanding the GH teams own judgement that finalising the physical set up of the office has taken longer than expected. In a short time, GHE has built a full team with a mix of locally-recruited staff (contracted by DFID E) and international staff (contracted through GH Global), in line with the VfM analysis in the Business Case and the strategic workforce plan, jointly agreed by DFID E and NF. There are good systems in place on most areas. For example, key decisions are approved by the Head of Office, DFID E, to ensure accountability and that decisions are

12

taken within delegated authority limits. In addition, GHE reports on a quarterly basis to GH global (NF) and DFID on finance, security, risk, Duty of Care, HR, and policy compliance. The quarterly Qualitative Assessment Scorecard exercise is innovative and strong and appears to contribute to good internal communications and a strong sense of team-working. GH Ethiopia use the exercise systematically to hold themselves to account against their milestones and develop quarterly workplans. Results are triangulated with external partners, who are interviewed by a third party. A great deal of progress has been made and, although some areas of improvement remain, for the most part it is clear that GHE is aware of issues and has a plan in place to address them. On a general level, the responsibilities and obligations of the two parent organisations and GH Global could be improved, for example in terms of defining and putting into practice the specific roles and responsibilities of DFIDE Head of Office and Head of GH Global in setting a strategic direction, and clarifying collaboration with the DFIDE SSDA on a more operational level. A formal agreement on Duty of Care and Security is required; we understand that this is under development by the DFIDE Deputy Head and the GHE Deputy Director. We recommend that Duty of Care and Overseas terms and conditions for overseas-based Girl Hub staff need to be clarified. We are confident that the new GH Global HR manager will support the further improvement of GH HR policies. In terms of procurement, it is clear that GHE has an excellent relationship with a key supplier: Yegna Bet. Representatives of Yegna Bet gave strong feedback on their experience of working with GHE and this appears to represent a good model of combining management consultancy and creative expertise. Yegna Bet expressed some concerns about current systems. For example, they mentioned that they sometimes pay for activities up front; and their perception is that they respond to three clients (GHE, DFIDE and NF). GHE has already addressed some of these concerns GHE finance and operations manager works with YB on a monthly, quarterly basis to ensure their forecasts are realistic and payments reflect the pace of delivery of the work and that accurate invoices are submitted to DFIDE for processing. GHE has been the subject of many audits since inception. An ICAI audit in November 2011 looked at all the GHs. An NAO audit visit to Ethiopia In December 2012 did a high level review of Girl Hub Ethiopia and found that all recommendations made by ICAI (relevant to Ethiopia) and DFID IAD have been implemented or were well in progress. A DFID internal audit of DFID Ethiopia in March 2012 recommended an SCS review of the GHE Business Case before submission. This was done in August 2012, as part of the process of writing the BC. A detailed internal audit of GHE was completed in June 2012, reviewing systems and processes across all operations1. All recommendations against this internal audit were completed in June 2013. Although GHE has very strong leadership, which is driving positive reforms in these areas, there are concerns about sustainability and succession following the Country Directors potential departure. It is essential that a succession plan be developed; the lack of a succession plan is a serious risk to operational delivery. Plans to move to a one embassy model in Ethiopia represent both an opportunity for, and a risk to the current operational delivery model of GHE. It represents an opportunity in that FCO may be strong supporters of GHE given aligned policy agendas (e.g. on sexual violence). However there is a risk that the work that has gone on between GHE and DFIDE into developing a functional operational model may need to be revisited. This will require DFID E leadership to champion the inclusion of both the GHE offer, as well as its operational needs during the process. GHE experience represents an important resource in the GH global family, and in the context of the development of GH 2.0 it is crucial that DFID and NF maximise their investment by making the most of this learning, and that DFIDE and GHE are consulted on these plans and remain aligned with them. This will require a consideration of resources required to service corporate elements of the collaboration, as well as implications for the timeframe of the GHE programme. DFID E should consider GHEs longer term future now, and alongside greater consideration of how GHE can continue
1

Audit Planning & Organisation Background, Payroll, Accounting Records, Financial Controls (Cash Management and Fixed Assets), Month-end Finance Procedures and Reporting, Insurance, Transport/Vehicles, Stock, IT, Human Resources, Project Documentation and Budget Management

13

to contribute to DFIDE strategic priorities, DFIDE should consider an indicative commitment of future funding to ensure GHEs sustainability, should GH 2.0 be delayed or significantly amended. Progress against expected results:
Indicator March 2013 milestone March 2013 2 Agree Scoring criteria for 2 Agree: Permanent office space fully functional; transfer of duties to Yegna CMA complete; ongoing recruitment of FTEs; systems and policies in place; more formalised collaborative arrangements with NF and GH offices Progress Score as of September 2013 2.5 Agree/Disagree The team agreed on a score of 2.5 Agree/Disagree as the office space is yet to be fully functional. In the last quarter, a score of 2 was given as we considered all staff operating in the GH E office is counted as being fully functional. However, to be fully functional there needs to be enough furniture and office facilities for all staff to smoothly carry out their day to day activities and it was felt that with additional staff the office setup has not kept pace with team needs. For this, we have lowered the score to 2.5 in this quarter. Not met/ met/ exceeded Not Met

Quality of GH E infrastructure, HR, governance systems and processes (financial mgmt, VfM, communications strategy, procurement, administration, learning, partnership strategy, collaboration with GH offices and Nike Foundation platforms)

This Output scores B. We recognise the strong progress that has been made, and appreciate the honesty of the self-reflection scoring. Although some areas of concern remain, credible plans are in place to address them. We have, nevertheless, included them as recommendations below, to ensure that they are tracked with other Annual Review recommendations. Recommendations: A plan must be put in place for the succession of the Country Director that takes into account her contribution and her holding of institutional memory. Put in place a service agreement between GHE and DFIDE Define and clarify the responsibilities and obligations of the two parent organisations and GH Global. Develop a contract with DFID E and GHE that defines what GHEs offer is, what DFID Es offer is, what NF offer is, and how the working relationship will function. It is important that any definitions of GH Es offer or the working relationships is well coordinated with GH 2.0. This should include clearly defined roles for key individuals including Head of DFID E; Senior SDA; GHE Country Director; GH General Manager. Clarify overseas terms and conditions for overseas-based Girl Hub staff as part of GH global review of staff terms and conditions The risk allocated to this output should be held at medium because of the reputational and legal risks to both DFID and NF associated with unclear HR, Duty of Care processes as well as uncertainty about sustainability and leadership succession. Work with Yegna Bet to document instances of expenditure in advance & work to resolve them

Impact Weighting (%): 15% Revised since last Annual Review? N

14

Risk: Low Revised since last Annual Review? N

If the project involves more than 4 Outputs please copy the box above and paste below.

Section B: Results and Value for Money.

1. Progress and results 1.1 Has the logframe been updated since last review? N
No. However, review and revision of the logframe was raised in May 2013 with DFID E. GH were advised to wait until this review, so revision has been identified by GHE and DFID E as a priority for the next phase of the project. This is required in order to address the following issues: Link between output and outcome in order to test the GHE theory of change, i.e., will social communications, combined with catalysing and convening, girl-centred research, M&E, plus an effective organisational structure, lead to attitudinal and behaviour change for girls on the ground? Are the right M&E systems in place for tracking & testing this theory of change, and are indicators relevant? Link between outcome and impact outcome statement (attitudes and behaviours changed to support girls voice, value and agency) incorporates effectively 2 steps. Outcome level should focus on attitude or behaviour change. The assumption is that this will lead to increased voice, value and agency for girls but need a clearer one-step outcome statement and then perhaps an intermediate step looking at voice, value and agency. Impact statement is currently very ambitious, which fits with Girl Hub culture but the overall ambition to inspire and transform girls world should perhaps be at a higher level. The logframe review also represents an opportunity to narrow down the sectoral focus (impact level indicators), so that the question becomes, if we focus initially on delay marriage and first pregnancy, does that have a knock-on effect on other sectors?

1.2 Overall Output Score and Description: The provisional overall Output score of A (met expectations). Where expectations have not been met, there are plans are in place for addressing concerns. DFID Annual Review scoring is explicit about making scoring decisions on the basis of what has been achieved, not what the programme is expected to achieve. We are confident that progress to date suggests that the likelihood of achievement in future is good, if plans are implemented as expected, and risks are well managed. This review marks a good moment to lay the foundations for the next phase, by defining more precisely and realistically the programmes logframe and theory of change. 1.3 Direct feedback from beneficiaries Direct feedback from beneficiaries is embedded in programme M&E, through, for example the listener groups, and Sensemaker. The Sensemaker pilot collected stories from 352 respondents, including 178 girls, and facilitated them in interpreting their own stories. The idea is that periodic story collection and analysis will allow the programme to follow behavioural/attitudinal trends. A baseline study has been conducted with over 4,100 respondents using a combined SenseMaker story collection and survey tool. GHE and the Population Council have begun to conduct participatory feedback sessions in Amhara region to share back and invite community members to challenge/validate the preliminary data. Participatory work with girls using the insights approach is also one of the ways in which GHE seeks to

15

influence partners to work in a more girl-centred way. For example, Insights work has been used in the PAGES project as a way to influence programme design and implementation. However the PAGES project provides an interesting example of the tension between drawing out and using girls voices as an influencing tool and the need to take into account other factors. The relative costs (time and resources) of adopting participatory approaches are well documented but little understood. It is important that, GH makes clear its offer to (as well as its demand of) partner organisations, including the costs and benefits of the GH model. It is important for GH and partners to document those benefits, assuming they transpire, as partnerships progress. The challenge for GH is how to adapt its approach to the needs of partners without compromising on its focus on girls. Listener groups provide feedback to the makers of Yegna and help to shape the programme. During the Annual Review, we met a listener group (6 boys, 4 girls and 2 facilitators) in Cherkos (a suburb of Addis Ababa). After listening to the radio drama and the talk show, the facilitators led a discussion on key themes, guided by an interview schedule. It was clear that the listeners found the programmes engaging, and the discussion was animated, focusing on both the content of the programme, and in particular gender relations portrayed in the programme. The listeners also (to a lesser degree) reflected on how key themes affected the listeners in their daily lives. For example, one of the themes of the drama was on sexual harassment of one of the main characters by a teacher at school; one of the participants remembered a teacher who used to write his phone number on girls exercise books. The group was well facilitated, although it will be important for the GH team to provide on-going feedback to the facilitators on supporting girls participation in the discussion. Although this visit provided only a very brief snapshot from which we do not wish to generalise, it raised a question for us on whether the function of the listener group is simply to provide feedback to the makers of Yegna or whether it provides an opportunity on which to build future programming. The Listener Groups are, effectively, community-based youth organisations and should be seen as an opportunity for GH to expand their impact, for example through signposting to existing services or direct interventions. We did not have the opportunity on this occasion to talk to girls who were not directly involved in Listener Groups.

1.4 Summary of overall progress

Overall, there has been mixed progress, with some very promising results, a lot of lessons learned, as well as some areas for development. We conclude that GHE represents a significant opportunity to work for Ethiopian girls as well as an opportunity to learn from, test, refine and improve the GH model. Given the significant investment in GHE to date, and promising signs, it is important that the programme is now given the opportunity to deliver results. However, a number of course corrections are recommended in order to ensure that lessons are learned from the start-up phase. In addition, organisational relationships and governance arrangements within and between DFIDE, GHE, GH Global and NF require revisiting in order to ensure that they are fit for purpose for all involved. Key achievements include the launch of Yegna, GHEs social communications platform; progress on developing key influencing/catalysing partnerships; contributions to the evidence base; development and testing of girl-centred research methodologies and an office set-up and systems that are wellresourced and with a plan for ensuring required systems are in place, if they are not already. In terms of progress against milestones and expected results, the programme largely had mixed progress against existing met targets, with the exception of listener milestone (Output 1). The logframe however requires updating as recommended above. It should be noted also that the GH programme is different from many DFID programmes in its portfolio-based approach to partnerships; the fact that it is led by a complex model of behaviour change; and in its high level of internal communication with parent organisations. These elements and innovations resonate with priorities expressed by the recent DFID Internal End-to-End review team which looked at ways of increasing efficiency and innovation in DFID making public-private partnerships work; and using adaptive/complexity-theory for better programming. These elements of GH Es work are not explicitly captured in the logframe, but nonetheless nurturing a different approach to development represents a key element of GHEs potential for success. As part of revisiting governance arrangements for the programme, consideration

16

should be given to expectations from DFIDE and NF that GHE staff contribute to the wider agenda framing the DFID/NF partnership and this work captured in workplans and performance management processes. It is important to note that GHE has critics; this must be acknowledged and addressed or refuted, whether or not they are justified. For example, some (DFID E and partners) feel that the Yegna product is too expensive and does not represent good VFM. A recent UK media story have also questioned the value for money of Yegna, in context of opposition to protecting UK development spending targets. This is a risk that was anticipated in the GHE Business Case and is tackled in the VFM section below. From within DFID E, there is a perception that GHE either communicates too little or too much; doesnt actually do anything or produce any concrete results; takes up too much senior level time; is over-resourced. The reviewers heard for themselves a number of scathing comments about GH, including personal and unfounded comments relating to GH staff, which were not acceptable. GH will need to address these perceptions through a careful internal and external communications strategy and a more clearly defined relationship with DFID E. 1.5 Key challenges Girl Hub Ethiopia has faced challenges, as is to be expected with a start-up programme. This is the first GHE Annual Review. Challenges were identified by the GHE team in their narrative report as follows: Delays to baseline study due to poor supplier performance Unrealistic listener figures as well as radio transmission problems have impacted on reach of Yegna Delay to implementation caused by new DFID approval procedures

Interviews with GH staff, DFID staff, partners and other stakeholders also led to the following challenges being identified: GHE, as a team of DFID E, require support from DFID E: this remains problematic, especially IT support, protocol, vehicles and Duty of Care. Cultural differences between DFID E and GHE can sometimes cause tensions leading to GH feeling misunderstood and marginalised; while DFID E, feel that they are being sold to, without being clear about the benefits of GH Slow pace of change when working with partner organisations GH HR not set up for overseas postings

1.6 Annual Outcome Assessment

The project outcome is attitudes and behaviours changed to support girls voice, value and agency. According to the logframe, baselines and milestones are due to be set by December 2012. The baseline was substantially delayed due to management of a poorly performing supplier but has been completed and analytical work is currently underway to set future milestones. The supplier for the baseline survey was selected through a pre-qualified DFID service provider. The relationship between GH E and the provider broke down since the provider was not ultimately able to deliver against the Terms of Reference within the agreed budget envelope. GH E worked closely with DFIDs Procurement Department to terminate the contract and made savings of 54,000, but at the cost of a delay to baseline implementation. The reviewers have seen the draft listenership survey. According to this survey, only 7.1% of survey respondents had seen the Yegna logo and 15% had heard Abet (music). However, only of respondents ever listen to the radio. One quarter of listeners listen to the drama and the talk show; 40% of listeners to the drama only, although some do not differentiate between the two programmes. Out of those who have listened to Yegna, close to 40% reported that they listen to Yegna each week, demonstrating an impressive degree of loyalty for a very new brand and radio program. Results of the listenership survey related to Outcome indicators are

17

56.1% girls said that it was slightly or much easier to talk about life issues with their family as a result of listening to Yegna (44.5% for talking to people outside the family); 66.2% boys said that listening to Yegna made them think differently about issues in girls lives 63.2% of all respondents said they were thinking differently about issues in girls lives 21.6% of listeners said they had taken action or changed their behaviour as a result of listening to Yegna2

The outcome statement for Yegna in the logframe needs to be revised to ensure it is tight enough, and that the indicators measure it adequately. Assessment of theory of change assumptions (taken from GHE results chain) GHE can understand and adapt to the evolving local political economy: not clear whether and how GHE is doing this although in theory the QAS process should capture this could perhaps be pulled out more strongly in a question about whether the PE context has changed Branded communications and outreach amplify behaviour and attitude changes: this is what is being tested in output 1 so an output rather than an assumption It is possible to reach the most vulnerable girls: articulate what are the barriers to reaching them how addressed? The strategic partnership encourages innovation and generates excitement about the Girl Effect: this is an aspiration, rather than an assumption. How is innovation defined and how is it planned for and monitored? A dedicated Girl Hub can align resources and actors to deliver for girls: how is this being tested? Decision makers are convinced by girl-driven data and evidence: again, a strategy rather than an assumption Unleashing girls potential can break the cycle of intergenerational poverty: isnt this the ToC writ large?

As part of the planned logframe review, it will be important to review assumptions and identify which are to be tested through the M&L strategy and how.

2. Costs and timescale 2.1 Is the project on-track against financial forecasts: Yes
The double financial system (NF and DFID) is a cause for potential concern the Annual Review should review all finances, not just DFID contributions. When compared with indicative figures in the Business Case, Output 1 appears to have overspent by 43%. There are two reasons for this revisions to budget and additional investment by Nike Foundation. There was an agreement with DFID Procurement Group to issue the contract for Output 1 based on a forecast for the first three months of implementation. In the first months of implementation, Girl Hub undertook a budget review of Output 1, for the remaining 33 months of the Yegnabet contract, again with the agreement of DFIDs Procurement Group. As a result, a more accurate costing and forecast for the whole of Output 1 implementation was produced by month three of implementation. The financial performance of Output One against the revised budget has been good: GH forecasting record with DFID over the last 6 months is 95%. 220,000 was moved forward into Year 1, which represents a change in the indicative expenditure set out in the BC of 19%, not an overspend.
2

The reviewers saw an early draft of the Listenership Survey these figures are taken from a subsequent draft not seen by the reviewers.

18

NF provided a one-off additional input of funds for Yegna PR, launch and marketing amounting to 630,000. According to GHE, the reason for the increased spend on launch was in order to drive listenership, and the decision was approved jointly by DFIDE and NF.

2.2 Key cost drivers The major costs identified in the BC were staff and operational costs. A second set of cost drivers
relates to the CMA [Yegnabet], various research, monitoring and learning, and professional fees for services. It was envisaged that cost would be driven by the availability of contractors with the required skill sets; rising inflation costs for general living costs and goods, as well as potential foreign exchange loss. Major cost drivers have not changed and as set out in the BC, the largest spend has been on Output 1. The VfM strategy was implemented in the design and delivery of the launch by ensuring cost economy by driving down unit costs using NF procurement processes and ensuring cost efficiency by considering alternative means of launching the Yegna brand and radio programmes. However, it is not clear why this expenditure was not budgeted for and it will be important for the Reach Strategy to consider marketing activities and their contribution to VfM for the programme.

2.3 Is the project on-track against original timescale: Y

There is no indication that the project is not on-track. Expenditure in year 1 combined with the need to review the mix of channels to reach the numbers of people set out in the logframe will have implications for later years.

3. Evidence and Evaluation 3.1 Assess any changes in evidence and implications for the project
The GHE theory of change is an integrated strategy including social norm changes, catalysing scale programmes delivering services, assets and information to girls through partners, and building the evidence that underpins design, implementation and monitoring, with girls voice at the heart. Currently, GHE projects are progressing in testing the assumptions of this theory, especially how it is possible to reach the poorest and most vulnerable girls, and whether branded social communications can accelerate social norm change. It is too early to test the evidence that GHE has produced results (beyond the progress milestones in the logframe). GHE M&L processes are in place and systematically gathering evidence, enabling the team to adjust the programme in response to new evidence, such as whether radio or peer to peer outreach is more effective in reaching the target populations. Regarding GHE influencing and catalysing work, the assumption that there is an appetite for better girlcentred programming has been demonstrated. What remains to be seen, according to the GHE team and at least one partner organisation, is whether GHE can effectively motivate partners to provide it. GHE have systems in place, especially the QAS process, to monitor their impact in this area and adjust resources accordingly. The introduction of a Portfolio Risk method of resources allocation is an opportunity to further hone their influencing work. What is clear is that there is a greater demand from partners to work with GHE than can be met. GHE plans to capitalise on this by making strategic choices on which partnerships to enter into; and has learned to plan with partners to focus GHs efforts on adding value and going with the grain of existing programmes and processes. The Smart Economics work is a strong approach to providing an evidence base for engagement with partners. It is an encouraging trend to see GHE moving beyond advocacy toward motivating investment in girls, from making the case for greater investment in girls to strong messages on where and how that investment should be deployed. This report also contributes evidence to the core

19

assumption that investing in girls can break the intergenerational transmission of poverty.

3.2 Where an evaluation is planned what progress has been made?

Girl Hub London will commission an independent evaluation of the Girl Hub collaboration, which will include GH Ethiopia and will be used to determine whether expected results have been met from the collaboration as a whole. The evaluation will draw on data collection already planned in Ethiopia and other countries and will provide evidence related to the GH theory of change. It is recommended that planning for this evaluation is undertaken as a priority, given the time required to contract, design and implement such studies; and that evaluation planning is used as a means to work with GH partners to ensure that they have the opportunity to input into the design.

4. Risk
4.1 Output Risk Rating: Low/Medium

4.2 Assessment of the risk level

The overall risk rating of the project is medium. This reflects the fact that although investment is relatively low, significant risks to the achievement of results exist, despite the fact that the project is well placed to start to produce results. This review recommends that risks identified in the logframe be reviewed as part of the GH E quarterly risk management process. The achievement of results is based on a set of assumptions about not just the programme, but also the operating environment, the parent organisations and a range of relationships. Although we are satisfied that positive results are, on the whole, likely to be achieved, the complexity of the change process means that they are difficult to capture, and the challenges of the start-up phase as well as challenges of working with and through partners means that implementation is proceeding at a slower rate than had been envisaged. According to the BC there is a GHE risk register (not seen) which is summarised as follows: Risk Strategic Political risk that Girl Hub is seen as inappropriate in Ethiopian context. Flawed theory of change or slow pace of partnerships means Girl Hub Ethiopia fails to deliver agreed results. Deployment - Issues in programme delivery impede Girl Hub Ethiopias ability to deliver results Probability Moderate Impact High Mitigation strategy Independent Advisory Group. Stakeholder mapping and influencing strategy completed. HMG/DFID political capital mobilised. Active portfolio management approach with scale partners. Implementation of QAS process. Annual planning is integrated with global planning, milestones are reviewed on a regular basis, Yegna business plan is signed off by Sept 2012, monitoring baselines in place by October 2012 Internal audit report recommendations are Risk following mitigation Moderate

Moderate

High

Moderate

Funding - Funding structure and

Moderate

High

Moderate

20

processes to not adequately support delivery of results Reputational - Girl Hub strategy and/or activities negatively impact stakeholders Operational Operational challenges impede Girl Hub Ethiopias ability to execute against short term and long term plans

implemented. Value for Money strategy. Moderate/low High Child protection strategy in place and audited, communication and advocacy strategy in place. Office move is well implemented. Milestone based contract developed with Yegna Creative Management Agency, full staff complement are recruited. Moderate

Low

Moderate

Low

We recommend that at the next planned quarterly review of the risk matrix be reviewed, the following risks that are not currently captured are included: The risk of negative media coverage negatively impacting on GHE ability to deliver results NF/DFID relationship GH/DFID E relationship Succession planning Evidence produced by GHE is not convincing to policy-makers Impact of GCO on operational delivery model Output 1) Branded social communications products and outreach strategies are activated and inspiring 2) GHE catalytic and convening role established and effective 3) Research and Girl-centred M&E generate real time, robust evidence and insight for use by communications, systems, investments and policies. 4) Girl Hub Ethiopia organisational development achieved, effective and sustainable Risk rating Medium Reason Branded communications and outreach amplify behaviour and attitude changes (assumption) GHE can understand and adapt to the evolving local political economy (assumption) Decision makers are convinced by girldriven data and evidence (assumption)

Medium

Low

Low

A dedicated GH can align resources and actors to deliver results for girls. Strategic partnership encourages innovation and generates excitement about the Girl Effect (assumption)

We recommend upgrading the risk of Output 4 to medium, due to the risk from unclear partnership arrangement between DFID E and GH; risks related to the Global DFID/NF partnership; as well as risks from lack of clarity on key systems, especially HR and Duty of Care.

4.3 Risk of funds not being used as intended No cases of misappropriation of GHE funds are reported by the team. Financial expenditure for DFID funding goes through the DFID ARIES system so all DFID systems to monitor correct use of funds and the application of ADAMANT principles. According to the VFM report, GHE audits Yegna Bet invoices, conducts quarterly asset spot checks and reviews finance & procurement performance on a regular basis. Girl Hub Ethiopia is considered a low risk programme for fraud and corruption according to DFID Es

21

anti-corruption and counter-fraud strategy (score 1.25); it is important that the programme despite being relatively low investment compared to other DFID E programmes is compliant with key elements of the DFID E Anti-corruption workplan including training/updating of staff skills and audit requirements. 4.4 Climate and Environment Risk According to the BC, the programme has minimal impact on climate change. Delivering these options was expected to require around 36 air flights (including within Africa, from UK and US). One question is whether this figure requires revision.

5. Value for Money 5. 1 Performance on VfM measures

GHE has a VFM strategy which outlines key VFM questions that GHE seeks to answer, states key VfM priorities and defines activities and measures progress. This strategy which is an excellent initiative rightly reflects the critical importance of pushing and measuring VfM in GHE as it is a new approach seeking to build an evidence base on how best to deliver results for girls, with expenditure on a range of products that are unusual for a DFID programme, and therefore rightly subject to a high level of scrutiny. As the VFM report points out, the GHE portfolio is difficult to measure in terms of VFM. However, we believe that in addition to the approach already developed, it would be valid to make comparisons with other social communications programmes (with caveats). It will also be useful to develop a narrative about internal VfM (i.e. how and why girls and other stakeholders value GHE activities). The VFM questions that GHE poses are: Is investing in girls better Value for Money than alternative Aid Allocations? Does working with, and learning from the private sector lead support the development of more cost effective, scalable social communications approaches? Participatory M&E is a cost effective way to increase girls ownership of programming, gain insights to inform programme implementation and generate data. According to the VFM report, current priority activities that support VfM include: Seeking cost economies through procurement policies and identification and monitoring of unit costs Learning about cost efficiency of the delivery mechanism and beginning to measure the relationship between inputs and outputs On economies, the VFM report claims that savings of 508,000 have been made over the past 12 months through, for example, contract negotiations by DFIDE contracts team, robust supplier management leading to discontinuing the contract for the baseline study supplier following budget increase demands; hiring an independent recruitment consultant to manage the recruitment of local staff leading to recruitment costs per candidate of 350% less than DFIDE. The report claims that DFIDE has made changes to recruitment practices as a result. On efficiency, the following unit costs have been collected for social communications activities: 0.15 per view of Abet video (not restricted to Ethiopia check) 0.50 per person reached by Yegna Ambassador 33 per Amhara Women Association member reached 40.80 per person engaged in M&E (cf 110 estimated in BC) 19.45 per learning symposium participant. 4.76 per person listening to Yegna Series 1

22

VFM activities are currently focused primarily on cost economies which is an important element of the strategy; it is important at the same time to start building evidence on, and developing a narrative, in answer to the three main VfM questions above. For example, a response to the question on whether working with, and learning from the private sector is a cost effective way to develop social communications and branding products could be under development through, for example, comparisons with social communications initiatives in Ethiopia that do not work with the private sector. Similarly, a response to the participatory M&E question could be under way by systematically comparing costs and results of other approaches to researching girls lives in Ethiopia. Recommendations: Review and verify whether VfM strategy objectives will enable a robust response to the 3 main questions posed above? Integrate VFM questions into M&L strategy. Integrate VfM assumptions explicitly into the ToC and logframe. Start to tell the GHE VFM story now.

5.2 Commercial Improvement and Value for Money

GHEs largest contract is with Yegna Bet, the consortium contracted to deliver Yegna. This contract was awarded following a competitive bidding process, using DFID procurement processes. Our understanding is that there was a strong market response to the call for proposals, despite the fact that the supplier was expected to deliver social communications products of an unprecedented quality standard for Ethiopia. The consortium has worked closely with GHE staff, in particular the Brand Manager, as well as external consultants, GH global, Nike Foundation and Nike Inc staff. According to the VFM report, GHE has worked hard through negotiation to achieve savings, both before and after signing the contract, citing savings of 34,000 made by reviewing all expenditure. As discussed above, the reason for what looks like an overspend in Year 1, according to GHE, is that the Business Case Year 1 figure was based on indicative costings. PrG approved issuing the contract based on the first three months workplan and indicative costing. Work planning and budgeting in the first two months of the contract explain the variance in spend. The Yegna budget overall remains within the budget set out in the BC. NF chose to make an additional investment which will affect VFM calculations in the longer term and additional investment to date is reflected in VFM calculations above. In the short term, GHE confirms that the use of those funds followed the same VFM steps as all contracts. Some questions and concerns were raised by respondents about the contract with Yegna Bet. These include: Claims that partner frequently has to advance money from own funds Contract does not incentivise capacity building and effective planning and resourcing of activities Effectiveness of internal measures to verify Yegna Bet invoices How does YB level of spending compare with other social communications work and verify that the level of quality being produced is indeed required to achieve the desired results. Similarly, whether the complete package of marketing products is required to reach the target population and what proportion of output is internal communications or communication directed at the international development community more broadly. Recommendation: Review marketing strategy in relation to budget. Ensure internal communications outputs are captured by the GHE logframe.

5.3 Role of project partners

Yegna Bet provides a high quality product and service. The Yegna products are unique and recognisable in the Ethiopian market. The Yegna Bet partners have a high level of personal ownership, pride and devotion to the project and this is evident in the product and in the decision to rename the consortium to reflect their commitment. The result has been a highly productive partnership with a skilled and professional team that is motivated to produce high quality outputs. However, questions remain as the sustainability of Yegna ; whether Yegna Bet is indeed successful in building

23

local capacity, or whether the work (and consultancy fees) is limited to a small group. There is also the concern that the package of Yegna products is more extensive than necessary and that therefore the full expense of these products is not warranted. Further, the contract with Yegna Bet was designed to cover four products (music, radio drama and talk show and Yegna Connections lifeskills programme). Launch events (marketing and PR) and the two promotional concerts were outwith the original contract.

5.4 Does the project still represent Value for Money :

Based on evidence and indicators from the VfM strategy, we think that GHE has in place a robust VFM strategy, which will need updating based on Reach Strategy recommendations and other comments above.

5.5 If not, what action will you take?

Yegna Bet requires monitoring from a VfM perspective; we agree with the GHE proposal to revisit the VfM strategy in the light of the Reach Strategy.

6. Conditionality 6.1 Update on specific conditions

n/a

7. Conclusions and actions

Overall, GHE has made significant progress, is starting to demonstrate achievements, and is well positioned to start to deliver results. Implemenataton is at an early stage, the project works within a complex organisational framework, and the start-up phase has been challenging. The programme is well-led and has credible plans for the next phase. GHE is a high profile programme in relation to the size of investment and it therefore carries disproportionate potential, as well as level of risk. GHE fills a clear niche in Ethiopia in a way that no

other organisation could. DFID could not completely absorb GHE programming and NF could not provide GH without DFID. The overall recommendation is for GHE to focus its offering and progress where it has the institutional capacity to deliver results and value for money within budget. GHE makes a strong argument for Output 1 (Social Communications) and Output 2 (catalysing and convening) as parts of a unique offering and it is clear that strong progress has been made. It will be important to retain a strong focus on delivering and communicating results. It is important to focus on using the products of Output 1 to deliver change and ensuring that the change can be measured. In addition, the cost/reach/impact balance must be reassessed. Output 3 (monitoring and learning) is adding to the evidence base. It is important that the research strategy is strategic, building on an existing body of research, will result in changes in policy and practice, and, ultimately, transform the lives of girls. Overarching VfM questions need to be integrated into the M&L strategy, as well as the implementation of the GH global collaboration evaluation plans at an early stage. In addition, the M&L strategy needs to draw on the opportunities and resources offered by DFID EVD, as well as contributing learning to 24

DFID EVD. Output 4 (organisational development) has provided challenges; this is not surprising in the start-up phase of a new programme with an untested delivery mechanism. However, GHE is attempting to put into practice a number of elements that DFID aspires to, including innovation, learning by doing, adaptive programming and portfolio management. GHE has been open about challenges and it is important that lessons are learned from this experience, which could be applied in other contexts. GHE is recording these lessons through their quarterly QAS process. It is of critical importance to the programme to ensure robust and consistent approaches to outstanding areas of concern, in particular Duty of Care and Human Resources policies. Finally, overall, the position of GHE in relation to DFID E and NF is not clear to partners, or indeed to DFID E staff. Although there is clear appetite for GHEs general offering of expertise and influence on girls, the offering is not clear. There is a niche for GHE in the Ethiopian context of restrictions on NGOs and CSOs working on gender, and this is an opportunity for Girl Hub, perhaps to act as a wider hub or resource for organisations working with and for girls in Ethiopia. It is clear that as GHE evolves, there will be trade-offs to make between impact on the development community, (mainstreaming girl-centred approaches in scale programmes which reach into the Ethiopian population) and increasing the number of new programmes with an exclusive focus on the most vulnerable girls. Recommendations have been kept to a minimum to reflect the fact that logframe revision, strategic (end state) planning and Yegna Reach Strategy, as well as other activities, are already scheduled. In addition to the output level recommendations above, we recommend that the 2nd Annual Review should be jointly conducted by DFID and NF. This will ensure that the review is part of joint programme management and planning by DFID and NF.

8. Review Process
The review was conducted by the DFID DRC Social Development Adviser, and an Entry Scheme Social Development Adviser from the HQ Gender Team. The Review team visited Addis from 31-3 November. This included interviews with GH, DFID GH London representatives, as well as representatives of partner organisations and a visit to a Listener Group in a suburb of Addis Ababa. Please note this was a light touch review based on discussions with a limited number of stakeholders. A review of key documents including logframe, progress and financial reports produced in the review period, research reports and other key documents.

25