Mid-Term Review of The Syria Needs Analysis Project

Syria Needs Analysis Project (SNAP) External midterm review
Andy Featherstone, Independent consultant, October 2013
External mid-term review of the Syria Needs Analysis Project
Syria Needs Analysis Project (SNAP): External mid-term review Andy Featherstone, October 2013 EXECUTIVE SUMMARY
The Syria Needs Analysis Project SNAP is a one-year project delivered through a partnership between ACAPS and MapAction running from December 2012 December 2013. SNAP aims to link with humanitarian actors working from operational hubs to facilitate a comprehensive independent analysis of the humanitarian situation in Syria and neighbouring countries, including conducting rapid needs assessments. In so doing, SNAP aims to equip the humanitarian community with information that would enable better humanitarian assistance to the populations affected by the Syria crisis. Purpose of the review and methodology The purpose of the review is to obtain an independent evaluation of the performance of the SNAP project against the targets outlined in the projects logical framework analysis, using the evaluation criteria of relevance, effectiveness, efficiency, connectedness, and coherence. The methodology for the review consisted of the following activities: Review of SNAP project proposal, reports and other relevant background material. Interviews with Assessment Capacities Project (ACAPS) Project Management Team in Geneva, MapAction management, key donors, United Nations (UN) agencies and Non-governmental Organisations (NGOs) headquarters. A 12-day fieldtrip which included interviews with SNAP staff, donors, UN agencies and NGOs in Jordan, Lebanon and Turkey An online survey of users of SNAP outputs (see annex 3 for the survey results).
The review was commissioned by the ACAPS Board and was managed by a steering committee (SC) composed of representatives from the ACAPS board, donor and partner representatives. Overall findings Over the period under review, SNAP has offered significant value to the humanitarian community in strengthening the targeting of assistance and in making an important contribution to a shared situation awareness. That is has achieved this in the context of a complex regional crisis and a fragmented humanitarian architecture is testament to the professionalism and tenacity of the SNAP team. Relevance The relevance of SNAP has stemmed from its ability to fill critical gaps in the information and analysis of the humanitarian community. The SNAP-authored Regional Analysis for Syria (RAS) is widely credited to have been the first coherent regional analysis, while the SNAP-supported Joint Rapid Assessment of Northern Syria (J-RANS) remains the only coordinated needs assessments that have been conducted in northern Syria. While the thematic reports and scenario documents tend to have less profile they are credited with providing much-needed analysis and in offering a common language on key humanitarian issues. SNAPs agility in identifying and responding to the needs of the humanitarian community has gone a significant way to bringing a level of coherence to a very complex and fragmented humanitarian context.
ii
Connectedness There was a general level of satisfaction expressed by review participants about the complementarity of SNAPs information and analysis with other providers. While in the months after its deployment comparable products were few in number, with time the number and variety has increased and while duplication is still relatively small there is a need to look proactively at ways in which SNAP could best complement the work of others. This is exemplified by the steps taken to develop a follow-up needs assessment in northern Syria. Coordinated by the Office for the Coordination of Humanitarian Affairs (OCHA) and IMPACT, SNAP is participating in this initiative and in so doing has taken an important step in strengthening collaboration. Coherence Despite SNAPs approach to expanding by seizing opportunities and responding to needs its information and analysis products are coherent and all make a contribution to its core purpose. That said a vision and set of principles that articulate SNAPs distinctive competencies and which can help future-proof it against loss of institutional memory would be a wise investment. Effectiveness SNAP has made impressive progress against the outputs in its log frame and is on track to meet expectations by the current close of the project. Several of its early publications were considered extremely timely and have laid a foundation for SNAP to build on. Interviews with NGOs, UN agencies, donors at project, country, regional and headquarters level provided a mosaic of uses for SNAPs information and analysis which was considered by most participants who participated in the review to be of high quality which helped foster trust and credibility with SNAPs key clients. It was more difficult to determine the perceptions of national and regional organisations or the Arabicspeaking world more broadly and there would be value in SNAP reflecting on the whether these are targets and if so how best to tailor its products to better include them. An exploration of perceptions of the independence of SNAPs analysis revealed that a majority feel that it is free from significant political bias which is considered of key importance to its users, particularly in an environment where access constraints and operational security concerns have served to degrade perceptions of the independence of information. Efficiency In establishing SNAP as an operational entity in Lebanon, Turkey and Jordan, ACAPS and MapAction have both worked outside their organisational comfort zones but have made some wise decisions. While a comparison of alternative models suggests that SNAP is comparatively resource-heavy, analysis suggests that it would have been impossible to achieve the same added value at distance, via a series of secondments or through a more modest deployment strategy. An analysis of SNAPs distinctive competencies relative to other information and analysis providers highlights its comprehensive regional analysis, its independence from a fragmented humanitarian architecture and its agility in identifying and seizing opportunities as being unique. A presence across the region has undoubtedly allowed SNAP to develop links, strengthen trust and build a comprehensive regional analysis and in so doing provides a compelling justification for the decision to adopt the model. Value for money (VfM) While it is methodologically complex to assign a quantitative value to an intangible outcome (a shared situational analysis), an analysis of SNAPs VfM suggests that it has striven to identify cost drivers and has identified means of strengthening the economy of the project albeit learning some important lessons about resourcing for the future. While the high cost of the SNAP model in comparison with the alternatives could lead one to question the efficiency of the project, SNAPs distinctive competencies in comparison to other information and analysis providers and its
iii
effectiveness in delivering against its objectives and making progress towards its outcome provides a compelling justification for an operational deployment. This has provided an important foundation for its success which would likely have been compromised by having a more limited deployment in the region, by adopting a model of agency secondments or by seeking to deliver the project remotely from Switzerland (ACAPS) and the UK (MapAction). This analysis supports a conclusion that the SNAP project does indeed offer value for money.
iv
Table of Contents Executive summary Contents Acronyms SNAP: Regional Analysis of Syria, 26 September 2013 1. The purpose of the review and methodology 1.1 The purpose of the review 1.2 Methodology 1.3 Limitations 2. The context of the humanitarian situation in Syria and SNAP 2.1 Context of humanitarian situation in Syria 2.2 Background to the Syria Needs Analysis Project 3. Relevance 4. Connectedness 4.1 SNAP information and analysis complementary to or duplicating the work of others? 5. Coherence 6. Effectiveness 6.1 Progress of SNAP against its objectives 6.2 An analysis of risks and assumptions 6.3 SNAPs contribution to strengthening targeting and promoting a shared situational analysis 6.4 Timeliness 6.5 The value of independent analysis in Syria 7. Efficiency 7.1 Distinctive competency analysis 7.2 Alternative means by which SNAP could have achieved its outcomes 7.3 Towards a judgment on the efficiency of SNAP 8. Value for money 8.1 Towards a judgment on value for money 9. Recommendations for the future of SNAP 9.1 Challenges for the future: Maintaining SNAPs relevance in a changing context 9.2 Reflections on the added value of the SNAP model for crises elsewhere 9.3 Lessons learned 10. Conclusion and summary of recommendations Annexes Annex 1: Review participants Annex 2: Fieldwork schedule Annex 3: Online survey Annex 4: ToR for the review i iv v vi 1 1 1 1 1 1 3 6 10 12 14 14 14 16 17 19 20 22 22 24 25 26 28 28 30 31 33 33
37 39 40 45
Acronyms ACAPS ACU AWG DFID DNA HC HCT HIU ICG IDMC IDP JHA J-RANS J-RASS NGO NRC OCHA OFDA RAS RRRP SDR SHARP STIMA UN UNHCR VfM WFP Assessment Capacities Project Assistance Coordination Unit Assessment Working Group Department for International Development Disaster needs Analysis Humanitarian Coordinator Humanitarian Country Team Humanitarian Information Unit International Crisis Group Internal Displacement Monitoring Centre Internally Displaced Person Joint Humanitarian Assessment Joint Rapid Analysis of Northern Syria Joint Rapid Analysis of Southern Syria Non-governmental NGO Norwegian Refugee Council Office for the Coordination of Humanitarian Affairs Office for U.S Foreign Disaster Assistance Regional Analysis for Syria Regional Refugee Response Plan Secondary Data Review Syria Humanitarian Assistance Response Plan Syria/Turkey Information Management Task Force United Nations united Nations High Commissioner for Refugees Value for Money World Food Programme
External mid-term review of Syria Needs Analysis Project
vi
SNAP: Regional Analysis of Syria, 26 September 20131
Source: SNAP (2013) Regional Analysis for Syria, Overview, 26 September 2013
Syria Needs Analysis Project (SNAP): External mid-term review Andy Featherstone, October 2013
1. The purpose of the review and methodology 1.1 The purpose of the review The goal of the review is to obtain an independent evaluation of the performance of the SNAP project against the targets outlined in the projects logical framework analysis, using the evaluation criteria of relevance, effectiveness, efficiency, connectedness, and coherence. At the outcome level, the purpose of SNAP is that populations affected by the Syria crisis benefit from improved targeted response provided by humanitarian stakeholders operating with a shared situation awareness and the focus of the review will be on assessing the progress made towards achieving this. The review covers the period from the commencement of the SNAP project in December 2012 until end-August 2013. Taking into account the innovative nature of the SNAP, it also seeks to identify lessons learned with a view to identifying the role that a project such as SNAP can play in other contexts. The review was commissioned by the ACAPS Board and was managed by a steering committee (SC) composed of representatives from the ACAPS board, donor and partner representatives. 1.2 Methodology The methodology for the review consisted of the following activities: Review of SNAP project proposal, reports and other relevant background material. Interviews with ACAPS Project Management Team in Geneva, MapAction management, key donors, UN agencies and NGOs headquarters. A 12-day fieldtrip which included interviews with SNAP staff, donors, UN agencies and NGOs in Jordan, Lebanon and Turkey An online survey of users of SNAP outputs (see annex 3 for the survey results).
1.3 Limitations While the fieldwork, travel to Geneva and follow-up telephone interviews permitted discussions with a representative sample of SNAPs users, the exception to this was in Syria itself which it was not possible to travel to. Despite efforts to contact and interview informants from 9 agencies, no interviews were conducted and only 2 of the 94 participants of the online survey worked in Syria (both for UN agencies). It is unclear why participation from within Syria was so poor although it is possibly linked to sensitivities concerned with sharing information over the telephone. While this is a weakness of the evaluation, there was good participation from agencies and staff working in Syria cross-border (10 agencies that participated in the review were working in this way) and there were a far greater number who were working with Syrian refugees in neighbouring countries. Discussions were also held with headquarters staff who managed operations within Syria and the surrounding countries.
2. The context of the humanitarian situation in Syria and SNAP 2.1 Context of the humanitarian situation in Syria The Syria crisis is complex and has caused extensive human suffering. An estimated 6.8 million people in Syria, almost one-third of the population, have been affected by the conflict, including
more than 4.25 million displaced inside Syria.2 On September 3, the UN High Commissioner for Refugees (UNHCR) announced that the number of Syrians displaced as refugees exceeded 2 million, with 97% fleeing to countries in the immediate surrounding region, including Turkey, Lebanon, Jordan, Iraq, Egypt, and other parts of North Africa.3 At the end of May 2012 an agreement was reached between the Syrian Government and representatives of the United Nations which allowed access by 8 UN agencies and a handful of NGOs to key locations for the delivery of assistance.4 By August 2013 the number of NGOs working inside had risen to 125 although significant gaps in the delivery of life-saving assistance continue to exist prompting the Emergency Relief Coordinator and the UN High Commissioner for Human Rights to jointly release a statement urging all parties to respect their obligations under international human rights and humanitarian laws to protect civilians and to allow neutral, impartial humanitarian organizations safe access to all people in need, wherever they are in Syria.6 Outside of the country, a growing number of organisations have responded to the challenges of delivering humanitarian assistance from within the country by supporting its delivery from neighbouring countries. In addition to struggling to meet the humanitarian crisis within Syria, huge needs exist in neighbouring countries where governments assisted by the humanitarian community are seeking to provide assistance to vast numbers of refugees. As of 15-September 2013, UNHCR estimated that there were 2.038 million refugees.7 Of this number, 739,000 were being hosted by Lebanon, 519,676 were in Jordan and 463,885 were in Turkey.8 A humanitarian crisis of such significant proportions has required huge resources and while donors have been generous, there has been a consistent funding shortfall. Two appeals were launched to meet needs inside and outside of Syria; at the time the review was conducted the 2013 Syria Humanitarian Assistance Response Plan (SHARP) with a requirement of $1.4 billion was 46.7% funded;9 the 2013 Syria Regional Refugee Response Plan (RRRP) with a requirement of $2.981 billion was 42.8% funded.10 The scenarios for the Syria conflict remain complex and agencies operating within Syria and the surrounding countries face a continuing struggle to prepare response mechanisms and to allocate
Source: OCHA Syria Humanitarian Bulletin, Issue 33, 27 August 9 September 2013 (internet). Available at http://reliefweb.int/sites/reliefweb.int/files/resources/Syria%20Humanitarian%20Bulletin%20Issue%20No%20 33%20%281%29.pdf 3 Source: UNHCR press release: 2 million Syrians are refugees, September 3 2013 (internet). Available at http://data.unhcr.org/syrianrefugees/documents.php?page=2&view=grid&Country%5B%5D=122 4 th Source: Humanitarian Policy Group (2013) Syria crisis: The humanitarian response, 15 June 2012, Roundtable, London Overseas Development Institute (internet). Available at http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/events-documents/4907.pdf 5 UN OCHA, Syrian Arab Republic: Humanitarian Presence: International NGOs (as of 7 August 2013) 6 United Nations press release, UN humanitarian and human rights chiefs urge immediate safe passage for civilians and aid workers in Homs and Aleppo, Syria, 12 July 2013 (internet). Available at http://reliefweb.int/report/syrian-arab-republic/un-humanitarian-and-human-rights-chiefs-urge-immediatesafe-passage-0 7 Of this number 1,851, 572 are registered and a further 186,766 are awaiting registration (see http://data.unhcr.org/syrianrefugees/regional.php accessed on 15/09/13) 8 Source: UNHCR, Syria Regional Refugee Response (internet). Available at http://data.unhcr.org/syrianrefugees/regional.php accessed on 15/09/13 9 Source: OCHA FTS (internet). Available at http://fts.OCHA.org/reports/daily/ocha_R1_A1007___1309150100.pdf. Accessed on 15/09/13 10 Source: OCHA FTS (internet). Available at http://fts.OCHA.org/reports/daily/ocha_R1_A1010___1309150100.pdf. Accessed on 15/09/13
humanitarian funding assistance. This insecurity increases the importance of accurate and timely situation analysis of the needs and vulnerabilities of populations affected by the conflict. 2.2 Background to the Syria Needs Analysis Project SNAP is a one-year project delivered through a partnership between ACAPS and MapAction running from December 2012 December 2013. It seeks to deliver independent analysis of the humanitarian situation and in so doing, facilitate a better targeted response and shared situational analysis through the collation and analysis of existing assessment data and the production of: Regional Analysis for Syria (RAS) - A monthly general brief produced by ACAPS and MapAction collated from a review of existing assessment and response data. The brief focuses on gap analysis, lessons learned, priority needs, displacement profile, sectoral analysis; Thematic Reports - specialised briefings on relevant sectors; topics and indepth country/ Governorate briefings; Scenario reports.
SNAP has also provided support to the humanitarian community on technical assistance on information management and joint assessments specifically on the Joint Rapid Assessment of northern Syria.
Figure 1: The Assessment Capacities Project and MapAction In accordance with fundamental humanitarian principles, and as endorsed by the Good Humanitarian Donorship Initiative, humanitarian aid should be based on a clear understanding of the needs of the affected population. However, providing a sufficiently coherent picture of humanitarian requirements, especially in the initial phases of an emergency can prove challenging. The Assessment Capacities Project (ACAPS) was established to help overcome these issues. ACAPS is an initiative of a consortium of three Non-Governmental Organisations (NGO), HelpAge International, Merlin and Norwegian Refugee Council. The objective of the project is to assist in strengthening global, regional and in-country needs assessment in coordination with its partners which includes NGOs, United Nations and academics. MapAction is a UK-based NGO that supports humanitarian agencies through the provision of information management and geographic information system services with a view to creating a shared operational picture to assist in the targeting of humanitarian assistance.
Why was SNAP launched? The SNAP project was borne out of a particular set of circumstances that were present in Syria towards the end of 2012 which included a crisis of growing proportions for which there was a lack of coherent analysis and scant information on the location and scale of the needs. One of the implications of the regional nature of the emergency which went far beyond Syrias own borders was that there was a lack of a shared situation awareness which was manifest in inconsistencies in how data was presented and the way in which needs were extrapolated. Specific impediments to creating a shared situational awareness include the following; The regional nature of the crisis which includes both internally displaced persons (IDPs) in Syria and refugees in neighbouring countries has resulted in information management being divided between two UN agencies, the United Nations High Commissioner for Refugees (UNHCR), which has responsibility for the refugees, and the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) which focuses on negotiating access to affected [displaced] people, information management and coordinatinghumanitarian response and advocacy.11
11
Source: OCHA in 2012 and 2013, Syrian Arab Republic (internet). Available at http://www.OCHA.org/ocha2012-13/syria. Accessed on 30/09/13
Complex challenges facing (i) agencies which are registered in Syria and working within the country and (ii) agencies that are working cross-border from neighbouring countries which has led to fragmented operations and reporting such that information-sharing between the two groups has been limited. The extent of the needs and the risks associated with working in Syria cross-border have also tended to mean that many agencies work in isolation of each other which has resulted in relatively few multi-agency or multi-sector assessments. For the agencies based in Syria, while an OCHA-led Joint Humanitarian Assessment (JHA) has been prepared, the methodology has been field-tested and enumerators have been trained, implementation of primary data collection is still pending approval from the Government and while some sectors have made headway in seeking to generate an overall picture of needs (most notably in the food sector), there has been very limited multi-sectoral or cross-lines assessment data. While the Syria Regional Refugee Response Inter-agency Information Sharing Portal12 has made significant progress in providing a coherent and comprehensive overview of numbers, needs and news and hosts a rapidly expanding library of relevant documents, the complexity of the refugee situation does not lend itself easily to analysis because of the differing status of those fleeing Syria13 and the manner in which they are accommodated which has implications for their visibility and the services they receive.
The lack of coherent analysis has numerous implications for the international response to the crisis, particularly in the following ways: There has been limited analysis of humanitarian needs which has adversely affected humanitarian appeals, particularly the 2012 Syria Humanitarian Response Plan (SHARP). In the absence of access in large parts of Syria and given the paucity of assessments, there was an important need to provide donors more accurate information about priority needs to allow funding to be mobilized. Operational agencies have been concerned about the impartiality of assistance; in the absence of comprehensive information about needs, it has been extremely difficult to make informed decisions about priority needs in government and opposition-controlled areas, cross-lines and cross-border. Given the scale of the needs throughout Syria and other countries affected by the crisis, there is no question about the importance of aid that is being provided but there have been concerns about a lack of knowledge about where the gaps are. The complexities to providing humanitarian assistance in Syria has fragmented humanitarian coordination and exacerbated existing weaknesses in how data is shared between organisations. It is these gaps in analysis that SNAP has sought to fill by collecting and synthesizing information, undertaking analysis and supporting coordinated needs assessment. In doing this SNAP has relied on its own horizon scanning and the support of an Amman-based Advisory Group comprised of representatives from donors, NGOs and UN agencies to guide its work. Brief history of SNAP The initial concept of the SNAP project14 was borne out of a joint MapAction/Assessment Capacities Project (ACAPS) scoping mission to Beirut in August 2012 which noted concern from many parts of the humanitarian community of information gaps which existed and was followed by discussions with the Department for international Development (DFID) which facilitated discussion with OCHA concerning the added value of an operational unit to liaise with humanitarian stakeholders in order
12 13
See http://data.unhcr.org/syrianrefugees/regional.php There are 3 different groups; (i) those residing in camps, (ii) those who have the appropriate papers and are therefore regularly residing in a country, and; (iii) those who are irregular, meaning residing in a host country without the required documents (Source: SNAP (2013) Legal status of individuals fleeing Syria, June 2013) 14 At the time the concept note was drafted in September 2012, the project was known as the Humanitarian Assessment Project (HASP) but this was later changed to SNAP.
to collect information and generate regular reports on the situation of the affected populations.15 To achieve this, a small ACAPS/MapAction team hosted by the Norwegian Refugee Council (NRC) was deployed to the region with an initial deployment in Beirut. The first product from the team was the Syria Disaster Needs Analysis published on 22nd December 2012. This was followed by the Regional Analysis of Syria (RAS) published in January 2013 which over the subsequent 9-months has become the backbone of SNAPs work. With a presence in the region, the SNAP team took an entrepreneurial approach to identifying priorities which prompted them to provide technical support to a group of NGOs and the ACU for coordinated needs assessment undertaken from the relative security of Turkey. An initial deployment in support of the emerging NGO Forum in January 2013, prior to the deployment of an OCHA team offered an opportunity to support the collation and synthesis of assessment data. While this deployment was initially outside SNAP (a direct secondment from ACAPS to an NGO), it rapidly became clear that SNAP should expand its role to include such direct support to assessment initiatives. A revised proposal was agreed by the donor whereby the technical support required to facilitate primary data collection in Syria, specifically the Joint Rapid Assessment of Needs (J-RANS) with reports completed on 17 February 2013 (J-RANS I) and 22 May 2013 (J-RANS II), was provided by SNAP. In the process of developing the RAS, gaps in humanitarian information and analysis came to light which the SNAP team has sought to fill through the publication of thematic reports with a profile of the Aleppo Governorate (building on the J-RANS) in April 2013 which was followed by reports on the legal status of individuals fleeing Syria in June 2013. As SNAPs knowledge and analysis of the situation has developed, so too have its opportunities to add value, with a document outlining Scenarios for the region released in February and updated in September 2013. These were produced with a view to supporting shared strategic planning, providing early warning and promoting preparedness activities. SNAP has also sought to harvest learning from its own work (particularly the J-RANS) and the work of others through the launch of a report on needs assessment lessons learned report in September 2013 (see figure 2).
Figure 2: Timeline of SNAP information and analysis products SNAP Output Title Regional Analysis of Syria RAS September Syria (part I) and Host countries (part II) RAS August Regional analysis of the Syria conflict RAS July - Syria (part I) and Host countries (part II) RAS June - Syria (part I) and Host countries (part II) RAS May - Syria (part I) and Host countries (part II) RAS April The crisis overview RAS March - Syria (part I) and Host countries (part II) RAS February - Syria (part I) and Host countries (part II) RAS January Syria (part I) and Host countries (part II) Needs assessment lessons learned Syrian border crossings Impact of the conflict on Syrian economy & livelihoods Legal status of individuals fleeing Syria Aleppo Governorate profile Second report First report Joint rapid assessment of northern Syria (J-RANS II) Joint rapid assessment of northern Syria (J-RANS I)
Date 26 September 2013 28 August 2013 31 July 2013 26 June 2013 30 May 2013 30 April 2013 28 March 2013 27 February 2013 28 January 2013 September 2013 September 2013 July 2013 June 2013 April 2013 September 2013 February 2013 22 May 2013 17 February 2013
Thematic reports
Scenario reports Coordinated needs Assessment
15
Source: Private email communication between ACAPS and OCHA, Geneva, 02 August 2012
3. Relevance Relevance is concerned with assessing whether the project is in line with local needs and priorities (as well as donor policy).16 In the context of the SNAP review the relevance criterion speaks to the extent to which SNAP has provided information and analysis that is tailored to the needs of humanitarian agencies seeking to provide assistance to those requiring it. At the time of SNAPs launch in December 2012, very limited information existed about the humanitarian situation in Syria due to the limited access that many agencies had to assessing needs in the country. The international organisations operating in Syria were reliant on obtaining government permission for the provision of assistance and while there were some clandestine crossborder humanitarian operations being carried by a handful of NGOs, the majority of agencies were not in a position to provide assistance. At the same time, estimated numbers of IDPs were increasing rapidly (see figure 3).
Figure 3: Estimated IDP numbers within Syria (August 2012 September 2013)
17
Outside of Syria, the situation was different; there was far greater access to the refugees but capacity was limited and while information was available, there was a lack of analysis. SNAPs arrival also coincided with a massive increase in refugee numbers in each of Lebanon, Jordan, Turkey, Egypt and Iraq such that agency capacity was stretched leaving very limited space for information collection and synthesis (see figure 4 below). Regional Analysis of Syria As a consequence of the rapid increase in numbers of people requiring humanitarian assistance and the limited capacity of humanitarian organisations and the limited access that they had to key parts of Syria, information on the crisis was severely fragmented. This was exacerbated by the lack of a single agency with a mandate to collate, synthesize and analyse information on the humanitarian situation of those displaced in Syria in addition to those that had sought refuge in neighbouring countries. For this reason alone the production of SNAPs first RAS was considered a significant and
16
Source: ALNAP (2006) Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies, Overseas Development Institute, London 17 Figures are not always available for the exact end of the month due to varying OCHA reporting times. Hence a months data may be taken from the earliest report of the following month. Source: OCHA Humanitarian Bulletin, Issues 1 33 in Ferris E, Kirisci K and Shaikh S (2013) Syrian Crisis: Massive Displacement, Dire Needs and a Shortage of Solutions, Brookings Institution, pp.iv
timely achievement. That the exercise coincided with the beginning of a spike in population movements and associated humanitarian need made the analysis all the more relevant to the needs of a diverse set of stakeholders within the humanitarian community who were unable to access this information from elsewhere.
The RAS has helped us make sense of the broken links between different projects within the region NGO Programme Director, Turkey Figure 4: Estimated number of Syrian refugees (March 2012 September 2013)
18
For operational NGOs and UN agencies, the lack of coherent information and analysis had militated against them developing a holistic picture of needs and while few agencies reported using the RAS to make programming decisions, the majority considered it the only resource that has consistently linked IDP needs in Syria with those of refugees in neighbouring countries making it invaluable for broader planning purposes. Staff newly deployed to the region spoke of it as being their sole reference and sector specialists spoke of it being a valuable resource for linking up their more detailed geographical or sectoral knowledge with sectors and areas beyond their expertise. Donor agency staff (OFDA, DFID, CIDA), analysts (Internal Displacement Monitoring Centre and NGO policy staff) and headquarters staff strongly endorsed the enduring relevance of the RAS pointing to its value as a reference resource to complement their own agency analysis and spoke of the value of the hyperlinks provided to source material used in the document. With the 9th version released at the end of September, many informants spoke of its value in documenting the changes in the humanitarian situation over time.
We consider SNAP to be highly relevant and extremely useful. The monthly updates provide us with a regional analysis and assist us in triangulating information on the ground Headquarters-based donor representative
A small number of informants spoke of the declining relevance of the RAS with time; in particular OCHA agency staff spoke of the information they received from sources both inside and outside of
18
Figures are not always available for the exact end of the month due to varying OCHA reporting times. Hence a months data may be taken from the earliest report of the following month. Source: OCHA Humanitarian Bulletin, Issues 1 33 in Ferris E, Kirisci K and Shaikh S (2013) Syrian Crisis: Massive Displacement, Dire Needs and a Shortage of Solutions, Brookings Institution, pp.iv
Syria which they considered provided more detailed information about needs and as a consequence they referred less to the document now than they had in the early months when there was far less information and analysis available. Joint Rapid Assessment of northern Syria SNAPs role in the J-RANS was to provide technical support both at the early stages of methodology development and assessment training, but also at the end of the process during the processing and analysis of the data, an activity which SNAP took a lead on with a view to doing this in partnership with the humanitarian community. The J-RANS was viewed as an exceptionally important contribution particularly as at the time the assessments were undertaken, there was an almost complete lack of analysis about needs in northern Syria. The problems this lack of data posed for the humanitarian community were considerable; while a small number of individual agency assessments had been conducted in northern Syria, information only begun to be shared with the establishment of the Assessment Working Group (AWG) hosted by the NGO forum which in OCHAs absence had taken on the role of information management and coordination. Operational agencies spoke of a dearth of information to target their programming decisions and donor representatives spoke of the challenges of seeking to allocate funds without needs analysis information to support their recommendations. While there was a general view that the J-RANS methodology was imperfect, given the constraints, particularly security, it was considered to have yielded a level of data that few could have envisaged being available and in terms of the humanitarian response, it is credited by many as the key to unlocking significant donor funds at a time when the response was under-funded. For this reason it is difficult to over-state the pivotal role that the J-RANS played. Beyond the relevance of the assessments in leveraging funding, they also highlighted the needs across large parts of northern Syria and in so doing provided the evidence that operational agencies needed to target their interventions. In the same way as the RAS is considered a trusted source of analysis on the humanitarian situation across the region, the J-RANS was considered by many to have shed muchneeded light on humanitarian needs in northern Syria and as such played an important role in targeting assistance.
At the time it was released, the J-RANS was cutting edge and very relevant. Nothing like that existed at the time and we used the analysis to inform our prog ramme NGO sectoral coordinator The J-RANS provided the basis for dialogue on needs. This was an invaluable contribution. Headquarters-based donor representative
For many agencies, particularly those working cross-border, the J-RANS has been the most relevant support provided by SNAP.
J-RANS I and II were invaluable to many NGOs and provided the cornerstone for many of our projects. However since then it has gone quiet; nothing has filled the gap NGO Information Management Officer, Turkey
While the assistance currently being provided by the Global Clusters to strengthen sectoral leadership and coordination in Turkey may start to address this gap, there was concern about the time that it could take to develop and refine sectoral analysis and the implications for an understanding of needs.
Thematic Reports The thematic reports tended to receive a more muted response during the fieldwork although the results of the online survey suggest they have each filled important information gaps. Some of the early thematic reports were considered to be very long which made them harder for users to digest although they were welcomed by sector specialists. The more recent Syrian Border Crossings report was extremely well-received and while one participant expressed concerns about the wisdom of focusing undue attention on unofficial border crossings, a far greater number of respondents considered it offered the humanitarian community a common frame of reference and as such was invaluable. Scenario documents There was a generally positive response to questions about the relevance of the scenario documents from the majority of those interviewed with several NGOS saying that they had incorporated either part or the whole of the document into their agencys scenario planning and as a consequence considered the documents to be extremely relevant. While a few participants pointed to discrepancies with an OCHA document released at a similar time, others considered this to be positive as they were able to compare the documents and triangulate them against their own views.
The scenario documents have been useful as it s helpful to get an outside perspective. Its easy for an organisation to become blinkered. NGO Director, Turkey The scenario document came out at the same time as OCHA released their document but we thought this was helpful as it allowed us to triangulate information and analysis. Regional NGO emergency coordinator, Jordan
With the humanitarian situation in the region being so dynamic, many agencies considered the update of the scenario document a useful and timely exercise. Evidence on use of SNAP products from the online survey The different ways in which SNAPs information and analysis products were used were endorsed by the results of the online survey (see figure 5 below). An analysis of the responses received offers some interesting results and trends in how the products were used; The J-RANS was the document used the greatest number of times by survey participants and offered the broadest selection of uses which cut across operational and strategic purposes. JRANS I & II were the documents most frequently used to target programme assessments, to support advocacy and for proposal writing. The RAS was the second most frequently used SNAP product and was the document most often used by survey respondents for purposes of background reading. It was also the document most often used for strategy development. It was used far less frequently to target needs assessment (16% of the respondents use the RAS for this purpose). The scenario document was widely used to inform strategy development (by 33% of those who participated in the survey) with a larger proportion using it as background reading (59%). The thematic papers understandably appealed to the smallest audience with 62% of respondents indicating that they had used the Aleppo Governorate profile. Of the three papers listed, the report on the Impact of the conflict on the Syrian Economy and Livelihoods was the most frequently read with 64% of survey participants having used it as background reading and 20% using it to inform strategy.
The survey supports a view that in addition to providing important contextual knowledge for humanitarian organisations, staff have used SNAPs information and analysis for different purposes, often strategic, but also to inform operational decision-making.
10
Figure 5: Analysis of how SNAPs information and analysis has been used
19
The results of the online survey also demonstrate the enduring relevance of SNAP to date with 63% of the 94 respondents agreeing with the statement that SNAP has adapted its products over time and in so doing has ensured its continuing relevance to the needs of humanitarian stakeholders. This is important feedback for SNAP as the humanitarian community is now better organized than when the project commenced and some of the more obvious gaps in information and analysis have now been addressed. That is not to say that SNAPs existing products are insufficient to justify its continued relevance, but it does suggest an important need to continue horizon-scanning for ways that it can most effectively add value in the current situation (this is discussed in more detail in section 9.1). 4. Connectedness In the context of the SNAP review, the connectedness criterion seeks to assess the extent to which SNAP has made links to and complements the work of other providers of information and analysis. While the situation in and around Syria has presented a significant challenge to information and analysis, there is evidence to suggest that capacity for analysis is improving as a result of a scale-up in the capacity of traditional information management (IM) agencies in addition to an increase in non-traditional providers. This has provided a complex web of agencies that are seeking to fill gaps in different aspects of information collection, processing and analysis and at different levels. A rudimentary analysis of these agencies provides a mosaic of efforts (see figure 6).
19
93 people responded to the question which provided 5 different uses of the data (including not used) and allowed participants to select as many from the list as were relevant for each of the products listed
11
Figure 6: Analysis of primary data collection and analysis initiatives in support of humanitarian response The figure seeks to identify stakeholders that are regularly providing information and analysis on the impact of the Syrian conflict and makes a judgment on the nature of their activities (data collection through to data 20 analysis) and how these are used (from operational to strategic).
In absolute terms, the provision of information and analysis has improved over time with both traditional and non-traditional actors offering a far wider variety of products now than existed 9months ago. In addition to what is presented in the figure, there are also a number of confidential briefings which are shared with only a select number of humanitarian agencies. While this may be justified for reasons of operational security, from an information management perspective it makes for a fractured analysis. For certain parts of the response such as for the refugee response in Jordan, Lebanon and Turkey, information about needs and assistance was considered to have improved significantly and information-sharing between humanitarian partners is generally perceived as good. However, in other areas, particularly inside Syria in both the north and the south interviews revealed significant gaps in knowledge about the humanitarian situation and while there have been some notable successes to articulate country-wide needs (by the World Food Programme for example) efforts to fill these gaps have not yet come to fruition.21
20
The size of each of the stakeholders relative to each other is for presentational purposes only and does not indicate importance. The position of each of the stakeholders should be viewed in the context of the two axes rather than in comparison with others (i.e. SNAP, the International Crisis Group and Brookings each provide analysis on the impact of the Syria conflict albeit with some primary data collection and their products are primarily strategic in nature). 21 For northern Syria, the ACU are in the process of establishing a Dynamic Monitoring System (termed Dynamo) while in the South, OCHA has developed a joint humanitarian assessment (JHA) which aims to provide a Syria-wide picture of humanitarian needs although this has been stalled pending final approval from the government
12
4.1 SNAP information and analysis complementary to or duplicating the work of others? The review found evidence that SNAP has been successful in positioning itself as a credible and trustworthy user of data and as a consequence it benefits from information from a wide range of agencies in support of its regular secondary data review. SNAP also plays a role in facilitating information exchange, sharing raw data with a trusted group of NGOs, donors and OCHA for purposes of input into its analysis. While figure 6 shows a relatively crowded space, a number of those agencies depicted have specific audiences (Integrity collect information on behalf of DFID, Brookings and ICG have an objective of making policy change) or their information and analysis is yet to be implemented (the ACUs DYNAMO is still under development and the JHA is yet to be implemented) which leaves a relatively small number whose products are directed towards the broader humanitarian community. Of particular importance is UNHCRs information management role for refugee responses, OCHAs role for other affected populations, IMPACT which offers bespoke information and analysis products to UN agencies and humanitarian organisations and SNAP. While at a field level, information-sharing between these organisations has generally been effective, the relationship between OCHA and SNAP has at times been strained, as outlined in an InterAction/ICVA mission report from April 2013. Describing the creation of the NGO Forum in Antakya, Turkey, the report explains that;
Supported by the Syria Needs Assessment Project (SNAP), the Forum has been able to complete needs assessments and map data, a function that has enabled the NGO Forum to play a strong leadership role in information managementWhile relations are cordial 22 between OCHA and the NGO community, a lack of organisational trust was evident.
While there was a concern from some members of OCHA that there was some duplication between its own products and those produced by SNAP, the review failed to find significant evidence of this. The online survey revealed a general level of satisfaction with SNAPs information and analysis which was considered to be complementary to that of others (see figure 7).23
Figure 7: Results of the online survey opinion ranking exercise on complementarity/duplication
There is significant duplication between SNAP and other information and analysis There is some duplication between SNAP and other information and analysis SNAP complements the work of other information and analysis providers SNAP complements the work of other information and analysis providers 0
2.25 13.48 38.2 46.07 10 20 30 40 50
% agreement
Of the 89 respondents to the online survey question, over 84% indicated that SNAP complemented the work of others either extremely or moderately well, with only 2.25% considering that there was
22 23
Interaction and ICVA (2013) Mission Report: An NGO Perspective on the Response to the Syria Crisis Staff from only a single agency felt that SNAP was duplicating the work of others during the field work that accompanied the review
13
significant duplication. A selection of the comments provided by those participating in the survey provides a more nuanced picture of the views of humanitarian stakeholders.
It provides a very useful complement to the official reports of others brings together in one place news and analysis from various sources. Otherwise wed have to spend a lot more time tracking down other resources. SNAP adds a robust analysis that no one else seems to provide holistically. Particularly for the opposition-controlled areas, SNAP is the only source of reliable and comprehensive information. The information/data of SNAP and other providers is relatively limited given access issues and so it is not surprising that there is some overlap. I appreciate the triangulation that is done for SNAP production and would suggest that it would be helpful to have a bit more emphasis on government/regime perspectives and identification of priorities.
Efforts taken by SNAP to coordinate with others SNAP has sought to coordinate with OCHAs Amman-based information and analysis unit by including them as part of an Advisory Committee which provides strategic guidance to SNAP. In this capacity they also receive SNAP products in advance of being published and hence can have input or make comments on the documents prior to wider circulation. SNAP participate in the coordination forums that are chaired by OCHA and there is regular contact between staff for purposes of information exchange. To date there has been less formal coordination with IMPACT although information is regularly shared on activities. This was an area that both agencies could seek to strengthen in the future particularly as common areas of interest emerge. Strengthening collaboration is an important issue for the future and the results of the survey provide some important feedback on this; while the fragmented humanitarian architecture is not a result of SNAPs presence, as an independent entity that can work with humanitarian partners but which is outside of the traditional humanitarian architecture, SNAP has provided important support to the humanitarian community and has made up for some of its shortcomings through the information and analysis it has provided. Despite this, the status quo of a fragmented humanitarian architecture is not desirable in the long-term and a case was made by many of the review participants for the humanitarian community working more closely together in the future.
The problem is the humanitarian architecture which is far bigger than SNAP; this needs to work more effectively Regional UN agency representative, Jordan
There is little doubt that the more joined-up the humanitarian community is, the greater the likelihood there is of achieving the best possible humanitarian outcomes for people affected by the crisis in Syria. In progressing this important agenda, SNAPs efforts to collect, synthesize and analyse data provides offers an important contribution. In this respect it is encouraging that OCHA, IMPACT and SNAP are currently working together on developing a methodology for a proposed coordinated needs assessment in northern Syria.
Recommendation: The review highlighted the importance with which many humanitarian agencies and donors attach to a follow-up coordinated needs assessment for northern Syria. While a J-RANS III was a strong recommendation of humanitarian agency staff at the time the review took place, progress has since been made in developing a methodology for a comprehensive baseline assessment using a set of indicators consistent with the JHA. SNAPs participation in this initiative will provide important support.
14
5. Coherence In the context of the review, the coherence criterion speaks to the extent to which the constituent parts of the SNAP project were clear and served to strengthen each other. SNAPs outcome statement speaks of improved targeting of response and the creation of a shared situational awareness and it is evident that all aspects of SNAPs work are consistent with these ambitions. While this represents a success of the project and suggests that SNAP has performed well in seizing opportunities that fit with its mandate, it also presents a challenge for SNAP in the future as it has relied on strong leadership and relationships within the team in order to identify and exploit relevant opportunities as they arise. This has been a strength of the project but it also could have been a weakness as there was a risk that a continual search for relevance that could have taken it in divergent directions that could have been to the detriment of the project. However, the RAS has provided an important anchor for SNAP and has allowed it a degree of freedom to innovate. The most significant innovation was in participating in the J-RANS and in doing so, taking on a support role for primary data collection which was a departure from ACAPS distinctive competencies in secondary data review and analysis. While there was the potential for SNAP to get drawn into operations, it did well to maintain a capacity building and analysis role and in facilitating the collection of data through coordinated needs assessment went considerable distance to furthering SNAPs core purpose of creating a shared situational analysis. As SNAP approaches the end of its first year, given the relatively high turnover of staff at key points in time and the impact this has had on progress, there would be significant benefit from seeking to more clearly articulate potential directions for SNAP, the possibilities that exist to build on its existing portfolio of products, how it will seek to work with others and the identification of a set of principles or parameters to guide its work as a means of future-proofing itself against potential staffing changes that may compromise institutional memory. A written strategy would also serve to militate against the potential for SNAP to lose focus and expand into areas that are outside its interests.
Recommendation: A written strategy which articulates SNAPs vision and identifies a set of principles or parameters to ensure the coherence of its work would future-proof it against a loss of institutional memory as a result of staff changes and militate against the potential for SNAP to expand into areas that are outside its interests
6. Effectiveness
Effectiveness measures the extent to which an activity achieves its purpose, or whether this can be expected to happen on the basis of the outputs. Implicit within the criterion of 24 effectiveness is timeliness.
6.1 Progress of SNAP against its objectives Figure 8 (below) provides an indication of SNAPs performance against its objectives and its progress against the milestones outlined in the logical framework.
24
Source: ALNAP (2006) Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies, Overseas Development Institute, London
15
Figure 8: Summary table of progress against third quarter (Q3) milestones Expected outputs Indicator Q3 25 milestone Output 1: Existing assessment data is analysed and fed into relevant coordinated mechanisms with gaps and priorities mapped and updated narratively and visually at country and regional level. Output 2: Tracking of key issues including priority needs, lessons learned, information gaps through continuous secondary data analysis and technical support to assessments in Syria and neighbouring countries. Output 3: Develop scenarios from information gathered and lessons learned to feed into contingency plans as well as other planning processes. Output 4: Technical assessment advice and leadership given to the Assessment Working Group in Turkey and other stakeholders in the region to enable the collection and analysis of primary data from Syria Common operational understanding of affected groups conditions through objective information gap and needs analysis Improved targeting and resource allocations for specific needs of different affected groups supported thematic analysis. Contingency planning processes and preparedness measures are better informed by raised situational awareness. Appropriate assessment reports to enable response planning in Syria and improved common operational understanding of priority needs and gaps 9 general analysis briefs
Progress against milestones Regional analyses (RAS) have been published monthly since project commencement. 9 RAS published by Q3
7 sectoral / thematic reports
2 scenario exercises
5 thematic reports published by Q3 including priority needs (Aleppo Gov. profile), assessment lessons learned and papers on economy and livelihoods, legal status of refugees and border crossings 2 scenario exercises published by Q3 and evidence of use by agencies for contingency planning purposes
Technical capacity is brought to and available to stakeholder forums. (number of meetings)
Inter-agency rapid assessment of 7 northern governorate s, 1 urban assessment for Aleppo town Technical capacity made available at 9 meetings
J-RANS I, I.5, II undertaken and results published and used widely by the humanitarian community for targeting assessments and operations, fundraising and advocacy
10 NGO Assessment Working Group meetings (in Antakya, Turkey) chaired and advised 12 NGO Coordination meetings (in Turkey) attended and assisted as required 8 Information Management and Assessment Working Group meetings attended and assisted as required
The outputs that have been achieved are in line with expectations and have been highly valued by many within the humanitarian community with feedback during the review showing a high degree of satisfaction with their quality. Interviews conducted during the review, responses from the online survey and download data from Reliefweb and the ACAPS website suggest that the documents are widely read and broadly trusted which suggests that progress is being made against SNAPs ambitions to create a shared understanding of needs.
25
Q3 milestones and progress taken from the SNAP Q3 report to DFID, September 2013
16
6.2 An analysis of risks and assumptions The risks and assumptions outlined in the log frame are worthy of brief discussion as they have at times played an role in enabling or inhibiting progress to be made. Assumption 1: Reputation of ACAPS and MapAction as reliable and non-operational partners Both ACAPS and MapAction have significant experience supporting humanitarian partners in strengthening information management and analysis in a range of humanitarian environments. Their reputation has played an important role in providing reassurance of the professionalism and integrity of SNAP which has played a role in developing relationships of trust with humanitarian partners. Of note here is that both ACAPS and MapActions model of deployments is usually short-term and at times it has been difficult to recruit to the longer-term profile required by the SNAP project. High staff turnover has eroded the trust that has been essential to the success of the project which has had implications for information-sharing and for retaining the projects agility to make the best possible use of opportunities. In saying this its also important to note that the continuity in SNAPs leadership in the region and the credibility this has bought the project has also played an important part in its success. Linked to the issue of continuity of staffing is the issue of how the project has been referred to during its lifespan particularly in Turkey where ACAPS and MapAction deployments were subsumed into SNAP several months into the project and several agencies commented on the confusion that this caused. While the SNAP brand is now far better known, the use of the ACAPS and MapAction logos on the products and the use of the ACAPS mailing address for staff and for circulating the products has caused confusion in the past. Assumption 2: Cooperation and willingness of agencies to share information and analysis Given the operational sensitivities of working in Syria and cross-border from neighbouring countries and the concomitant complexities of information-sharing, there was the potential for SNAP to fail to have access to key assessments and analysis which would have compromised the quality of its work. While SNAPs establishment was supported by a small group of NGOs which it initially relied on to provide it with information and to facilitate contact with other humanitarian agencies. Since this time the team has been extremely successful in making its own contacts and in demonstrating SNAPs added value. It is of interest that some of the less supportive review participants considered that SNAPs analysis was based on information available in the public domain. This is a fallacy and the trust that many agencies, particularly NGOs, have in SNAP and the responsible way in which sensitive information has been handled has resulted in SNAP having access to data that is not shared with others. This has its challenges, particularly in how to address issues of attribution and several review participants raised concerns about SNAPs use of closed source information.26 It is the view of the reviewer that this is necessary given the operational security implications of sharing sensitive information. That is not to say that SNAP has optimal access to information and there are a number of agencies that choose not to share assessment information with the team which includes a number of Damascus-based organisations (UN, a number of the INGOs and the Syrian Arab Red Crescent) which reflects the disconnect between insiders and outsiders that is discussed in section 1.3
26
Throughout SNAPs products, information sourced from personal interviews with persons unknown to the SNAP project are referred to by the descriptor PI. Information sourced as a Trusted Source refers to information received from an actor known and trusted by the project.
17
Assumption 3/4: Access to humanitarian actors/assessment forums continue to exist for the purpose of harmonised needs assessment With the exception of a handful of agencies, SNAP has benefitted from good access to humanitarian actors and donor agencies through the coordination forums which now exist in the countries outside Syria and through direct contact between SNAP analysts and agency representatives inside Syria. Where coordination has been slow to be established, SNAPs analysts have played a particularly important role in collecting information and analysis about the situation. As has been noted elsewhere in this report, face-to-face contact between SNAP and the humanitarian community plays a particularly important role in building trust and facilitating the collection and use of assessment information. In providing feedback on the breadth of SNAPs information gathering and secondary data rev iew, several respondents spoke of a perception that the RAS was compiled from international or western data sources as opposed to Arabic or Syrian sources. While SNAP does draw its information and analysis from a diversity of sources and has a number of Arabic-speaking analysts on the team, there would be value exploring the reasons for this perception and also reflecting on whether SNAPs analysis is primarily targeted towards an international/western audience or whether theres an ambition for it to have greater influence and a larger audience outside of the international/Englishspeaking humanitarian community.
Recommendation: SNAP should clarify who its target audience is and tailor and disseminate its products appropriately. If it does want to have the potential for greater influence with national and regional organisations then there will be a need to review its translation and dissemination strategy to ensure the SNAP products are more easily accessible to non-English-speakers and adequately incorporate regional perspectives
6.3 SNAPs contribution to strengthening targeting and promoting a shared situational analysis At the outcome level, the purpose of SNAP is that populations affected by the Syria crisis benefit from improved targeted response provided by humanitarian stakeholders operating with a shared situation awareness. Reflections from review participants Any analysis of the extent to which SNAP has strengthened targeting and contributed to a shared situational awareness is beset with problems of attribution, but the review highlighted examples of how SNAPs work has contributed to meeting its purpose. While the majority of respondents considered that the RAS was of too high a level of extraction to be of use in targeting assistance, the J-RANS was cited as having contributed to the decision-making of humanitarian agencies on geographic and sectoral targeting.
While the RAS is too big-picture for operations, the J-RANS I and II directly informed humanitarian programmes. UN agency representative, Turkey
It is important to note that as a coordinated multi-sector needs assessment the J-RANS did not always provide the detail required to initiate operations and some agencies spoke of a subsequent need to reassess according to more detailed sectoral methodologies. There was also feedback provided about the time it took to process, analyse and present the assessment data which some felt was too slow given the important need to mobilise the humanitarian community and respond quickly to the needs that had been identified.
The food security information [from J-RANS II] was a good indicator of the needs and helped us target more detailed assessments. Food security sector representative, Turkey
18
The time lag between the assessments and the findings was too long; a month is a long time to wait and in such a dynamic situation had implications for the operational use of the data NGO Team leader, Turkey
The RAS, thematic reports and scenario documents have also played an important role in contributing to a shared situational analysis; there was almost universal appreciation of the role the RAS played in offering a comprehensive regional overview of the humanitarian situation. The majority of NGOs, UN agencies and donor representatives interviewed in Turkey, Jordan and Lebanon considered that it offered a level of coherence in its analysis that could not be found elsewhere. While there were some conflicting recommendations about the length of the document (having a shorter document versus not wanting to lose any of the analysis) and also about the frequency with which it is published (having a quarterly RAS with shorter updates in the months between versus wanting to maintain a monthly RAS), there were no such concern about its overall impact as a monthly regional humanitarian digest.
As a Desk Officer, I am tasked with piecing together the information on the situation and triangulating what we know. The RAS pieces it all together and is perfectly credible. Headquarters-based NGO desk officer For modest-sized donors like ourselves, there is no other way we would have access to this sort of information if it wasnt for SNAP Headquarters-based donor representative The RAS is the only analysis available that covers all sectors for all countries affected by the crisis. Theres no other strategic document of this type NGO director, Jordan
The high level of credibility of the RAS with humanitarian agencies and donors means that it has played a valuable role in influencing opinion and advocacy, informing funding decisions and supporting the development of humanitarian strategy. That is not to say that the RAS is read from cover-to-cover by all its users and the review highlighted a multitude of ways in which users engaged with the document which tended to change with proximity to the crisis: Project staff with a detailed knowledge of a specific sector or geographic area used the RAS look outside of their area of technical/geographic specialism Heads of agencies working at a country- or regional-level tended to skim the RAS, looking for new information. The longer the person had been deployed in the region, the less they tended to refer to the document. Headquarters staff tended to work through the document in more detail and considered it an essential monthly digest of the humanitarian situation. Country, regional and headquarters-based advocacy staff considered the RAS provided an important baseline, they appreciated the web-links to the sources where these were made available and considered it a trusted and credible source of data.
The RAS has also been an important vehicle for building trust and respect for SNAP and has created a platform to launch other products. A credible product has played a key role in getting stakeholder engagement. The use of SNAP information and analysis by advocacy and media organisations An indicator for the effectiveness of SNAP is the use of its information and analysis by third parties in support of policy and advocacy work which provides an indication of its credibility. Given that such documents are generally targeted at policy-makers and are shared widely, this also contributes to creating a shared situational awareness. While a comprehensive review of citations was not undertaken, an analysis of several recently-published policy documents and use of SNAPs products
19
(and products supported by SNAP such as the J-RANS) in the media provides an indication of how the information, analysis and mapping data have been used (see figure 9).
Figure 9: SNAP citations in recently published policy and advocacy documents Author Title SNAP document(s) cited in the report Brookings Institution Syrian Crisis: Massive displacement, dire needs and a shortage of solutions Aid inside Syria: Too little but not too late Situation of internally displaced persons in the Syrian Arab Republic Open Letter: Let us treat patients in Syria The regime digs in J-RANS II, June 2013 RAS, May 2013 RAS, June 2013 RAS, July 2013 Thematic report, Syrian Border Crossings, Sep 2013 J-RANS I, January 2013 Aleppo Governorate profile, March 2013 RAS, May 2013 J-RANS II, May 2013
27
Date 18 Sep 13
Refugees International United Nations General Assembly Not disclosed Economist
25 Apr 13 15 Jul 13
Aleppo Governorate profile, March 2013 Conflict map reproduced http://www.economist.com/news/middle-east-andafrica/21579494-president-bashar-assad-and-hisforces-have-won-new-lease-life-regime-digs/print Conflict map reproduced http://www.bbc.co.uk/news/world-middle-east22798391
16 Sep 13 15 Jun 13
BBC
Mapping the conflict
5 Sep 13
6.4 Timeliness The timeliness criterion was of greatest importance in the early stages of SNAPs deployment at the end of 2012 at which time the humanitarian community had very limited access to large parts of Syria and as a consequence information was very poor. This coincided with significant increases in the level of humanitarian needs as a consequence of an escalation in the conflict within Syria which caused additional displacement and which initiated the flight of conflict-affected people into neighbouring countries (see figures 3 and 4). SNAPs arrival and publication of the first RAS in January 2013 was undoubtedly well timed and by providing the first comprehensive analysis of the humanitarian situation early editions of the report were considered by many participants of the review to have made an extraordinary contribution. Donors spoke of the importance of having access to a document that was comprehensive, that provided big-picture analysis and that they could trust. The lack of such analysis had been a key driver in DFIDs decision to fund SNAP and in large part it delivered what they needed to justify their funding; representatives from other donor agencies, senior humanitarian and policy staff trying to understand and communicate the scale of the crisis gave a similar account of the importance of the analysis in the early months of the project. With the passing of time, feedback is now more mixed about the timeliness of successive RAS publications. While many still consider the monthly report an important barometer of change and value the consistency of publication which permits the identification of trends, some others who were either closer to the situation or had access to their own sources of information tended to feel that a less frequent quarterly publication would be more appropriate.
27
Each of the documents studies is fully referenced in the bibliography
20
It is difficult to overstate the timeliness of the information that came from the two J-RANS that were undertaken. Review participants spoke of a multitude of uses of the information; donors used it to justify programming their funds; policy analysts used the figures to strengthen their messages about the severity of the crisis and its impact on the Syrian people; NGOs used the data to justify their programmes to donors and agencies working in northern Syria or who wanted to work in northern Syria used the data as a basis to target their operations or to advocate on the importance of initiating operations. While some review participants quibbled over some aspects of the methodology, there was consensus over the timeliness of the exercise as a means of promoting a shared analysis of the situation and in galvanizing much-needed action. Feedback wasnt so unequivocal for other SNAP products although the scenario document released in February 2013 was considered to have come at a time when a number of agencies were seeking to strengthen their contingency planning and so were well-received. The second scenario document was similarly well-timed, being released at a time of heightened concern over security but its publication soon after a similar OCHA document did result in mixed feedback both about the utility of the document (some felt it provided important balance while others thought it served to duplicate what already existed) and as a consequence, the timeliness also. Of the thematic reports, the greatest number of comments was received about the Border Crossings document. Agencies were very positive about the timeliness of the publication which documented official and unofficial border crossings some of which had no commonly agreed name and in so doing SNAP provided a common language for the humanitarian community. As an overall comment on the timeliness of key products, SNAP has performed well in providing information and analysis that are considered to have filled important gaps in a timely manner. The challenge for SNAP in a humanitarian community that is now more mature and better organized than it was earlier in the year is whether it can continue to identify opportunities and exploit gaps in knowledge in such a timely way as it did for its earlier products such as the J-RANS. 6.5 The value of independent analysis in Syria Perceptions of SNAPs independence were explored in the online survey which asked the question, to what extent do you consider that the J-RANS and RAS are independent or free from significant political bias. Respondents were requested to assign a score of 1 to 5 with 1 being independent and 5 being not independent at all. The results are presented in figure 10. Over 60% of survey respondents considered the J-RANS and RAS as broadly independent28 with approximately 10% assigning a score which suggested concerns about a lack of independence.29 The figures from the survey are best interpreted in the context of a discussion about the comparative independence of different sources of information and analysis. As outlined in the first section of this report, there are numerous challenges to gathering data on humanitarian needs in Syria which in large part provided the raison detre for SNAP as a vehicle for the collection and analysis of independent information on needs. The majority of those interviewed during the review spoke to the success of the project in meeting this objective, speaking of the credibility of the data, the independence of the analysis and as a consequence, a level of trust in how data is used and shared that has played an important part in agencies sharing some of their more sensitive assessment data.
28 29
Those who assigned a score of 1 or 2 Those who assigned a score of 4 or 5
21
Figure 10: Perceptions of the SNAPs independence - results of the online survey
We have found that communications are not good between Damascus and our regional office in Amman and so SNAPs analysis has provided an important source of independent information. Headquarters donor representative The key added value of SNAP is that it can report on issues that no one else can talk about. Its not operational and hence is unaffected by the politics surrounding numbers. In so doing it can contribute to more effective programming Regional Donor represen tative, Jordan
That is not to say that everybody trusts SNAP documents and several informants from a UN agency spoke of their concerns about the data in the RAS particularly for information where attribution is not provided. A few people also questioned the use of ACU enumerators for the J-RANS and questioned the extent to which they could be considered independent. In view of concerns about this issue, when the methodology was being developed for J-RANS II, steps were taken to pair ACU enumerators with NGO staff in order to address such concerns. A number of those interviewed commented on the differing estimates of need offered by SNAP and the UN and opinion was mixed about the positive or negative implications of this. The variation may be explained in large part by the methodology adopted for the J-RANS II which rather than identifying people in need identified the number at risk which makes it impossible to compare the 2 figures. The confusion surrounding estimates of need, as evidenced by the quote below, adds weight to seeking to ensure greater coherence in assessment methodologies in the future.
I would consider SNAP to be independent but some of the numbers appear large and at times, anecdotal. However, at the same time the figures in the SHARP are too small and conservative Sectoral coordinator, Turkey
The above said, the vast majority of those interviewed were satisfied with the methodology used for the J-RANS exercises and also with the overall quality of the survey and as a consequence felt that the results were credible.
22
7. Efficiency Efficiency measures the outputs qualitative and quantitative achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output, to see whether the most efficient approach has been used.30 A comprehensive analysis of the efficiency criterion would seek to compare SNAP either with comparators (being projects with similar outputs and outcomes) or with established benchmarks. An analysis of the means by which inputs were converted into outputs and the cost incurred for SNAP and each of its comparators would provide an indication of its efficiency. However, for SNAP, this is complicated; while assessing the efficiency and technical quality of a project such as latrine construction where inputs and outputs are tangible may be undertaken with comparative ease, doing similar for SNAP is extremely complex as the key outputs, outcomes and indicators of success are largely intangible and hence do not lend themselves to objective and easily measurable assessment (the outputs and outcome are analysis and the indicators of success include a common understanding output 1, improved targeting output 2, better situational awareness output 3 and common operational understanding output 4). Furthermore, there are a lack of comparators for SNAP. While there are other entities that provide information and undertake analysis (e.g. OCHA, UNHCR, IMPACT) the inputs, outputs and outcomes of each differs to such an extent that any comparison would be meaningless. For the reasons given above the efficiency criterion will seek to undertake 2 analyses (i) a distinctive competence analysis with the aim of identifying SNAPs comparative advantage in relation to other information and analysis providers31, and; (ii) a comparison of alternative means by which ACAPS and MapAction could have promoted a shared situational analysis which will assess in broad terms the resource requirements and potential effectiveness of each of the different models. 7.1 Distinctive competence analysis The results of the review suggest that SNAP has distinctive competencies in the following areas: The regular publication of a regional analysis of the humanitarian impact of the Syria conflict The majority of those who participated in the review considered that SNAP offered the only comprehensive regional analysis of the humanitarian impact of the Syria crisis. That it is produced in a form that is broadly considered digestible and on a regular basis which allowed trends and changes to be analysed was considered to be a key benefit. While UNHCRs dashboard has been developed to an extent that it now provides an important source of real-time operational information in addition to hosting documents which include significant analysis (including SNAPs publications), its primary function is to provide information and so its contribution to analysis is limited. This gap is filled in part by the Inter-agency regional response for Syria refugees published by UNHCR which provides a weekly digest of the situation of refugees across the region drawing from respective host governments and over 100 partners. This offers a condensed set of highlights of the refugee
30
Source: ALNAP (2006) Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies, Overseas Development Institute, London 31 Feedback from the Review Advisory Group proposed a comparative advantage analysis is undertaken. Comparative advantage is used in economics to refer to the ability of a party to produce a particular good or service at a lower marginal and opportunity cost over another. In the absence of cost data or comparable organisational structures this is not possible, however, a distinctive competence analysis which is the competitive advantage that sets an organisation aside from others. In the context of the SNAP review this will seek to identify the areas in which it can offer added value and which it should focus on in the future.
23
response across the region while OCHA provides the missing piece of the jigsaw, the humanitarian situation in Syria on a less frequent basis through its Humanitarian Needs Overview, which to date has provided two 6-monthly updates that have been timed to coincide with the launch and review of the SHARP. While OCHA undoubtedly benefits from better information than SNAP from within Damascus, there are restrictions placed on sharing some of this information with parts of the humanitarian community which is a limitation. There were no other organisations interviewed during the review that provide a regular and comprehensive digest of the humanitarian impact of the Syria crisis and interviews with humanitarian staff suggest that SNAPs comprehensive analysis is one of its key areas of distinctive competence and was the explanation most often provided for the significant interest that there continues to be in the RAS. In the context of an emergency where information is not routinely shared, a high value was placed on SNAPs trusted and independent analysis Despite increases in the information available on the humanitarian impact of the Syria crisis, many of those who participated in the review raised concerns about the quality and quantity of information that was shared. Many (but not all) of the agencies working cross-border censor the information they share with other agencies, particularly those that report to a Damascus-based country headquarters; the opposite is also true that Damascus-based agencies have their own restrictions on what is reported to external agencies. As a consequence of these restrictions there are general concerns about the partiality of data that is shared. SNAPs triangulation of the data it receives and its approach to handling sensitive information was respected by most of those interviewed and endorsed by the online survey. In a situation where the humanitarian community is fragmented, SNAPs position as an independent entity was considered by many of those questioned to be of benefit. In saying this it is noteworthy that in partnering with individual humanitarian agencies IMPACT has followed a different path to that of SNAP and has had considerable success in strengthening operational decision-making mainly through support to primary data collection. In working in this way it has gained a reputation for high quality work and while IMPACT was not the subject of the review, it appears to have a very strong reputation despite working more closely with the traditional humanitarian architecture than SNAP. SNAPs positioning is a key issue for its future in the region and there were a number of its supporters who also considered that the IMPACT model, that of partnering with UN agencies was relevant to SNAP. In a crisis of unprecedented scale, SNAP has had success in identifying needs, acting on them and successfully meeting its commitments From its early work with the NGO Forum in Turkey in late 2012 to fill a gap in information and analysis prior to OCHAs deployment, to providing a regional overview at a time when there were no comparable products and in seizing an opportunity to work with the NGO Forum and the ACU to support the only 2 coordinated needs assessment undertaken to date in northern Syria, SNAP has built a reputation of being agile and of identifying opportunities and filling important gaps in humanitarian information and analysis. That these initiatives have broadly been successful and the results are considered to have been timely and effective by many in the humanitarian community has served to strengthen this reputation. IMPACT has also had significant success in building a reputation for credible and rigorous primary data collection. With a mandate spanning both information and coordination, and as part of the established humanitarian architecture, OCHA has had a broader role to perform which has doubtless been complicated by its presence in Syria and the limited mandate it has in refugee-receiving countries.
Feedback from the review suggests that SNAP has three distinctive competencies; these are areas of its work where it has an advantage over others and includes (i) being the only entity that provides a regular, regional humanitarian overview, (ii) has fared well at addressing the fragmented nature of information sharing between members of the humanitarian community; and (iii) has a track record
24
for agility and successful innovation. It is in these areas that SNAPs success has been built and which it should look to in determining its future in the region. 7.2 Alternative means by which SNAP could have achieved its outcomes In addition to the existing model of the SNAP project, there are 3 alternative ways in which a shared analysis of the humanitarian situation of those affected by the Syria crisis could have been achieved: Current model of SNAP resourcing SNAP comprises a team of 8 people deployed between Beirut (5 staff including the project lead, lead analyst, GIS specialist and 2 information management staff), Amman (1 analyst and an assessment expert) and Antakya (1 analyst). Analysis of the situation in Egypt and Iraq is covered from the Beirut office and liaison with agencies and contacts in Syria is split between team members. Support is provided from the ACAPS office in Geneva including a dedicated Head of Operations, additional support for analysis and assistance with communications. Alternative option 1 To deliver SNAP remotely from Geneva with short-term deployments to the region to provide technical support and assistance The remote model is the one that ACAPS currently uses to research, prepare and disseminate its Disaster Needs Analysis (DNA) of global humanitarian crises globally. This requires a small team of information analysts to collect assessment and other secondary data and to prepare the report and was the model used for the series of Syria DNAs released in December 2011, February 2012, June 2012 and December 2012. An operational learning review of similar products for Libya undertaken by the Feinstein International Centre found that there is a demand for a secondary data review product that is timely, user-friendly, reliable and delivered by an independent and respected source on a consistent disaster by disaster basis.32 The review did, however, highlight that closer links with the clusters would have been beneficial and that an operational deployment may have strengthened verification of the data and use of the analysis to inform agency responses. This in no way diminishes the value of secondary data reviews, but in the context of Syria more than most responses, where there are sensitivities about information-sharing, and there is generally a dearth of data in the public domain, longer-term field deployments have been found to be an effective means of accessing far greater amounts of data and SNAP has become a vehicle for some agencies to get data into the public domain. While this could have in part been achieved through having a series of field deployments, the failure of these to offer long-term relationship-building may have diminished the contribution that they could make to identifying opportunities (which has been a significant success of SNAP) and in building the trust which has been necessary for agencies to share sensitive information. It would have likely been possible to replicate the J-RANS exercises through short-term deployments although continuity in staffing would still offer the best possible chance of success. Alternative option 2 2 analysts deployed to the region with remote mapping capacity This model is mid-way between that which is currently in use and the DNA model described above and was the initial concept for SNAP. It would have permitted relationships to be developed and trust to have been built between SNAP and humanitarian stakeholders and would to some extent have permitted opportunities to have been identified, however, its agility to respond to need and
32
Montgomery K (2011) Operational Learning Review of Libya, Secondary Data Review (SDR), Feinstein International Center, Tufts University (internet). Available at https://www.cimicweb.org/cmo/libya/Documents/Humanitarian%20Response/operational-learning-reviewof-sdr-libya-sdr-review-final-2.pdf
25
ability to innovate would have been significantly restricted due to the limited resources requiring that priorities be made to determine which products to support and it would have been more likely that opportunities would have been missed. A smaller team would have required far more travel per team member and in practical terms, a decision would likely have had to be made between supporting the RAS or undertaking the J-RANS and on focusing on thematic reports or scenario documents and a smaller operational footprint would have significantly degraded the ability of SNAP to develop relationships across the region which would have impacted on the quality and scope of the documents (particularly the RAS for which geographic responsibility is spread across the whole team). The use of remote mapping capacity would have compromised the ability of the team to identify and respond to key user-needs. The implications of this are that opportunities would have been missed and analysis and overall quality of the product would have been weaker. Given the limited analysis that has been undertaken by other agencies, this model would have likely added value but it would not have been as effective or had the influence that SNAP has had which would have had implications for the use of its products to support advocacy, to leveraging funding and to promote a shared situational analysis. A smaller team would have achieved significant cost-savings compared to the existing model. Alternative option 3 Direct secondment of staff into UNHCR and OCHA in the region as requested/required The model of deploying staff to UN agencies and other humanitarian partners is one that both ACAPS and MapAction use elsewhere and which they have also used in the Syria crisis. ACAPS have seconded a number of assessment experts to UNHCR to perform time bound assessment duties. While the added value of these deployments have not been evaluated, feedback and anecdotal evidence from similar deployments suggest that while they can have impact at the micro-level, it is only in the longer-term that they can make a significant contribution to changing organisational culture or influence a shared situational awareness. The nature of ad-hoc country-level deployments also offers very limited opportunity to work strategically on strengthening regional analysis. The results of the review and associated survey place a high value on SNAP being able to offer independent analysis and it would not have been possible to achieve this through a secondment approach. For this reason, a significant part of SNAPs current added value would not have been achieved. 7.3 Towards a judgment on the efficiency of SNAP The evidence suggests that while the current configuration of an operational field-based team with both a regional and country-level presence is not the cheapest in terms of cost, it is the model that has the greatest opportunity of maximising the teams access to information and in so doing offers the greatest opportunity to develop a comprehensive regional analysis. This is borne out by an analysis of SNAPs distinctive competencies which suggests that SNAPs unique contribution has been its independence, its agility in responding to needs and the breadth of its analysis, all of which have been maximized through the personal contacts made possible through its operational presence (see figure 11).
Figure 11: Comparative cost and perceived effectiveness of different models of SNAP implementation SNAP implementation model Comparative cost Effectiveness Current model of country-based analysis and mapping capacity Geneva-based analysts with short technical deployments 2 analysts in region with remote mapping capacity Secondment of staff into other agencies
26
8. Value for money The approach used to assess Value for Money (VfM) for the Review was taken from the paper prepared by ITAD (on behalf of DFID) because of its comparative conceptual clarity and (see figure 12).
Figure 12: Defining Value for money
33
VfM analysis requires both a quantitative measure (economy and efficiency) and a qualitative one (effectiveness). Definitions for the terms economy, efficiency and effectiveness are as follows; Economy A measure of what goes into providing a service; the cost of inputs required to acquire, run and dispose of assets or resources (were input costs as low as possible for the desired quality?) Efficiency A measure of productivity, in other words how much you get out in relation to what is put in. This examines the relationship between inputs and outputs (were output costs comparable with benchmark output costs, and if not, what were the reasons for the discrepancies?) Effectiveness qualitative and quantitative measures of increase or decrease in outcomes that show the extent to which a programme has achieved its intended objectives.
While an assessment of VfM is becoming a more commonplace in evaluation, it is far from straightforward in the case of SNAP and suffers from the same practical difficulties as undertaking a quantitative assessment of the efficiency criterion in section 7. For the purposes of the review, the economic analysis will seek to document the extent to which the SNAP team have sought to proactively manage key cost drivers; the efficiency analysis will draw on section 7 to review alternative models that could have been adopted by SNAP with a focus on the implications that these have for issues of productivity; and effectiveness will take this analysis further by seeking to understand the extent to which the model adopted by SNAP has made a contribution to the projects anticipated outcome of creating a shared situational awareness. Economy The key cost driver that accounts for an estimated 87% of project expenditure is staff (including associated subsistence costs). A brief analysis is provided below of means through which key cost
33
Barnett C et al (2010) Measuring the impact and value for money of governance and conflict programmes, Final Report, ITAD (internet). Available at http://r4d.dfid.gov.uk/PDF/Outputs/Mis_SPC/60797_ITAD-VFMReport-Dec10.pdf
27
drivers have been managed in the project to deliver value while at the same time ensuring appropriate levels of quality: Use of rosters: ACAPS initially drew on its roster to find experienced staff for SNAP analysts and assessment experts which provided some success for short-term deployments. However, the roster is not designed to resource long-term analyst deployments. Where the rosters fared less well was in seeking to fill longer-term posts which led to a succession of staff on some occasions, particularly in Turkey. Partnership with the humanitarian community: The service provided by SNAP is achieved only in partnership with humanitarian agencies; the quality of the RAS is in large part determined by the assessment data provided to SNAP by operational agencies. In a similar way, the J-RANS was implemented by the ACU and NGOs who also participated in the analysis of the data. Working in partnership serves a dual purpose of ensuring engagement of the humanitarian community in the SNAP project, and from an economy perspective, sharing the cost burden associated with the generating and analysing data. Working with a host agency: The use by SNAP of a host agency offered considerable benefits in terms of project teams quickly being able to establish themselves through the ability of being able to draw on their host for some key project support functions. Had SNAP deployed as an independent entity, the time taken for it to establish itself would have been far greater and the costs associated with this would have been considerably higher. It is also important to note the challenges of the host agency model which has at times led to frustration from both sides and on one occasion SNAP has had to look for a new host which had resource implications. Recruiting locally: SNAP has been smart in the way that it has recruited by forming a mix of experts with international staff working alongside Arabic speaking locally-recruited analysts. This approach not only provides cost-savings, but also strengthens the quality of SNAPs analysis.
Recommendation: The ACAPS roster is not designed to deliver the profile of staffing required by SNAP. If the SNAP project is to be extended or similar projects are envisaged in the future it will be important to ensure ACAPS has recruitment processes in place to resource longer-term field-based staffing needs
Efficiency (see also section 7) The efficiency criterion identified alternative models that SNAP could have used and linked this with an examination of SNAPs distinctive competencies which underlined the importance of a field team for purposes of maintaining independence, providing capacity to respond to information gaps and building contacts across the region all of which were considered essential for delivering against SNAPs objectives. While alternative models offered lower input costs compared with the outputs, the analysis suggests that the outputs would have been of lower quality which would have limited the potential for SNAP to achieve its outcome. For these reasons, the current model was considered to offer the greatest likelihood of achieving the project outcome. Effectiveness (see also section 6) Three quarters of the way through the project timeframe SNAP has delivered satisfactory results against its objectives and making significant progress towards achieving its outcome, that populations affected by the Syria crisis benefit from improved targeted response provided by humanitarian stakeholders operating with a shared situation awareness. While the J-RANS have made an important and timely contribution to strengthening targeting by humanitarian agencies, the RAS, scenarios and thematic reports have all served to strengthen a shared situational awareness. That SNAP is considered by most of those interviewed to be uniquely positioned to offer an independent and comprehensive analysis serves to underline the progress that has been made.
28
8.1 Towards a judgment on value for money While it has not been considered possible to quantitatively assess the VfM of the SNAP project, measures have been identified through which key cost drivers were managed, an analysis has been undertaken of alternative models as part of a discussion about efficiency and judgment has been made on the effectiveness of the project. In so doing the review has demonstrated the complexity of measuring the VfM of the project, but it has also made some qualitative judgments that demonstrate the value for money which the SNAP project offers. While it is methodologically complex to assign a quantitative value to an intangible outcome (a shared situational analysis), an analysis of SNAPs VfM suggests that it has striven to identify cost drivers and has identified means of strengthening the economy of the project albeit learning some important lessons about resourcing for the future. While the high cost of the SNAP model in comparison with the alternatives could lead one to question the efficiency of the project, SNAPs distinctive competencies in comparison to other information and analysis providers and its effectiveness in delivering against its objectives and making progress towards its outcome provides a compelling justification for an operational deployment which has provided an important foundation for its success which would likely have been compromised by having a more limited deployment in the region, by adopting a model of agency secondments or by seeking to deliver the project remotely from Switzerland (ACAPS) and the UK (MapAction). This analysis supports a conclusion that the SNAP project does indeed offer value for money.
9. Recommendations for the future of SNAP In addition to looking backwards the review also provided an opportunity to look forwards and participants were asked to offer reflections on how SNAP can best add value in the future. The responses received are grouped below by theme. Expand SNAPs current 12-month lifespan There was near-consensus on the importance of SNAP continuing beyond its current funding horizon. Given the perceived added value of SNAPs information and analysis and the strong likelihood that humanitarian needs will continue for those affected by the Syria crisis the foreseeable future, there was a strong voice of support for continuing to support the humanitarian community in through the provision of information and analysis. The view was that SNAP longevity should be linked to the context and the value that it adds and for this reason it made no sense to close the project at this time. What was less clear from the review is a sense of how long the project should continue for. While this is in part a consequence of the lack of optimism about an end to the conflict and associated humanitarian needs, there was also a sense from review participants that there was no appetite for reducing capacity for analysis at a time when the humanitarian response is still in flux and there continues to be a lack of information. At the same time, there was also interest from some members of the donor community about the circumstances under which SNAP would either hand over its work or scale-down its activities. This presents an important question which would make sense for the team to consider as part of a broader discussion about the future of SNAP.
Recommendation: Given the added value of SNAPs information and analysis , the complexity of the context and the likelihood of continuing humanitarian need there is a compelling case for SNAP to extend the project. Recommendation: As part of a discussion about project extension, there would be value in SNAP identifying the circumstances under which it would stop the project and/or how and under what circumstances it
29
would scale down and/or handover the project.
Redouble efforts to support the targeting of assistance Given the trust and credibility of SNAP and its track record of successfully supporting coordinated needs assessment in Syria, there was a strong recommendation from NGOs in Turkey for SNAP to support a J-RANS III, particularly in the absence of a functioning ACU-implemented Dynamic Monitoring System and the lack of alternative coordinated needs assessments. In the time between the fieldwork and the completion of the evaluation report, progress has been made by the members of the STIMA in Turkey to define a common set of indicators and agree a methodology for a coordinated needs assessment in northern Syria which has the potential to supersede suggestions of a follow-up J-RANS. Beyond an assessment of needs in northern Syria there were also suggestions that SNAP could support coordinated needs assessment elsewhere in the country, with recommendations for it to support a Joint Rapid Analysis of Southern Syria (J-RASS). There can be no disputing the urgency of having a stronger analysis of needs through a coordinated needs assessment process although since the time the evaluation fieldwork was conducted there have also been progress made towards realising the aspirations of the OCHA-led Joint Humanitarian Assessment (JHA) with the government having approved an amended methodology. While there is little sense in undertaking competing assessments, given that neither the J-RASS nor JHA have reached the stage of implementation and given the importance of strengthening knowledge of humanitarian needs in Syria, there is value in progressing discussions on the J-RASS in parallel with ongoing negotiations concerning the JHA. In doing this it will be important for the SNAP team to keep in mind how best to position the project to create a shared situational analysis across Syria as a whole. There was significant support from participants of the review for more strenuous efforts to find commonalities between different assessment methodologies in order to better link assessment data to inform a coherent analysis, a message which was endorsed by the Jordan-based SNAP Advisory Committee and echoed by a senior donor representative based in Beirut.
There is a risk of looking at Syria in 2 parts; the NGOs from the North and the UN from the South. What is needed is to find ways to join the pieces together so that it can be viewed as a whole. Given we are going into a second round of appeal planning with a lack of joined-up analysis, this must be our priority
While resolving this challenge is beyond the ability of SNAP alone, it does make it incumbent on the project to seek practical ways to advance this agenda.
Recommendation: The potential for SNAP to support a J-RASS has been under discussion for some time. While the OCHA-led JHA initiative has the potential to gain momentum, the lack of certainty linked to the compelling need to better understand needs across Syria mean that there is more to be gained from the existence of two initiatives than a single one. SNAP should continue to explore whether there is sufficient capacity and agency engagement to support an assessment of this complexity and work with participating organisations to determine a workable methodology. If the assessment is viable, a coalition of agencies should seek to build greater support for the exercise across the wider humanitarian community and steps should be taken to build linkages with the JHA.
Continue to promote a shared situational awareness The RAS is widely respected within the humanitarian community and continues to inform agency strategy, humanitarian advocacy and donor programming. Although some participants of the review felt that it offered little to operational staff, there was not considered to be another document that came close to providing the level of coherent regional analysis that the RAS achieves.
30
In terms of the future, there was not a strong consensus from review participants on changes that could be made to strengthen the document although there was significant interest in continuing to build on its strong reputation as a trusted and independent source of analysis. Formatting and content-related issues that were recommended for consideration include the following; A continuing focus on producing a RAS light with the full document published only quarterly. The potential for the RAS light to be significantly condensed into a few pages. Improved use of infograms to highlight key developments each month. Better signposting of the most significant changes and presentation of these in the body of an email rather than as an attachment to stimulate interest and engage readers. The most significant recommendation offered in the review was for the RAS to be replaced by an online dashboard which could offer interactive regional analysis that can be updated in real time.
Strengthen and diversify the current portfolio of thematic reports The thematic reports were felt to be of significant value with some, such as the Border Crossings document offering a common reference for the humanitarian community. During the review, participants were asked to recommend potential gaps in analysis that could be the subject of thematic reports in the future. The list below provides a summary of the suggestions received: A dedicated scenario document for Lebanon that reflects the dynamic situation and the potential for instability in the future. An analysis of the situation of unregistered refugees in Lebanon which focuses on numbers, the support that different groups are receiving and the gaps that exist in assistance. The reach of the humanitarian community into Syria from neighbouring countries. This could be done in tandem with a J-RAS III to give a sense of both needs and coverage. SNAP is considered to be the only entity that is positioned to do this. A coherent regional analysis of the situation of IDPs waiting to cross international borders. The changing humanitarian situation in besieged areas inside Syria which identifies priority needs.
Support capacity building for assessment A number of agencies spoke of the limited assessment capacity they had within their teams and the potential for SNAP to play a capacity building role. This offers the potential for a general assessment training course which would be in line with SNAPs purpose of strengthening needs-based targeting of humanitarian action. This service would be a good fit with the recent addition of an assessment specialist to the SNAP team.
Recommendation: With the arrival in the team of an assessment specialist, there is a demand for SNAP to support capacity building for assessment. Providing support to an interagency assessment training course would provide a good fit with SNAPs mandate and would also strengthen its links with agency assessment teams.
9.1 Challenges for the future: Maintaining SNAPs relevance in a changing context The review has provided evidence of the relevance and effectiveness of SNAP in addition to identifying its distinctive competencies and providing a justification for its current model of operational deployment. At the same time, it also noted the fragmented humanitarian architecture that existed in Syria and parts of the broader region which was not of SNAPs making but provided an opportunity for the project to achieve some important results. Its position as collaborating with, but independent of the formal humanitarian architecture allowed it to overcome some of the barriers that traditional humanitarian actors have been encumbered by although this has not been without its cost and SNAP has a small number of critics who are far less convinced of its added value.
31
While the collaboration between the diverse parts of the humanitarian system across the countries affected by the humanitarian impact of the Syria conflict is subject to numerous conflicting pressures, there is a sense from some parts of the community that the fractures are beginning to be addressed and even where this is less evident, there is a strong desire to strengthen collaboration in the future in order to provide the greatest opportunity for providing assistance to those who most require it. Steps taken to establish sectoral leadership and to strengthen sectoral coordination in Turkey have been accompanied by efforts to develop a methodology and undertake a coordinated needs assessment. These are important steps for the humanitarian community and are consistent with the felt need of strengthening collaboration. A key challenge for SNAP in the future will be its ability to successfully navigate these changes in humanitarian architecture to support collaborative initiatives to strengthen information and analysis. While SNAP has performed well in the absence of strong coordination, there may be new opportunities that present themselves from the stronger collaboration that may now be emerging. SNAPs success has been built on the strong relationships and collaboration it has been able to foster within key parts of the humanitarian community. Looking forwards, there will be a need for SNAP to embrace positive changes in collaboration to position itself to continue adapting and identifying opportunities. Strengthening relationships with IMPACT and OCHA through joint work in Turkey represents an important step towards achieving this. 9.2 Reflections on the added value of the SNAP model for other contexts Given the unprecedented nature of SNAPs deployment and the effectiveness of the role it has played, there is understandable interest from within the organisations that support SNAP and from members of the humanitarian community about the potential role for a similar entity in other humanitarian crises. The general view was that in many humanitarian crises, while information and analysis was not always of the quality or quantity that was required, the traditional humanitarian architecture should be in a position to meet these needs and so there was relatively little enthusiasm for an operational analysis unit such as SNAP to be deployed as an accompaniment in every humanitarian response. That is not to say that independent analysis was not valued and SNAPs model of Secondary Data Reviews for humanitarian crises has been proven to add significant value albeit based out of Geneva. However, there was also broad agreement that under certain circumstances, an agile entity that could deploy quickly and work to a high standard of professionalism could add significant value. While no blueprint exists, there were broadly considered to be 3 scenarios where this would be merited: In complex, politicised environments where traditional actors, particularly the UN is unable to deliver against its mandate or where the situation is so sensitive that there is a need for an agile organisation to take on a specific role in collecting, synthesizing and analyzing information on humanitarian needs. Examples that were given include Sudan and Mali. In regional crises where existing architecture often struggles to adequately link up information and analysis on the humanitarian situation. The food crises in the Sahel and in the Horn of Africa were cited on a number of occasions. In situations where the specific mandate of different UN agencies means that responsibility for information and analysis is divided between more than one agency (specifically UNHCR and OCHA).
This view was broadly endorsed by the online survey; of the 89 respondents who answered the question, in your opinion is the SNAP model of information and analysis relevant only to the
32
humanitarian context in the Syrian Arab Republic or is there a potential role for it to be played in other humanitarian crises?, 82% or 73 respondents felt that there it had relevance in crises elsewhere (see figure 13).
Figure 13: The relevance of a SNAP model of information and analysis in other humanitarian crises
From a review of the written responses which accompanied the survey question (of which 27 were submitted) while there was a small number that suggested a similar model should be used in either all or no crises, the majority of responses provided more nuanced feedback that is consistent with the 3 categories outlined above. Discussions during the fieldwork elicited similar responses.
it would have relevance in other parts of the world especially when there are political issues which enables SNAP to release more and different information than UN agencies or single humanitarian organisations. Response to online survey There is a role for SNAP to support complex regional responses such as the Horn of Africa response where coordination functions are split and there is a dispersed humanitarian response over several countries. In such a context, a project like SNAP has the potential to bring disparate information together and provide analysis NGO programme director, Turkey
9.3 Lessons learned Given that this way of working between ACAPS and MapAction is unprecendented, it offers significant lessons for similar initiatives in the future should a relevant opportunity or request arise. Some general principles are offered in addition to specific lessons from SNAP which have been listed by theme. With two exceptions, the lessons that have been dealt with under each of the evaluation criteria will not be repeated in this section. The importance of shared objectives and an enabling environment SNAP arose from a joint scoping mission to the region by MapAction and ACAPS, proactive support from a donor agency and a coalition of NGOs that provided an initial commitment of support for the project. There was a level of agreement about the problem and a shared belief in the ability of SNAP to address this. Such support provided an important foundation and a receptive audience which played an important part in the success of the project. A willing host The project has benefitted from hosting arrangements that have allowed SNAP to deploy across the region and has offered cost efficiencies. SNAP has no legal basis in the countries in which it works and so is entirely dependent on the support of others. While the current arrangements provide
33
SNAP with the support it needs, initial hosting arrangements in Turkey were not so successful and caused significant upheaval to the team. Hosting arrangements in the future need to be based on a common understanding of the benefits and risks associated with the role in order to ensure a sustainable arrangement. Recruiting and deploying the team SNAP has benefitted from continuity in leadership and from talented and committed staff which are the most important resource that the programme has. A recommendation has already been made earlier in the report about the difficulties ACAPS has encountered in recruiting long-term analysts which given the small size of the team and the importance of building relationships of trust, is essential for the success of the project. The lack of staff continuity in Turkey at key moments has at times compromised the achievement of key activities. Recruiting the right staff at the right time for the required duration is essential to the success of the project. The need for clarity about brand and the services offered As both ACAPS and MapAction have interests in the region in addition to their commitment to SNAP, early decisions to deploy an ACAPS assessment expert and a MapAction staff member to Turkey to support the emerging NGO Forum which was later superseded by a decision to incorporate them into the SNAP project led to significant confusion among humanitarian partners and may have compromised the momentum that had been built. Using ACAPS internet address to circulate the SNAP products and using MapAction and ACAPS branding on recent documents also risks confusion. A recommendation was made during the review to better communicate the structure of SNAP and its links to MapAction and ACAPS as well as the services it offers to ensure clarity and to assist people in understanding how to engage with the project. The importance of a strong initial proposition The space that SNAP has to innovate and to take risks has been based in large part on the reputation it has gained from the publication of the RAS. The strength of SNAPs initial analysis and its ability to fill a gap in humanitarian information and analysis has played an important role in establishing its credibility and allowing it to have a voice in what is a relatively crowded humanitarian environment. The benefits of forming broad-based stakeholder groups The decision to form an Advisory Committee in Jordan formed of key NGO, UN and donor stakeholders has been important for ensuring engagement in the project, in offering it links to some of the key stakeholders and in providing strategic direction to SNAP. That Jordan is also home to many regional offices and hence has reach into the surrounding countries has offered a level of support for the project across the region.
10. Conclusion and summary of recommendations Over the period under review, there is evidence that SNAP has offered significant value to members of the humanitarian community in strengthening the targeting of assistance and has made an important contribution to a shared situation awareness. That is has achieved this in the context of a complex regional crisis and a fragmented humanitarian architecture is testament to the professionalism of the SNAP team. Recommendations are included under the relevant criterion and have been colour-coded according to the priority (red denotes priority recommendations and blue represents secondary recommendations). Relevance
34
The relevance of SNAP has stemmed from its ability to fill critical gaps in the information and analysis of the humanitarian community. The SNAP-authored RAS is widely credited to have been the first coherent regional analysis, while the SNAP-supported J-RANS remains the only coordinated needs assessments that have been conducted in northern Syria. While the thematic reports and scenario documents tend to have less profile they are credited with providing much-needed analysis and in offering a common language on key humanitarian issues. SNAPs agility in identifying and responding to the needs of the humanitarian community has gone a significant way to bringing a level of coherence to a very complex and fragmented humanitarian context.
Secondary recommendation With the arrival in the team of an assessment specialist, there is a demand for SNAP to support capacity building for assessment. Providing support to an interagency assessment training course would provide a good fit with SNAPs mandate and would also strengthen its links with agency assessment teams.
Connectedness There was a general level of satisfaction expressed by review participants about the complementarity of SNAPs information and analysis with other providers. While in the months after its deployment comparable products were few in number, with time the number and variety has increased and while duplication is still relatively small there is a need to look proactively at ways in which SNAP could best complement the work of others. This is exemplified by the steps taken to develop a follow-up needs assessment in northern Syria. Coordinated by OCHA and IMPACT, SNAP is participating in this initiative and in so doing has taken an important step in strengthening collaboration. Coherence Despite SNAPs approach to expanding by seizing opportunities and responding to needs its information and analysis products are coherent and all make a contribution to its core purpose. That said a vision and set of principles that articulate SNAPs distinctive competencies and which can help future-proof it against loss of institutional memory would be a wise investment. Effectiveness SNAP has made impressive progress against the outputs in its log frame and is on track to meet expectations by the current close of the project. Several of its early publications were considered extremely timely and have laid a foundation for SNAP to build on. Interviews with NGOs, UN agencies, donors at project, country, regional and headquarters level provided a mosaic of uses for SNAPs information and analysis which was considered by most participants who participated in the review to be of high quality which helped foster trust and credibility with SNAPs key clients. It was more difficult to determine the perceptions of national and regional organisations or the Arabicspeaking world more broadly and there would be value in SNAP reflecting on the whether these are targets and if so how best to tailor its products to better include them. An exploration of perceptions of the independence of SNAPs analysis revealed that a majority feel that it is free from significant political bias which is considered of key importance to its users, particularly in an environment where access constraints and operational security concerns have served to degrade perceptions of the independence of information.
Priority Recommendations Given the added value of SNAPs information and analysi s, the complexity of the context and the likelihood of continuing humanitarian need there is a compelling case for SNAP to extend the project. A written strategy which articulates SNAPs vision and identifies a set of principles or parameters to ensure
35
the coherence of its work would future-proof it against a loss of institutional memory as a result of staff changes and militate against the potential for SNAP to expand into areas that are outside its interests. As part of a discussion about project extension, there would be value in SNAP identifying the circumstances under which it would stop the project and/or how and under what circumstances it would scale down and/or handover the project. The potential for SNAP to support a J-RASS has been under discussion for some time. While the OCHA-led JHA initiative has the potential to gain momentum, the lack of certainty linked to the compelling need to better understand needs across Syria mean that there is more to be gained from the existence of two initiatives than a single one. SNAP should continue to explore whether there is sufficient capacity and agency engagement to support an assessment of this complexity and work with participating organisations to determine a workable methodology. If the assessment is viable, a coalition of agencies should seek to build greater support for the exercise across the wider humanitarian community and steps should be taken to build linkages with the JHA.
Secondary recommendations The review highlighted the importance with which many humanitarian agencies and donors attach to a follow-up coordinated needs assessment for northern Syria. While a J-RANS III was a strong recommendation of humanitarian agency staff at the time the review took place, progress has since been made in developing a methodology for a comprehensive baseline assessment using a set of indicators consistent with the JHA. SNAPs participation in this initiative will provide important support. SNAP should clarify who its target audience is and tailor and disseminate its products appropriately. If it does want to have the potential for greater influence with national and regional organisations then there will be a need to review its translation and dissemination strategy to ensure the SNAP products are more easily accessible to non-English-speakers and adequately incorporate regional perspectives.
Efficiency In establishing SNAP as an operational entity in Lebanon, Turkey and Jordan, ACAPS and MapAction have both worked outside their organisational comfort zones but have made some wise decisions. While a comparison of alternative models suggests that SNAP is comparatively resource-heavy, analysis suggests that it would have been impossible to achieve the same added value at distance, via a series of secondments or through a more modest deployment strategy. An analysis of SNAPs distinctive competencies relative to other information and analysis providers highlights its comprehensive regional analysis, its independence from a fragmented humanitarian architecture and its agility in identifying and seizing opportunities as being unique. A presence across the region has undoubtedly allowed SNAP to develop links, strengthen trust and build a comprehensive regional analysis and in so doing provides a compelling justification for the decision to adopt the model. Value for money While it is methodologically complex to assign a quantitative value to an intangible outcome (a shared situational analysis), an analysis of SNAPs VfM suggests that it has striven to identify cost drivers and has identified means of strengthening the economy of the project albeit learning some important lessons about resourcing for the future. While the high cost of the SNAP model in comparison with the alternatives could lead one to question the efficiency of the project, SNAPs distinctive competencies in comparison to other information and analysis providers and its effectiveness in delivering against its objectives and making progress towards its outcome provides a compelling justification for an operational deployment. This has provided an important foundation for its success which would likely have been compromised by having a more limited deployment in the region, by adopting a model of agency secondments or by seeking to deliver the project
36
remotely from Switzerland (ACAPS) and the UK (MapAction). This analysis supports a conclusion that the SNAP project does indeed offer value for money.
Secondary recommendation The ACAPS roster is not designed to deliver the profile of staffing required by SNAP. If the SNAP project is to be extended or similar projects are envisaged in the future it will be important to ensure ACAPS has recruitment processes in place to resource longer-term field-based staffing needs.
37
Annex 1: Review participants ACAPS, MapAction, SNAP staff Lars-Peter Nissen, Project Director, ACAPS Miro Modrusan, Head of Operations, ACAPS Liz Hughes, Chief Executive Officer, MapAction Jonny Douch, Operations Director, MapAction Nic Parham, SNAP project Lead Kevjn Lim, Information analyst, SNAP Turkey Lynn Yoshikawa, Information analyst, SNAP Jordan Leonie Tax, Senior Analyst, SNAP Lebanon Greg Vaughan, GIS specialist, SNAP Lebanon Rebecca Whiting, Information Management Officer, SNAP Lebanon Rasha Fahs, Information Management Officer, SNAP Lebanon Anne-Katherine Moore, Operations Director, ACAPS (formerly) Patrice Chataigner, head of Analysis, ACAPS Wilhelmina Welsch, Information Analyst, ACAPS (formerly deployed to Turkey) Antakya, Turkey UN agency, Coordination staff member NGO Forum Coordinator INGO, NFI Coordinator INGO, Deputy Team leader INGO, Project Development Officer INGO, Area Coordinator INGO, Representative INGO, Information Officer DFID, Humanitarian Advisor OFDA, Humanitarian Advisor Food Security & Livelihoods Global Cluster, Coordinator ACU, Information Management Unit Gazientiep, Turkey INGO Emergency Response Manager INGO Representative ACU, J-RANS II Assessment team leader (formerly) Amman, Jordan Philippe, World Vision International, Regional Response Director Yves Kim Creach, Senior Emergency Manager, Middle East North Africa Region, Danish Refugee Council Alex Tylcote, Integrity Iliana Mourad, Regional Coordinator, World Health Organisation Iesha Singh, Syria Regional Humanitarian Advisor, Department for International Development Anissa Toscano, Syria Regional Humanitarian Advisor (formerly) Lynne Marie Thomas, Jordan DART team leader, Office of U.S. Foreign Disaster Assistance Carsten Hansen, Jordan Country Director, Norwegian Refugee Council Renaud Rodier, Senior Emergency Coordinator, United Nations Relief and Works Agency for Palestine Refugees in the Near East Geraldine Ansart, Humanitarian Programme Officer, International Organisation for Migration Roger Hearn, Regional Director for the Middle East, Save the Children International Hugh Fenton, Regional Middle East North Africa Director, Danish Refugee Council Brendan Macdonald, Team Leader, United Nations Office for the Coordination of Humanitarian Assistance Isabelle Moussard Carlsen, Regional Operations Director, Action Contre La Faim Charlotte Schneider, Emergency Pool Coordinator, Action Contre La Faim Rasmus Egendal, Deputy Regional Emergency Coordinator, World Food Programme Beirut, Lebanon Simon Little, Humanitarian Advisor, Department for International Development Louisa Medhurst, Syria Representative, Department for International Development Colin Lee, Country Director, International Medical Corps Magalie Vairetto, Head of Mission, Handicap International Andres Gonzalez, Country Representative, War Child Holland Paolo Lubrano, Country Director, Action Contre La Faim Branko Dubajik, Country Health Director, MERLIN Dalia Aranki. ICLA Project Manager, Norwegian Refugee Council
38
Bill Marsden, Director of Programme Accountability and Development, Save the Children International Olivier-Charlie Rapoport, Emergency Coordinator, United Nations Relief and Works Agency for Palestine Refugees in the Near East Raoul Bittel, Deputy Head of Delegation, International Committee of the Red Cross Jean-Paul Cavalieri, Deputy Representative, United Nations High Commissioner for Refugees Christian Oxenboll, Regional Information Management Officer, United Nations High Commissioner for Refugees Geneva, Switzerland Tanya Wood, Senior Policy Officer, International Council of Voluntary Agencies Abraham Abraham, Director of Operations for Asia and Liaison Officer, INTERSOS Gareth Price-Jones, Humanitarian Affairs Representative, Oxfam International Carole Laleve, Information Management/External Relations Officer, Bureau for the Middle East and North Africa, United Nations High Commissioner for Refugees Nathalie Herlemont Zoritchak, Handicap International Losane retta, Care international Julie Mackay, Advocacy Advisor/humanitarian Coordinator, Australian Council for International Development Diane Keller, Senior Humanitarian Advisor, Save the Children International Guillaume Charron, Country Analyst, Middle East, Internal Displacement Monitoring Centre
Darlene Tymo, Director, WFP Office in Geneva Andy Wylie, Officer-in-Charge, Programme Support Branch, United Nations Office for the Coordination of Humanitarian Affairs Paul Shanahan, Global WASH Cluster Elaine Jepsen, Rapid Response Team Member, Global Child Protection Cluster Sabine Rakotomalala, Deputy Coordinator, Global Child Protection Working Group Headquarters (non-Geneva-based) Bendik Sorvig, Syria Programme Advisor, Norway, Norwegian Refugee Council Vincent Taillandier, Director of Operations, Paris, Action Contre La Faim Douni Dekhili, Emergency Desk, Paris, Medicins Sans Frontieres Sharmel Genthon, External Relations Officer, Office of U.S. Foreign Disaster Assistance Dianna Long, Programme Officer, Syria Response Management Team, Office of the U.S. Foreign Disaster Assistance Neil Barry, Syria Humanitarian Advisor, Department for International Development Luca Pupulin, Executive Director, Impact Initiative Reena Ghelani, Chief of Section, Middle East and North Africa region, United Nations Office for the Coordination of Humanitarian Affairs Tsedeye Girma, Humanitarian Field Support Section, Office of Emergency Programmes, UNICEF Francois Ducharme, Emergency Specialist, Middle East, Southern, Eastern and Northern Africa, UNICEF Representatives, Canadian International Development Agency
39
Annex 2: Fieldwork schedule

Date Monday 9 September Tuesday 10 September Wednesday 11 September Thursday 12 September Friday 13 September Saturday 14 September Sunday 15 September Monday 16 September Tuesday 17 September Wednesday 18 September Thursday 19 September Friday 20 September Jordan, Amman Jordan, Amman Jordan, Amman Lebanon, Beirut Lebanon, Beirut Lebanon, Beirut Country Turkey, Antakya Turkey, Antakya Turkey, Antakya Turkey, Antakya Turkey, Gazientiep Activity & meetings Travel, OCHA NGO Forum, International NGO, Food security and livelihoods sector staff Syria Information Management Group, DFID NGO representatives NGO representatives Travel DRC, Integrity, WHO, DFID OFDA, NRC, SNAP Advisory Group IOM, OCHA, ACF, MapAction DFID, IMC, Handicap International, War Child, MERLIN, UNRWA, ICRC, NRC, SCI IFRC, DRC, UNHCR DFID, Travel
Wednesday 24 September Thursday 25 September
Switzerland, Geneva Switzerland, Geneva
WASH Global Cluster, OCHA, IDMC, SNAP ICVA, Intersos, ACFID, Oxfam, Care International, Impact, Handicap International, WFP, UNHCR, Child Protection Global Cluster
40
Annex 3: Online survey results An online survey (available at http://www.surveymonkey.com/s/RYHBYTG) accompanied the review and elicited responses from 94 people who were familiar with the project and who worked in a range of humanitarian roles on issues related to the humanitarian situation in Syria. The results from the survey are presented below. Relevant data from the survey has been incorporated into the body of the report. Question 1: Which of the following SNAP information and analysis products have you used and for what purpose?
41
Question 2: On average, between January and July 2013, how often did you use information and analysis produced by SNAP?
Question 3: To what extent is each of the following SNAP information and analysis products relevant to the humanitarian situation in the Syrian Arab Republic and neighbouring countries?
42
Question 4: Which of the following statements about SNAPs information and analysis products do you agree with? I use SNAP to
Question 5: To what extent do you consider that the J-RANS and RAS are independent or free from significant political bias?
Question 6: Which of the 2 statements about the relevance of SNAPs information and analysis products over time do you most agree with?
43
Question 7: To what extent do you consider SNAP complements or duplicates the work of other information and analysis providers?
Question 8: In your opinion is the SNAP model of information and analysis relevant only to the humanitarian context in the Syrian Arab Republic or is there a potential role for it to be used in other humanitarian contexts?
Question 9: Which of the following best describes your organisation?
44
Question 10: Where are you/were you located?
45
Annex 4: ToR for the review 1. BACKGROUD Two years on the internal conflict in Syria continues to escalate. By March 2013, over 70000 people were reported killed. Several peace initiatives have failed to hold. The estimated number of internally displaced persons exceeds 3 million and a further 1.1 million people are registered or awaiting registration as refugees in neighbouring countries. Humanitarian conditions in Syria have worsened, with organisations operational in Syria claiming that the current humanitarian capacity incountry is unable to meet the growing needs. The Government of Syria has granted all main UN agencies and eleven international NGOs (INGOs) access to conflict areas. As a result, the scenarios for the Syria conflict remain complex and agencies operating within Syria and the surrounding countries are in a holding pattern with regards to preparing response mechanisms, allocating humanitarian funding assistance and identifying coordination mechanisms. This insecurity increases the importance of accurate and timely situation analysis of the needs and vulnerabilities of populations affected by the conflict. As a response to this situation, ACAPS in partnership with MapAction in December 2012 initiated the Syria Needs Analysis Project (SNAP). SNAP is a one-year project with the aim to strengthen the shared situation awareness among humanitarian stakeholders involved in the Syria conflict. 1.1. Project Background SNAP is a one-year project running from December 2012 December 2013. It aims to link with humanitarian actors in the affected region to consolidate, collate, review and analyse information on the evolution and impact of the Syria crisis. The project supports immediate and longer-term response planning mechanisms, inter-agency harmonisation of assessment and analysis processes through the provision of information analysis products as well as technical support if requested. SNAP links with humanitarian actors working from all operational hubs to facilitate a comprehensive rapid assessment of the humanitarian situation in Syria. Furthermore, SNAP will advocate for and support humanitarian actors to expand this rapid assessment including a monitoring system to allow for continuous information updates. SNAP aims to enable populations affected by the Syria crisis to benefit from improved targeted response provided by humanitarian stakeholders operating with a shared situation awareness. This is to be achieved through collation and analysis of existing assessment data and production of: A monthly general brief in line with the disaster needs analysis mandate produced by ACAPS will collate and review existing assessment and response data. The brief will focus on gap analysis, lessons learned, priority needs, displacement profile, sectoral analysis (as per new or changing information). Specialised briefings on relevant sectors; topics and in-depth country/ governorate briefings. Scenarios 2. SCOPE AND FOCUS OF THE REVIEW 2.1. Purpose of the review:
46
The purpose of the review is to evaluate the performance of the SNAP project against the targets outlined in the projects logical framework analysis, using the evaluation criteria of relevance, effectiveness, efficiency, connectedness, and coherence. Taking into account the innovative nature of the SNAP, to identify lessons learned with respect to role a project such as SNAP can play in other contexts. 2.2.Review questions The overall questions that the review seeks to answers is: Is SNAP on track in terms of achieving its outcome: Populations affected by the Syria crisis benefit from improved targeted response provided by humanitarian stakeholders operating with a shared situation awareness? Analyzing this issue will entail determining to what extent has SNAP been successful in contributing towards: consolidation, collation, review and analysis of information on the evolution and impact of the Syria crisis; immediate and longer-term response planning mechanisms, inter-agency harmonisation of assessment and analysis processes; a comprehensive rapid assessment of the humanitarian situation in Syria; a monitoring system to allow for continuous information updates;
In addition the review should seeks to answer the following questions: What were the primary motives for setting up SNAP, and How effective have the linkages been with the rest of the humanitarian system involved in response to the Syria crisis? How has the information produced by SNAP been used by decision-makers, and what impact, if any, on decisionmaking has there been? Has the project delivered value for money? How can Value for money be improved?
2.3 Evaluation criteria The Performance of SNAP should be evaluated against the evaluation criteria of: Relevance: Who are the primary stakeholders of the project (donors, humanitarian responders, Syrian Authorities, ACU) and have they remained the same throughout? What were the identified information and analysis needs of the different stakeholders and have these needs changed in any way? To what extent was the project design sound and geared towards responding to the identified information and analysis needs? What consultations were done and with whom prior to establishment of the project? Effectiveness: To what extent is SNAP effectively responding to the identified information and analysis needs of the different stakeholders? How could effectiveness be improved? How exactly has the information and analysis generated by SNAP been used? What products, processes and decision-making has it supported? Efficiency:
47
How cost-efficient is the project in terms of qualitative and quantitative outputs achieved as a result of the inputs?
Coherence: Was the design of the individual deployments and the overall project clear to the stakeholders and did they build on each other? Connectedness: How and to which extent has SNAP linked with relevant stakeholders? How and to what extent has SNAP been coordinated with the international humanitarian coordination architecture (HC/HCT) for Syria. How and to what extent is SNAP supporting collective processes led by the HC/HCT (needs analysis and prioritisation, response planning, resource mobilisation, monitoring). How has SNAP coordinated with the Information and Analysis Unit in OCHA Jordan? What steps have been taken to avoid duplication of effort? Has there been any duplication of effort? What can be done to strengthen the interaction between SNAP and other stakeholders? 2.4. Suggested Methodology Review SNAP project proposal, reports and other relevant background material. Interviews with ACAPS Project Management Team in Geneva, MapAction management in the UK, key donors, UN agencies and NGOs headquarters. Fieldwork: interviews with SNAP staff; donors; UN agencies and NGOs in Syria, Jordan, Lebanon, Turkey, including senior decision-makers (HC, Heads of UN agencies and NGOs) in the field and at HQ. An online survey of users of SNAP outputs (if feasible).
3. EVALUATION TEAM 4. Expertise required Experience with humanitarian contexts and systems. Experience with monitoring and evaluation techniques. Knowledge of assessment and independent analysis techniques and processes. Knowledge of humanitarian information management practices. Knowledge of humanitarian response decision--making processes. Knowledge of Middle Eastern context, including Syria crisis, is desirable. Excellent communication English written and verbal communication skills. Working knowledge of Arabic preferred.
5. MANAGEMENT The review is commissioned by the ACAPS Board and will be managed by a steering committee (SC) composed of representatives from the ACAPS board, donor and partner representatives. The ACAPS Project Director will serve as the secretary for the SC and as the focal point for the consultant (or Team Leader if more than one consultant). The key role of the SC is to ensure the independence of the review. Specifically the SC will: Select the consultant to carry out the review from a shortlist Approve the inception note prepared by the consultant
48
Comment on drafts of the report Approve the final report Comment on the management response to the report.
Day to day support, facilitation and coordination will be provided by ACAPS. SNAP staff in Lebanon and Turkey will provide support with arranging interviews, facilitating travel and informing relevant stakeholders of the evaluation. They will also provide documentation related to the project. 6. EXPECTED OUTPUT AND TIMEFRAME It is estimated that the consultancy will be completed in no more than 30 working days, including preparation and report writing. The final report will be disseminated in English and disseminated to ACAPS stakeholders and made available on the ACAPS website. The tentative timeframe for the review is as follows: 10 July Inception Report (max 10 pages) including evaluation framework, methodology, support requirements, interview guide and a draft outline of the report 14-19 July Preparation of field work (approximately 10 Field work foreseen) 2 August Field work completed 9 August Draft report 23 August Final report

Mid-Term Review of The Syria Needs Analysis Project

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Mid-Term Review of The Syria Needs Analysis Project

Enviado por

Direitos autorais:

Formatos disponíveis

Syria Needs Analysis Project (SNAP) External midterm review

Andy Featherstone, Independent consultant, October 2013

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of Syria Needs Analysis Project

SNAP: Regional Analysis of Syria, 26 September 20131

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

Scenario reports Coordinated needs Assessment

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

2.25 13.48 38.2 46.07 10 20 30 40 50

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

7 sectoral / thematic reports

Technical capacity is brought to and available to stakeholder forums. (number of meetings)

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

Refugees International United Nations General Assembly Not disclosed Economist

Mapping the conflict

Each of the documents studies is fully referenced in the bibliography

External mid-term review of the Syria Needs Analysis Project

Those who assigned a score of 1 or 2 Those who assigned a score of 4 or 5

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

would scale down and/or handover the project.

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

Annex 2: Fieldwork schedule

Wednesday 24 September Thursday 25 September

Switzerland, Geneva Switzerland, Geneva

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project

External mid-term review of the Syria Needs Analysis Project