Monitoring, Evaluation and Learning
Learning Review of the Global Climate Innovation Centre Programme, for DFID 2018-19.
DFID is part of a multi-donor Trust Fund implemented by the World Bank to roll out climate innovation centres with a supporting funding and evidence eco-system. The aim is to support supports the incubation and scaling-up of early stage, small and growing businesses helping to address climate change. We used extensive document review, stakeholder interviews and three country case studies to draw lessons for existing and future work in this sector. GYA provided the team leader who led the review and case study teams in Ghana and Ethiopia.
Evaluation of the Making All Voices Count (MAVC) programme, for DFID 2013 – 2018
GYA provided the team leader for the Evaluation Management Unit team for this US$47 million, multi-donor challenge fund programme supporting innovation, scaling up, and research to deepen existing innovations and help harness new technologies to enable citizen engagement and government responsiveness. MAVC ran from 2013-2017 and, from mid-2014, evolved into a two-tier approach: country strategies and extensive brokering in tier one countries (Kenya, South Africa, Indonesia, Philippines, Tanzania, Ghana) with lighter, strategic thematic targeting for tier two countries (Bangladesh, Pakistan, Mozambique, Nigeria, Uganda, Liberia). The MAVC Theory of Change (ToC) was based on the view that there were: ‘Important new opportunities emerging to use mobile & internet technologies to close the gap between citizens and governments, through wider information availability, more opportunities to express citizen demands, and new ways to enable citizens to work together and with government. Mobile & internet therefore have potential to add value to wider work to improve transparency & accountability.’
We undertook formative and summative evaluation using contribution analysis and theory-based approaches with ten quasi-experimental and one experimental survey-based case studies in six countries. In order to promote learning within the programme, we acted as a ‘critical friend’ to implementing partners, for example, feeding back findings from the grant case studies to grantees and participating in MAVC learning events and working constructively with the monitoring, evaluation and learning team. We set out a number of lessons for the use of tech in governance and for innovation programmes in the final evaluation report.
Project Completion Review, Sustainability in Chinese Outbound Mining Investment – Sustainable Mining Action Plan, for DFID 2018
GYA provided the team leader (and undertook stakeholder interviews in China and internationally) that provided DFID with an assessment of the extent to which this programme achieved its purpose and the lessons that should be learned. We made a number of recommendations to inform any future UK-China collaboration on sustainable mining.
Cost-effectiveness measurement, Global Resilience Partnership, 2016 – 2019
Our director, Dr Gil Yaron, works with the ITAD team and has particular responsibility for advising mainly INGO grantees on how to measure the costs and benefits of their interventions. This has involved developing written guidance, delivering webinars with global participants and providing follow-up support.
Evaluation of the End Child Marriage Programme (ECMP), for DFID 2013 – 2018
GYA provided both the team leader and the economist for the mid-term review (MTR) and final evaluation of the DFID-funded End Child Marriage Programme (ECMP) in Amhara, Ethiopia. The ECMP had been using two intervention packages: the “intensive” involved working with many stakeholder groups within target communities for the maximum effect; the “expansion” approach was much lighter and relied on training in key local institutions influencing wider behaviour change. Our value for money analysis identified that the Government of Ethiopia ability to pay was far lower than the cost/girl implied by the intensive intervention. Hence this approach could not be replicated by Government – a key requirement for sustainability. The MTR team then worked with the implementing partners to identify how the low-cost intervention option could be adapted (based on learning to date) to deliver the required outcomes. The programme went forward with an “expansion plus” model that proved to be successful.
Evaluation of the Global Climate Development Knowledge Network (CDKN), ITAD for DFID (MTR Nov 2012 – March 2013 and Year 5, Aug 2014 – March 2015)
The Climate and Development Knowledge Network (CDKN) mission is to support decisionmakers in designing and delivering climate compatible development by combining research, advisory services and knowledge management in support of locally owned and managed policy processes. Within the broad scope of climate compatible development, the programme works across four strategic themes:
- Climate compatible development strategies and plans
- Improving developing countries’ access to climate finance
- Strengthening resilience through climate-related disaster risk management
- Supporting climate negotiators from the least developed and most vulnerable countries
GYA provided the team leader for these theory-based evaluations of CDKN as well as specialising in reviews of Climate Finance and Negotiations Support. This involved both performance and impact approaches synthesising evidence from both country and global case studies. We used contribution analysis with in-depth interviews in a number of countries as well as tracking survey and document reviews. The performance evaluation led to 15 recommendations to CDKN designed to improve delivery. The CDKN management response acknowledged the value of this evaluation to the organisation and sets out how they will address the recommendations.
Programme level M&E for the Strategic Climate Institutions Programme (SCIP) in Ethiopia, LTSi for DFID, 2012-2016
SCIP was a multimillion- dollar fund aimed at building Ethiopia’s capacity to cope with climate change and to respond to the challenges of transitioning to a climate resilient green economy. The fund comprised of contributions from the UK Government, the Norwegian Government and the Danish Government. Through a flexible grants process, the programme funded organizations that delivered strategic projects with a multiplier effect, by removing key obstacles to building climate resilience, or seizing opportunities in the green economy and supporting critical institutions charged with delivering Ethiopia’s Climate Resilient Green Economy Strategy. GYA provided the team leader (Gil Yaron), who led the development of programme Theory of Change with stakeholders, logframe indicators, establishing baseline data (managing teams of local consultants) and has worked on producing mechanisms to learn from interventions for government and for the programme. For the endline evaluation, he undertook detailed cost-benefit analysis of two case study projects to report on programme value for money.
Building Resilience to Climate Extremes and Disasters (BRACED) evaluation, ITAD for DFID, 2015 – 2018
The three-year, £110 million UK DFID BRACED programme aimed to build the resilience of up to 5 million vulnerable people against climate extremes and disasters. It was launched in January 2015 and supported over 120 organisations in 15 consortia across 13 countries in East Africa, the Sahel and Asia. Our director has worked with ITAD, leading the team working on the impact evaluation component of this evaluation – a quasi-experimental evaluation of climate resilience interventions with BRACED programme partners in Myanmar.
The BRACED Myanmar Alliance was a three-year project aiming to ‘build the resilience of 350,000 people across Myanmar to climate extremes. The project worked in 7 states, 8 townships and 155 communities. The main impact for project populations was intended to be ‘improved well-being and reduced loss and damage despite climate shocks’, and the project sought to do this by addressing immediate hazard-related needs at community level while encouraging longer-term solutions driven and delivered by communities and subnational and national government. Community Resilience Assessments (CRAs) were the first activities delivered as part of the project, and the list of community-identified needs became the basis from which local-level project interventions were selected. The selection typically involved an infrastructure requirement (linked to addressing a natural hazard, and sometimes shared between communities); a package of livelihood support (assets and trainings); capacity-building on climate change/resilience topics; and village savings and loans association (VSLA) support. A particular emphasis was placed on women’s empowerment, and leadership trainings and support to women’s self-help groups were provided. The model of delivery often required a level of reciprocity from communities, especially for infrastructure, which the project part-funded, and trainings, many of which were on a training-of-trainers model. The CRA process also encouraged community engagement with township-level government institutions and led to local government contributions to project infrastructure interventions in many cases.
We used quasi-experimental methods with panel data on the same survey respondents at baseline and endline to quantify the difference made by combinations of interventions. These methods allowed us to identify a counterfactual (what would have happened in the absence of the project interventions) by comparing the difference seen by individuals in counterfactual (non-target) and intervention (target) groups. Some of the key findings were that:
The project had an overall positive net effect on household resilience and a ‘package’ of interventions made the most difference, but there is substantial variability by location and across dimensions of resilience. The increases (while statistically significant) are modest and mainly reflect household capacities to anticipate and adapt. Project interventions have been most effective at building anticipatory capacity through improving decision-making and planning. The community resilience-planning approach seems to increase participation of women and children in planning fairly rapidly. Yet, securing significant increases in assets and widespread changes in livelihoods required to build adaptive and absorptive capacities would appear to be too high a hurdle for project interventions in a relatively short timeframe. Engaging communities in planning and prioritisation of activities appears to be effective, but equity issues need to be addressed and funding distribution and limitations need to be recognised
Environmental Economics
Cost-benefit analysis of community-based interventions to strengthen resilience to climate shocks, 2015 – 2017, DFID and ADB.
GYA provided ITAD with the team lead for this work that draws on the DFID BRACED programme findings on costs and benefits of community-based interventions supported by an NGO consortium in Myanmar. The approach is innovative in that it combines participatory methods to identify resilience changes resulting from the programme with rigorous economic valuation of lives saved as well as livelihood benefits. The latter involves careful use of benefit transfer evidence from the literature, for example, adapting regional estimates of the value of a statistical life.
Applying this methodology to four case studies townships, we found estimated economic benefits over a 10-year period (typically based on 12-18 months of post intervention data) are significantly greater than estimated costs over this period. The ratio of discounted benefits to costs varies from 2.4 to 11.
The highest returns are from relatively small-scale infrastructure investments planned with communities and local government, drawing on BRACED finance with community contributions of labour. Microfinance and pig breeding interventions have generated positive net financial returns for the households involved. This contribution to improved livelihoods has been seen within 12-18 months of the interventions, however, we will only know whether the additional capacity to absorb and adapt to shocks and stresses translates into increases in perceived resilience when the end of programme evaluation is undertaken.
Innovative micro-finance for the poor in India, DFID RIU 2008 – 2011
The Rojiroti project is funded by the Research Into Use (RIU) Programme, a major initiative of DFID that aims to improve access to knowledge and technology for poor people whose livelihoods depend on natural resources. Rojiroti promotes an innovative approach to supporting community-level self-help groups that enables very poor and socially disadvantaged rural people to improve their livelihoods. The project’s target, over a three-year period, was to directly benefit some 500,0 00 people in Bihar, Uttar Pradesh and Madhya Pradesh States. The project was successfully implemented by a coalition for delivery consisting of three partners: GY Associates Ltd, UK: overall project direction, design of monitoring and learning processes, and support for communication strategy. Centre for Promoting Sustainable Livelihoods, Bihar (CPSL): delivery of the community development and service provision elements of the project, and capacity-strengthening to support the volunteer network. Indian Council for Agricultural Research (ICAR) Research Complex for the Eastern Region, Patna: support on agricultural technologies and participatory technology development, and development and implementation of the project’s communication strategy. Our M&E work involved establishing panel data (with around 4000 observations) to undertake a quasi-experimental assessment of project impact in three States and combining this with extensive participatory ethnographic evaluative research in approximately 40 villages. Cost-benefit analysis is also undertaken based on detailed case study data.
The Contribution of Environmental and Natural Resources to Economic Diversification and Poverty Eradication, Botswana, GYA with BIDPA for UNEP-PEI, 2011-12.
This study provides an understanding of how natural resources contribute to employment and economic growth in Botswana and the ways in which improved natural resource use can contribute to poverty eradication. Botswana is critically dependent on her natural resources. Approximately 75% of paid employment can be linked to natural resources in agriculture, mining, tourism (hotels and restaurants) and water. In addition, virtually all subsistence employment in agriculture, fishing and veld food collection (which is unrecorded in official employment statistics) depends on natural resources. Although a great deal of the discussion of economic growth and poverty eradication is framed in terms of GDP, the critical importance of NR use is not adequately captured by GDP for two reasons. Firstly, most of the ecosystem services from land, water and air are simply not captured in monetary terms by GDP – no value is given to the regulating services provided by carbon storage for example. Hence, GDP measures understate the contribution of ecosystems to Botswana. Working with a team of local economists and environmental scientists, we reviewed existing academic studies and produced new case studies on tourism and horticulture. Methods used include the economic valuation of ecosystem services. Case studies were used to illustrate the limitations to poverty reduction of current practice and to illustrate costs and benefits from alternative strategies
Malawi Poverty Environment Initiative Economic Study, UNDP-UNEP PEI, 2008-10.
The Government Malawi Poverty and Environment Initiative (MPEI) – implemented with support from UNDP and UNEP had the broad aim of enhancing the contribution of the sustainable management of natural resources to poverty reduction, food security and economic growth. In this context, we worked with leading academic partners in Malawi to assess the economic costs and benefits of natural resource use in Malawi and to assess the economic loss of unsustainable use. Working with a multi-disciplinary team from two local universities, detailed economic case studies of the major natural resource sectors (forestry resources, fisheries resources, wildlife resources and soils) were undertaken and findings presented to a high level Government working group and the 15th Poverty Environment Partnership international donor meeting
Economic incentives for sustainable environmental management and poverty reduction in the context of Uganda’s National Development Plan, UNDP-UNEP PEI and DFID, 2008.
The Government of Uganda used the revision of their Poverty and Environment Action Plan to consider the linkage of environment and natural resources and poverty-reducing growth. GYA worked with a local multi-disciplinary team to look in detail at six environmental policy choices taken by Ugandan policy makers. Areas covered include fiscal instruments and broader instruments to influence behaviour such as payments for environmental services. Working with national stakeholders we presented an evidence-based case (using local and international evidence) for policy changes to generate additional revenue for the Government and improve environmental outcomes
Contingent valuation survey to identify willingness to pay for improved electricity supply in two Indian States, DFID.
GYA worked with The Energy and Resources Institute surveying some 500 manufacturing units and 900 farmers were surveyed in the south Indian state of Karnataka using a two-stage random sampling to provide interval estimates of the cost of unserved energy for industrial and agricultural consumers. The results from the survey help in providing guidance on consumer perceptions and their willingness to pay different or higher tariffs and provide provides useful inputs in designing effective policies for the power sector. We found that the estimated economic loss due to power outage in the agriculture sector varies from 1.9% to 3.6% of total State Domestic Product (SDP),
Minimizing the effects of tariff increases on the poor in Kyrgyzstan, DFID
Subsidy arrangements for energy in Kyrgyzstan were developed under the former Soviet system. However, these subsidies were regarded as being disadvantageous to the poor, delivering benefits to groups previously privileged under the former Soviet system. The challenge was to provide an evidence base for policy makers in Kyrgyzstan to enable them to examine how the rapid increase in energy costs were affecting the poor and to assess whether existing subsidy mechanisms were effective and, where necessary, to explore alternatives. GYA led the international advice for the social impact and protection component of this DFID-funded tariff reform project (under contract to IPA). Working with a team of local and international researchers we used both quantitative and qualitative techniques involving:
- Focus group work with vulnerable groups in various locations. This qualitative research was important to build a picture of how energy use was changing in the post-Soviet era and how vulnerable groups were coping. Powerful case studies were developed to illustrate key issues for policy makers and to inform questionnaires in the subsequent household survey work.
- National survey research on energy use and household expenditure (We led the design, supervision and data analysis and produced terms of reference for local market research companies). We then used this data to estimate how existing energy policy was impacting the poor
- Working with the deputy minister and her team to build confidence in this research (getting observation of focus groups & presenting at workshops and meetings with policy makers)
- Using the survey data to quantify the economic cost of existing subsidy mechanisms and to illustrate graphically who benefited and how this failed to target the poor. We also helped the local team estimate the cost (saving) of alternative subsidy mechanisms.
- Working with other donors (such as the World Bank) to reinforce the case of misallocation of resources under existing subsidy arrangements.