Metrics
Impact evaluations

Impact evaluation: which way forward? ... Impact evaluation can be an expensive, complicated business. Our panel offers 15 suggestions to demystify the process, and ensure it supports accountability and learning

Burgess COMMENTARY

Peter Burgess

Impact evaluation: which way forward? Impact evaluation can be an expensive, complicated business. Our panel offers 15 suggestions to demystify the process, and ensure it supports accountability and learning How should the impacts of development work be measured? Photograph: Getty images Rose Mary Garcia, director of the monitoring and evaluation practice, Crown Agents, Washington, DC, USA Involve all stakeholders: Development is a participatory process. Although every time you include more of a project's stakeholders in its evaluation, the cost of the evaluation increases, it is critical to include all stakeholders in the evaluation process. This is often missing from impact evaluations of development programmes. Use control groups for evaluation: While measuring inputs into development programmes is important, unless we measure their impact we will not be able to learn to be good implementers. In my opinion, measuring attribution is critical, and we can't do that unless we use control groups to compare them to. Increase the capacity of groups to self-evaluate: The greatest contribution to development we can give as impact evaluators is to increase capacity in governments and local institutions, so they carry their own impact evaluations. Training by the Millennium Challenge Corporation (pdf) and the poverty action lab at 3IE are good examples of how build local capacity. The World Bank is also establishing regional training centres to carry out capacity training. Nathanael Bevan, evaluation advisor, Department for International Development (DfID), London, UK Seek out funding: There is a growing commitment amongst most government agencies to take evaluation seriously, and to back this up with funds. Around 3-5% of DfID programme funds are devoted to evaluation, where appropriate. Also, a strategic impact evaluation fund has been set up to provide development groups with funding, and expert advice for impact evaluation, through the World Bank. Use external reviews to ensure evaluations are up to standard: Quality control of impact evaluation is a challenge, and there isn't a single solution. External peer reviews of evaluation designs and final reports are helpful in providing objective feedback to evaluation providers about international standards on research. Predict project impacts and evaluate frequently: It's not easy to know when to start measuring a project's impact. The impact of a project after two years may be very different to its impact eight months later, for example. This underlines the importance of having a good understanding of a programme's likely impacts, and repeating impact evaluations to test if results are replicable. Social development specialist Michael Woolcock has written a very helpful paper on this topic. Fabrizio Felloni, senior evaluation officer, independent office of evaluation at the International Fund for Agricultural Development (Ifad), Rome, Italy Use cost-saving tools: It makes economic sense to embark on an expensive evaluation using large sample surveys if the project being evaluated is large. However, sometimes it may be possible to use data from other national surveys (living standard measurement surveys, for example). Information technology and mobile phones can also be used to reduce data collection costs. Evaluations should support accountability and learning: Done well, impact evaluation supports both accountability, by verifying whether what was promised was also delivered, and learning, by drawing from the experience in order to better design and manage future interventions. Jennifer Sturdy, director of independent evaluation, Millennium Challenge Corporation (MCC), Washington, DC, US Partner government contribution and buy-in to the evaluation questions is critical: The results of the independent evaluations should feed into future decision-making – for the donor organisation, the partner government and hopefully, the broader development community. This is again a difficult task, particularly given that evaluation timelines can last 5 or more years. It is a difficult task to first establish buy-in, but then maintain it and ensure a feedback loop over many years. We are working to overcome this by establishing 'post-compact' monitoring and evaluation plans with partner governments, which help to solidify the feedback loop once evaluation results come in, even if it is years after a project ends. Evaluations are as good as their project design: Getting projects designed right means meeting a mixture of conditions. These needs to be strong institutional commitment, due consideration of constraints analysis, problem identification, literature reviews, and formulation of a programme's logic and theory of change. Implementing projects in line with their original design is tough, but meeting these necessary conditions helps ensure that results show impacts of the programme, and aren't just the result of poor execution. Richard Palmer-Jones, senior research fellow, University of East Anglia, Norwich, UK Evaluations can serve multiple purposes: UK non-profits organisations use evaluations for advertising or organisational promotion as often as they are for organisational learning, according to research by the third sector research centre. The research also shows that the way organisations use evaluations internally is different to how they present them externally. Agent based modelling can help measure the unmeasurable: Impacts are difficult to evaluate if we don't know much about them. For example, training is challenging to evaluate because it's not very clear how education works. One potential approach is through agent based modelling (ABM), which simulates the actions and interactions of individuals or groups, in order to assess their effects on a system as a whole. This approach might not seem as rigorous as counterfactual approaches, and it's expensive to do it well, but it can promote strong research approaches to evaluation. Claire Mcloughlin, research fellow, governance and social development resource centre, University of Birmingham, Birmingham, UK Find new ways to measure influence: There is an acute need for new methodologies for evaluating influence and other hard to measure activities such as advocacy, lobbying, negotiation and knowledge uptake. A recent GSDRC report grouped the methods used to evaluate 'influence' into three types: theory-based, case-based and participatory methods. These methods are still not well developed, and need more focus from the evaluation community. Mainstream evidence rating: The UK Department for International Development (DfID) model for evidence rating is relatively thorough, as it takes into account the size of an evidence base (the critical mass of evaluation studies), its consistency (how far different studies come to the same conclusions) and its context (where it has been tried geographically and how far its findings can be generalised). However, there is not enough consistency within evaluation rating more broadly. Jyotsna Puri, deputy executive director and head of evaluation, International Initiative for Impact Evaluation (3ie), Geneva, Switzerland Develop broad skills: Impact evaluations are an applied skill and require a large mix of methods. You could be a good qualitative anthropologist or an astute applied statistician, a gregarious and sensitive interviewer or a geeky mathematician who only wants to bootstrap standard errors. All these skills are required for doing impact evaluations. This content is brought to you by Guardian Professional. To get more articles like this direct to your inbox, sign up free to become a member of the Global Development Professionals Network