Metrics
Data

Impact Evaluation Databases: Can You Find EVERYTHING You’re Looking For? 3ie's Platform

Burgess COMMENTARY

Peter Burgess

Impact Evaluation Databases: Can You Find EVERYTHING You’re Looking For?

We’ve all been there. You’ve just come up with an interesting international development research question, or your NGO or government agency is about to embark on a new project, and you need to find research evidence. You set off in search of the most rigorous kind available – impact evaluations (IEs). But where do you start? Perhaps your research library is expansive, and you have too many portals to choose from. Or maybe your organization is too small to afford annual database access. One thing is for sure, the search for relevant studies can be a time consuming, and frustrating endeavor.

From this perspective, the prospect of a centralized database of IEs is certainly compelling. So it comes as no surprise that a number of organizations have taken up the call to fill this gap. Bill Savedoff and Ted Collins mused over where to find such databases in a recent blog post from the Center for Global Development. Their hopefully iterative List of Impact Evaluation Databases should help to take some of the frustration out of the discovery process for researchers and policymakers (3ie offers a new solution). But even with this growing list as a resource, two problems remain: there are still a huge number of databases to choose from, and not all databases are created equal.

So how should a researcher or policy maker choose where to start? The answer is not always straight forward. For those of us in search of evidence from an “impact evaluation database,” it’s important to remember that (whether by deliberation or default) the owners of each database make three important strategic decisions about the content they provide:

Each database has different sources for content – articles might be drawn from search engines, websites, and other databases (ex. BLDS, PubMed), or they may be limited only to in-house publications (ex. IDB, IFPRI).

Each database has criteria for which content to include – whether explicit or implicit, each repository has selection criteria such as publication type (journal articles, white papers, grey literature), sector, geography, or methodology (ex. J-PAL’s database is exclusively RCTs, while CGEA embraces quasi-experimental methods).

Databases have different quality standards – these could be extremely lax, or very rigid, and again they may be plainly stated or inadequately understood (some databases might even have competing notions of what an impact evaluation is).

Of course, from a knowledge gathering perspective, a few other important questions arise about the functionality and “search-ability” of different platforms. Does the platform allow Boolean searching? Are studies categorized with thesaurus terms? Can I save a search history? While these concerns are not insignificant (to a systematic reviewer they can mean the difference between several hours of searching, and several weeks), they should take a back seat to the content provided. But those of us in search of that content often have little time to consider the nuances of every resource available.

What makes 3ie’s platform so different?

For the last six months 3ie has considered (and at times struggled with) many of these decisions regarding our own repository of impact evaluation studies. Inspired by the past efforts of Turner and others, 3ie has undertaken a plan to grow its currently static IE database into a dynamic Register of Impact Evaluation Published Studies (or RIEPS). This new “register” is intended to be a living repository of every published impact evaluation study available in international development. For the researcher and the policymaker alike, RIEPS should be the first stop in the search for impact evaluations.

Here is why:

• We search in every place an impact evaluation might exist – websites, databases, search engines (both public and private), and across multiple sectors (education, public policy, global health, economics, etc.).
• We include all content that addresses a development question, takes place in a developing nation, and uses rigorous techniques to identify a counterfactual (what would happen in the absence of the program).
• We select studies through an intensive screening process based on a rigorous but reasonable set of quality standards.
• RIEPS is an iterative search platform outlined in our exhaustive (and at times exhausting) RIEPS Protocol Working Paper. This paper is a recipe for our comprehensive systematic, semi-annual search of over 45 databases, search engines, journal collections, and websites. To date, 3ie has collected over 56,000 records and begun the screening process. We expect a full transition to the new platform to take place during the summer of 2013.