Idea
Way to do measurement

Why do RCTs in ICT4D when you can do A/B tests ... with TPB comment

Burgess COMMENTARY
I did engineering and economics at Cambridge. Later I became a Chartered Accountant and had a US based career in international corporate management before moving on to consulting in socio-economic development for organizations like the World Bank, the United Nations and others. I have always been interested in getting the most (good) results from the least use of resources, and this includes the collection and use of data to improve performance and decision making. Using these criteria Randomized Control Trials (RCTs) in the development assistance arena are of quite low utility and very expensive. It is true that international development is complex, and the idea of the RCT is to measure relative performance in a complex environment, but RCT's are not the only way to assess performance. I have been very effective at getting improved performance in many different situations and have made advanced use of 'common sense' and 'management by walking around'. Huge improvements in performance can be achieved by very simple oversight and good basic accounting, and in much of the international development arena, all of this is missing. Peter Burgess - TrueValueMetrics Multi Dimension Impact Accounting
Peter Burgess

Recently, D. Jerome Martin tweeted that he was happy that 50% of USAID’s Development Innovation Ventures grantees were conducting randomized control trials in their interventions. He felt it was a move in the right direction, that big data drives big impact. I disagree.

I think its great we are measuring more of our activities in development – we need to end Oscar Night Syndrome – but I don’ think RCTs are necessarily the best way to go. There many critiques of RCTs (here, here, and here are a few), but my main issue is timing. RCTs take years. In ICT4D, technology changes each quarter, if not faster. An RCT started with BlackBerry phones but not tablets in 2010 and published today? Useless.

What technology can teach us

In web design, we have A/B testing, where you develop two (or more) version of a page and test to see which one has the better response rate. Google is a big fan.

What if interventions were A/B tested? Say the top two ideas were awarded pilot funding and the service that had the best intervention result received full funding to scale? Or if USAID required A/B testing of interventions throughout the program life, and we were all honest about funding the effective one, killing HiPPO’s in the process?

Wouldn’t that make A/B testing a faster, better way to do aid, especially technology infused aid, than randomized control trials?

My point here is not to rule out RCTs in development, only that often we do not have the luxury of years for an RCT experiment to run its course. Not in the timescales of development nor in technology. So let’s not slow down innovation by stifling ICT4D with RCTs.