Issues
The high cost of data

Cost of gathering data on new development goals could be crippling ... Have we met the millennium development goals? Which targets are closest? Honestly, we don’t really know the answers yet

Burgess COMMENTARY

Peter Burgess

Cost of gathering data on new development goals could be crippling ... Have we met the millennium development goals? Which targets are closest? Honestly, we don’t really know the answers yet

MDG : millennium development goals targets and poverty : Pollution in Bangladesh

Children play amid tanneries in Dhaka, Bangladesh. Most of the numbers on poverty eradication, for example, are based on projections and estimates. Photograph: AM Ahad/AP For the past decade and a half, the world has made a few smart promises with the millennium development goals (MDGs): to halve the proportion of people who suffer from hunger and poverty, get all children in school and dramatically reduce child mortality. We have seen a move towards success, although not all targets will be met.

Surprisingly, we have little information about what exactly we have achieved. While you can go on websites and, for instance, see how well Botswana is doing with poverty eradication, the truth is that most data is based on only one household survey – from 1993. Most of the available numbers are not based on data but on projections and estimates. In total, there are more gaps than real observations, and the observations themselves are often dubious.

This matters because the world is discussing the sustainable development goals (SDGs), a new set of targets that will take effect in 2016 and last until 2030. Of course, attention is on high-profile issues such as health, education, food, water and the environment.

But we also need to set aside resources to measure how well we tackle all these issues. How much this will cost and how much the international community can justify spending is discussed by Morten Jerven, a professor at Canada’s Simon Fraser University, in a recent paper for the Copenhagen Consensus Center.

Take the original MDGs. There were 18 simple goals. Data collection for these targets was patchy, with many gaps, and much of the information collected was of dubious quality. However, Jerven collates the information we have about survey costs and estimates that properly monitoring all 18 targets and 48 indicators would have cost $27bn. That is a significant sum, but given that the world will spend about $1.9tn on development aid over the same period, it is perhaps not unreasonable to suggest that 1.4% of this spending should go towards getting information.

The problem is that the next set of targets is growing ever larger. A high-level panel featuring the UK prime minister, David Cameron, Indonesia’s president, Susilo Bambang Yudhoyono, and Liberia’s president, Ellen Johnson Sirleaf, along with leaders from civil society and the private sector suggested 59 targets (pdf) and advocated building “better data-collection systems, especially in developing countries”.

And last month, 70 UN ambassadors in the open working group proposed a vertiginous 169 targets. One of these many targets is that, by 2020, the world should “increase significantly the availability of high-quality, timely and reliable data disaggregated by income, gender, age, race, ethnicity, migratory status, disability, geographic location and other characteristics relevant in national contexts”.

Even minimum data collection for all these 169 targets would cost at least $254bn, or almost twice the entire global annual development budget, according to Jerven.

And this is a very low estimate, since it does not take account of basic administrative data-gathering by national governments. Neither does it include costs for all the household surveys, which are recommended, since costs for these cannot be calculated. Countries where data has not yet been collected are likely to prove even costlier: six of 49 countries in sub-Saharan Africa have never had a household survey, and only 28 have had one in the past seven years.

There is also a serious question of capacity. Only about 60 countries have the basic registration systems needed to monitor trends in social indicators. Many poor countries do not have the capacity to collect useful data on a national basis. In the $254bn estimate there is no allowance for maintaining the statistical office, training and retaining personnel, and analysing and disseminating the data. There is ample evidence that the MDG agenda has already stretched statistical capacity and strained statistical offices in poor countries – 169 new targets will only make it much worse.

Industrialised countries such as Norway and the UK spend about 0.2% of their GDP on official statistical services. Using this figure as a measure of willingness to pay would suggest that we should aim more at four SDG targets, which could be properly monitored, rather than an unwieldy 169.

• This article was updated on 25 September 2014 to include the name of the thinktank that published Morten Jerven’s paper.

Post-2015 Consensus: Data for Development Assessment, Jerven http://www.copenhagenconsensus.com//publication/post-2015-consensus-data-development-assessment-jerven

Summary of the most high-yielding targets from the paper

Data for Development Target Benefit for Every Dollar Spent ... Enable the High Level Panel’s data revolution for the OWG’s 17 goals and 169 targets Likely to be < $1

Summary

The UN High Level Panel has called for a data revolution. The world's population should be counted, measured, weighed and evaluated. This information should be collected, compiled, aggregated, and presented in such a form that it can usefully inform policy makers and citizens in aggregated forms, and disaggregated according to region, village, gender and population group.

It is tempting to think that having the correct information will improve policy choices, but there is no automatic connection. This paper therefore focusses on the cost of the data revolution rather than its possible benefits. The simple starting point is that data do have a cost.

The baseline for any surveys and comparisons is a population census. If this costs roughly $1 per head, then a worldwide census in 2015 would require $7 billion, which is equivalent to a quarter of the USAID budget or the entire combined aid budgets of Norway and Denmark. Even that may be a conservative figure. Although the cost of a census in India is 40 cents per capita and $1 in China, the last census in America cost $13 billion, or almost $42 a head. The cost of the data revolution will be considerable, but has been missing from the MDG debate so far. As well as the financial cost of monitoring, there is the opportunity cost resulting from competing demands on survey capacity. Particular indicators also have an influence on behavior by skewing activities in favor of completing goals with quantifiable targets.

The MDG agenda comprised an ambitious list of eight goals and 18 targets, with no clear idea of where data would come from. 48 indicators where then designed by a technical group led by the World Bank, but there has never been an analysis of how much these data would have cost to provide, nor whether the agreed goals were either feasible or the best use of resources. This paper aims to shift the discussion in this direction for the current round of target-setting: the Open Working Group proposed 17 goals and 169 targets and the final list will undoubtedly be quite lengthy. If the cost of providing data is about $1.5 billion per target over the post-2015 period, as I estimate below, then this suggests a cost of $254 billion to provide data in support of the new targets, almost two times the total annual spend on Official Development Assistance. Considerable trimming is needed unless the Open Working Group thinks it is a good idea to spend up to 12.5% of the annual aid budget on statistics.

While the Millennium Development Goals were set with little thought about where the information would come from, with the primary question being ‘what kind of development should we target?’, this paper suggests the right question should be ‘what kind of development are we able to measure?’ Simply demanding more data without being prepared to bear the cost and responsibility could be a ‘tragedy of the commons’ for statistical services.

More data are only better data if they contain meaningful information and there are no opportunity costs to their supply. The funding available for MDG measurement is very limited and every effort must be made to ensure it is spent responsibly. This paper brings together and reconciles the existing measurement types as a reference for scholars concerned with MDG feasibility and operationalization.

How much would the previous MDGs cost?

Although it is possible to download MDG reports with country data in them, the contents are more often projections and estimates than real data. There are more gaps than real observations, and many of the observations are themselves of dubious quality.

The previous MDGs have been criticized because of their use of absolute measures which failed to take account of relative gains in many countries and also because of the apparently arbitrary nature of the indicators. There have been calls for the inclusion of groups not represented in the surveys, including street children and people in institutions. One UN suggestion is to address inequalities through:

“Setting tailored targets and disaggregating data in order to address inequalities within all goals, targets and indicators: Disaggregation of data will help measure the gaps between social and economic groups and identify who is being left behind. Setting targets to reduce these gaps (e.g. in health and education outcomes, in incomes and employment) will ensure that the most deprived are not “left until last”. This will further help to focus attention on and address direct and indirect discriminations between groups that underpin inequalities. Data should be disaggregated by at a minimum by age, sex, location, ethnicity, income quintiles and disability.”

But here it is suggested that ambition should be tempered by moderation and an appreciation of the resources needed to supply the data demanded.

Looking now at what the data for the previous MDG would have cost, we come first to the need for a baseline set by a census every ten years. This would be supplemented by an annual smaller survey to measure progress and more sizeable surveys every five years to ensure reasonably accurate reporting. According to the best estimates available, the cost of solely supporting the MDG surveys from 1990 to 2015 would have been $27 billion.

In practice, although we demand data on, for example, poverty headcount to be available and assume they are, six of the 49 countries in sub-Saharan Africa have never had a household survey and only 28 have been surveyed in the last seven years. Only about 60 countries in the world have the registration systems required to monitor basic trends in social indicators. The even more ambitious agenda now being put forward would either widen the gap between ambitions and realistic achievements or the allocation of development spending on statistics will have to be dramatically increased.

Methodology

The figure of $27bn was reached by estimating the costs of providing annual data from the most widely used survey methods, in addition to establishing benchmark data with a population census. However, the costing is based heavily on guesswork and extrapolation from known costs, because so little relevant information is in the public domain. We have also assumed, unrealistically, that there is sufficient existing statistical capacity in developing regions to cope with the work, so the estimate should be seen more as a marginal case. For these and other reasons, the final estimate is conservative, but still high enough to make reconsideration of the data demands in the post-2015 debate necessary.

We need to distinguish between two types of data: administrative and survey. Administrative information is that which is regularly collected by governments in day to day operations. The cost is borne by the government and the marginal cost of supplying such data is not included in our calculation. One of the objectives of the data revolution might be to shift the balance in data collection from survey to administrative, but this would itself require greater capacity in the offices dealing with it. OECD figures show that $2.3 billion was allocated to statistical development in 2010-2012. If the annual average amount had been the same over the period 1990 to 2015, the total cost would have been $19 billion. Of the 60 MDG indicators in use in 2008, 21 were compiled wholly or partly administratively and 49 wholly or partly via surveys. The balance tends to vary from country to country, with data objectivity generally believed to be higher in survey data. Improvements in poor countries often tend to be overstated in the administrative data. However, countries in the GDP bracket below $1500 per capita will have great difficulty in supplying the resource-intensive survey data without direct donor interest and funding.

Ignoring the administrative costs, the focus is on costing the required surveys and population censuses. The suggested minimum data requirements are:

• Population census every 10 years

• Demographic and Health Surveys every 5 years

• Living Standards Measurement Study every 5 years

• Core Welfare Indicator Questionnaire annually

Finding figures on which to base a cost estimate was challenging, with figures for the different survey types being largely unavailable or undisclosed. Generally this is either because the information is considered commercially sensitive or financial records were never kept because of various in kind contributions. We wanted to include an annual Multiple Indicator cluster survey in our proposed package but this lack of financial information meant we could not estimate the cost. There remain a number of caveats to this research, but I still maintain a reasonable level of confidence in this ‘back of the envelope’ analysis.

Results

Population size is the main determinant of cost for both censuses and surveys. For small (less than 5 million population), medium (5-20 million) and large (more than 20 million) countries, we assume a cost of conducting a census of $1, $2 and $3 per head respectively. Costs of the other surveys would vary with country size, but all fall into the range of hundreds of thousands to $1.5 million per country per survey. As much as possible of this is based on actual costs, but in other cases it has been necessary to make estimates and extrapolations. From this we derive an overall cost estimate of $27 billion.

This total overshoots what is currently earmarked for statistics in development assistance by a considerable distance, but could still be an underestimate. Countries such as Sudan and the Democratic Republic of Congo may prove more costly to survey and the need for administrative data is not included for any country. Most importantly, there is no allowance for maintaining statistical offices and building the necessary capacity. The gap between demands and capacity is likely to be further stretched by the new list of post-2015 targets and it seems likely that an (inadvisable) ad hoc approach will be taken to determine what is needed before considering how it might be paid for.

The benefits of good data and the costs of bad data

It is not feasible to analyse the potential upside and costs for every target suggested by the Open Working Group, but it is possible to suggest a broad framework for thinking the issues through.

Good data has real benefits. For example, surveys in Uganda between 1991 and 1995 showed that only 13% of government funds allocated to primary schools actually reached them. An advertising campaign in local newspapers enabled schools to check what they should have been receiving and, by 1999, 90% of the funds reached their destination.

But such data are unlikely to come from monitoring of post-2015 development targets. Governments need disaggregated, high frequency data linked to accountable administrative sub-regions, whereas MDG goals emphasise global standards and international comparability. For example, poverty headcount data are important as a baseline, but Ministries of Finance and Central Banks need monthly data on employment and inflation for day to day policy work which will have a major impact on long-term trends, whereas it is metrics of these long-term trends which are important for the MDGs. I suggest that the list of indicators should be designed with a view to directly increasing accountability. One way of assessing them and the data needs is not only their cost, but also the likelihood that they can be provided in a timely fashion, and in a form that makes the data useable for domestic policy making and that is digestible for media and civil society to further policy debates and accountability.

Bad data can also be generated. Increase in demand may be met by a supply of inferior data, particularly if demand overshoots supply and the data provision process is incentivised through a system of rewards and punishment. Unfortunately, these conditions were met in the previous MDG agenda.

As an example, administrative data on Kenyan education showed a steady increase in primary enrolment rates, with a big jump in 2003, when all primary school fees were abolished. Schools at this stage had an incentive to exaggerate numbers, since allocation of teachers and funding was based on this, but surveys from the Kenyan National Bureau of Statistics and the Demographic and Health Survey showed actual enrolment rates to be flat over the same period.

Estimating the potential cost of data for the post-2015 list is challenging, but a rule of thumb from the previous round suggests about $1.5 billion per target. If the list currently being discussed by the OWG turns out to be close to the final version, there will be about 169 targets, compared with 18 in the MDG agenda. In the costing of the 18 MDG targets it was assumed that you only needed a population census, a household budget survey every five years, a demographic health survey every five years and annual surveys to update on basic health, education and living standard metrics. The 169 new targets also include areas such as agriculture, industry and employment and some mention the industry share of GDP and the rate of economic growth. There are expensive and time-consuming data requirements and assuming a constant marginal cost may well be an understatement, given the need for capacity building in statistical offices.

If 18 targets from MDGs cost $27bn, 169 targets would roughly amount to $254bn for the 2015-2030 MDG round. That is a fairly big number. It is close to the annual global total spent on Official Development Assistance in any recent year. It thus seems that the emphasis on measurement in the post-2015 agenda needs a radical change. The real question is – if we are serious about actually measuring the targets – how much do we want to spend on data? At 169 targets, we are looking at spending about 12.5% of all the ODA in the period 2015-2030 on getting data; 90 times what Denmark spends on aid annually.

By way of comparison, Statistics Norway had a budget of about 0.2% of total government spending in 2013. If the international community is about as willing as the Norwegian government to spend on statistics, then the same proportion of the ODA budget would allow data collection for just five targets rather than the 169 proposed. Using a significant percentage of the aid budget on measurement is neither realistic nor desirable. It is absolutely certain that 169 targets would not be measured appropriately. It is very likely that success and failure in the Post-2015 Agenda will be measured with deficient and bad data unless the list of targets is radically shortened. Conclusions

In the 1990-2015 MDG database there were more gaps than observations. The previous agenda suffered from a mismatch between ambition in monitoring and ability to measure. The post-2015 agenda might end up being much more expensive than the $27 billion estimated to have been necessary to do a proper measurement job on the current one. What would have been the benefit if they had used an extra $27bn to get good data in the previous agenda? Would the benefits of revolutionizing the data supply for the post-2015 period outweigh an allocation of $254bn? I hazard a guess that the cost to benefit ratio is below one, and that therefore the data revolution as currently envisaged is a bad idea.