Microfinance
The Question of Metrics

The Art of Pointless and Misleading Microcredit Impact Evaluations

Burgess COMMENTARY
I like this analysis of the analysis. I have been a critic of the process of analysis through monitoring and evaluation (M&E) and studies for a long time in all aspect of socio-economic development not only in the analysis of microfinance.

Bottom line, the performance of the organization is of less import than the performance of society. In traditional M&E the analysis is done from the perspective of the organization and the project ... and likely the funder. What is missing from this is the performance of the people and the place (the community). What is missing, is the important part.

I have worked in various roles in the world of business and socio-economic development and metrics for several decades. Getting the measurement right makes a huge difference to what is accomplished ... and integral with this is the ability to validate the measurement. Hence in the field of socio-economic development I like to have measurements that relates to a place as well as a project and organization. The project and organization can close or move away, but the place remains in place. In my experience progress and performance depends not only on the way the organization and the project work, but also on the way the place works. Metrics that are going to be meaningful have to take into consideration the state of the place as well as the progress and performance in the organization and in the project. There are places where microfinance can be valuable ... and other places where it does nothing, or aggravates a situation. I think it is interesting that Muhammad Yunus moved on from simple micro-finance many years ago, and became an advocate for social business that would have the potential to address the bigger issues of society at scale with the funding and technology that has mainly been reserved for the for-profit business organizations and their investors. Dr. Yunus did not renounce microfinance, but I would argue that he understood the limitations. Peter Burgess TrueValueMetrics
Peter Burgess

The Art of Pointless and Misleading Microcredit Impact Evaluations May 29, 2013 in Development, Labour markets and employment, Microfinance | Tags: Bangladesh, Bolivia, Development, industrial policy, Latin America, microcredit myth, Microfinance, Milford Bateman, RCTs This guest post is provided by Milford Bateman who is a Visiting Professor of Economics at Juraj Dobrila University of Pula in Croatia and a development consultant. He recently accepted a two-month position as Distinguished Visiting Professor of Development Studies at St Mary’s University in Nova Scotia, Canada, to be taken up in late 2013. Four of the most high-profile research teams have in recent months released papers summarising the results of multi-year projects that aimed to assess the impact of microcredit. All of these projects claim to have found some small residual value in the increasingly de-bunked concept of microcredit which, the authors quickly go on to say, suggests to them that it is too early to agree with the growing number of nay-sayers and abandon the microcredit model in favour of other local development models. The four papers I refer to are: (most recently) ‘Win Some Lose Some? Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco’ by Manuela Angelucci, Dean Karlan, and Jonathan Zinman (hereafter AKZ ‘Microfinance at the Margin: Experimental Evidence from Bosnia and Herzegovina’ by Britta Augsburg, Ralph De Haas, Heike Harmgart and Costas Meghir (hereafter AHHM) ‘The Miracle of Microfinance? Evidence from a Randomized Evaluation’ by Esther Duflo, Abhijit Banerjee, Rachel Glennerster and Cynthia G. Kinnan (hereafter DBGK) ‘Are microcredit participants in Bangladesh trapped in poverty and debt?’ by Shahidur R. Khandker and Hussain A. Samad (hereafter KS). Dazzling econometrics and pioneering impact methodologies aside, the most important thing these four papers all have in common is actually something else: they all go to great lengths to avoid exploring the most awkward downside issues that lie at the heart of microcredit and, to do so, they choose to deploy some faulty logic along the way. Ignore the incumbents who lose – displacement of others The first problem with all four papers is that they ignore the issue of displacement, which is the employment or income lost in existing microenterprises as a result of the entry or expansion of yet more microcredit-assisted microenterprises and self-employment ventures. This is a very curious omission, given that displacement has been a standard issue in labour market analysis for many years. The result is that so much of the argument in favour of microcredit rests on a false contention: that the local supply of simple products and services associated with microcredit-induced informal microenterprises will automatically create sufficient local demand to absorb the increased supply. As a growing number of development economists now accept, notably including the late Alice Amsden, this form of “Say’s Law” (supply creates its own demand) is completely false, yet it lies at the heart of so many misguided local interventions in developing countries of late. The real situation, of course, is that cramming more and more informal microenterprises into an already struggling community of informal microenterprises, especially in the growing numbers of mega-slums to be found right across the developing world, inevitably leads to a very high level of displacement of existing ventures, and so generally no net employment or income gains. Our own simple experiments explicitly focusing on the displacement issue, such as in the poor neighbourhoods in the city of Medellin in Colombia, strongly suggest a very high degree of displacement typically exists in such poor communities. Moreover, an increasingly over-crowded informal sector also seriously undermines – or “crowds out” – the operations of the formal small and medium enterprises (larger than microenterprises) that are otherwise widely seen as the optimal foundation for sustainable local economic development (see below). For example, a good part of any demand actually located by informal microenterprises is actually demand originating from large companies, including multinationals, that has been diverted away from formal, unionised small and medium enterprise suppliers paying decent wages, and opportunistically channelled toward cheaper informal microenterprise suppliers that can offer a cost advantage because they are more exploitative of their employees (including migrants and children), contribute nothing to the local tax system, and have scant regard for the environmental consequences of their operations. Nonetheless, all four papers cannot see any way to factor into their analysis the impact of this important issue. AKZ do mention displacement as a potentially negative impact – though they misleadingly term it “business stealing” – and even claim that their survey factors it in. But not having undertaken any direct comparison of clients and non-clients operating in the same locality and market sub-sector, their conclusions are extremely weak. AHHM and DBGK choose not to factor in displacement issues at all. KS also ignored the issue of displacement, just as Khandker did in his other more famous 1998 paper co-authored with Mark Pitt, which was perhaps the single most important academic justification supporting the rise of the microcredit movement. Thus, in Bangladesh communities that for very many years have been over-crowded with simple informal microenterprises, and where the entry of millions more microcredit-stimulated entrants has taken place in exactly the same over-crowded local market segments, we must somehow believe KS that struggling incumbents cannot have been disadvantaged in any possible way. This is a cruel fantasy through which an entirely false picture of the impact of microcredit in Bangladesh, and by extension everywhere else, has been constructed. Of course, this falsehood is also what Muhammad Yunus had the world believe back in the late 1980’s when he was canvassing for international support for his Grameen Bank experiment, arguing that “[a] Grameen-type credit program opens up the door for limitless self-employment, and it can effectively do it in a pocket of poverty amidst prosperity, or in a massive poverty situation. Ignore the failures – most microenterprises don’t succeed Second, and like displacement a standard feature in labour market analysis, is the issue of exit. The exit issue is also hugely important because the very high failure rate of microenterprises is inextricably linked to long-term financial, social and other forms of damage to the hapless individual(s) and families involved in establishing a microenterprise that quickly fails. For example, a failed microenterprise often means the loss of additional family assets, as when remittance income or a pension is used to repay a microloan, or, even worse, when family land, a house or fixtures and fittings are sold to come up with the necessary cash to repay a microloan. So, if internationally respected SME policy academic David Storey could argue back in the 1980s that “the single most important fact to be borne in mind when implementing measures for smaller firms is the high death rate of such businesses” then it is telling that, once more, none of the four evaluation teams mentioned here even refer to the core issue of exit raised by Storey, never mind centrally factor it into their supposedly sophisticated analysis. In Bosnia, for example, the dramatically high number of quick microenterprise exits, often within just a year or so of beginning operation, is well known to both international researchers and Bosnian policy-makers. Far too many individuals in Bosnia have been unsuccessful in their microenterprise project and, as a result, have lost important assets trying to repay their microloans under pressure from microfinance institutions. Worse, even those not involved in the microenterprise, but who simply agreed to act as a guarantor for a microloan, are now losing out big-time as they are forced into repaying a microloan on behalf of those individuals now defaulting. Yet this hugely important downside to the powerful microcredit sector in Bosnia was not even discussed in the impact evaluation undertaken by AHHM. Instead, fancy econometrics were used to construct a scenario that, apart from some problems in terms of young people losing out on their education, suggested the microcredit model was overall making a minor yet positive contribution to the development of the country. Argument from ignorance – you only find what you’re looking for So, both displacement and exit are almost entirely absent from the four evaluations recently released. Why did all four teams of researchers choose not to refer to these absolutely fundamental issues? One might argue this was because these factors represent serious downside issues, and analysing them would inevitably complicate the core issue, as they saw it, of coming up with as positive an assessment of microcredit impact as was humanly possible. One cannot avoid the very worrying conclusion that impact evaluations, and particularly the use of the supposedly more accurate Randomised Control Trial (RCT) methodology, are still being deliberately deployed and wilfully misused by mainstream researchers, many of whom are all too often funded by the international development agencies that support microfinance (e.g., AHHM, KS), or the microfinance banks themselves (e.g., DBGK, AKZ), in order to perpetuate the conceit that “microcredit (sort of) works”. [1] However, the third logical error made by three of the above research teams is perhaps the most serious of all: this is the device of making far-reaching conclusions on the basis of a hugely unrepresentative short time period, one that simply does not allow for the (adverse) structural impact of microcredit credit to be registered. That is, with the exception of KM (although KM also studiously ignore the critical issue of whether or not the local economy created by microcredit in Bangladesh is really growth-oriented), these papers analyse a very short and unrepresentative time period – introducing microcredit into a community where before there was none. They then simply aggregate the short term results in such virgin territory into a generally upbeat assessment of the longer-term impact. This is utter nonsense. Among other things, we know from economic history that many of the most destructive economic as well as positive interventions only show themselves to be so when they reach a meaningful scale or after a reasonable time period, and when they have been able to fully adjust the surrounding institutional framework. And we know that the most damaging impacts registered by the microcredit model have all taken place in locations where the microcredit model has been operating for some years and has reached a meaningful scale of operations. Just like slow acting poison, it is generally only into the longer term when the impacts of microcredit are finally exposed as quite lethal – that is, when the overall economic structure is found to be an entirely unsuitable one upon which to promote economic development and poverty reduction. Thus, simply introducing microcredit into one community and assessing the impact in the first few years of operation is like drinking some strange coloured liquid and assessing its toxicity to the body exclusively in the first few minutes. In a just released article in the Mexican journal Ola Financiera (the English language version is here) I provide a brief assessment of the impact upon Latin America’s economies as a result of the growing ubiquity of the microcredit model. I conclude that the programmed shift into microcredit in Latin America has been a seriously adverse development, and it now probably ranks as one of the most damaging policy interventions of all time. Put simply, the expansion of microcredit, and also the inevitable crowding out of many other forms of enterprise finance, notably SME finance, has both established and accelerated debilitating local processes of deindustrialisation, informalisation and infantilisation. Going further, more growth-oriented formal small and medium enterprises in Latin America have clearly been “crowded out” by the dramatic rise in informal microenterprises and self-employment ventures that unfairly (because they do not pay taxes, do not provide standard social benefits, do not provide basic health and safety measures, do not provide employee training, etc) accumulate market share at their expense. For example, Vargas finds this particular development to be a very serious problem in Bolivia, undermining the activities of all formal enterprises, surprisingly even including the largest formal enterprises. Self-employment instead of productivity improvements, decent wages and job security The end result that is now becoming apparent all across the continent (and elsewhere around the world) is that local economies are structurally dumbed-down thanks to the growing supply of microcredit, and they are now increasingly incapable of generating the required formal sector jobs and growth in average incomes that might facilitate an exit from poverty. You don’t have to take my word on this important point, since even the otherwise microcredit-friendly Inter-American Development Bank (IADB) came to the same conclusion in its 2010 publication “The Age of Productivity“, reporting that economic policy in the 1980s and 1990s resulted in “the pulverisation of economic activity into millions of tiny enterprises with low productivity” (p. 6). The IADB’s overall conclusion (just shy of actually naming the culprit, microfinance) was a stunningly concise rebuke to those in Latin America and elsewhere supporting more informal microenterprises and self-employment ventures: “(T)he overwhelming presence of small companies and self-employed workers is a sign of failure, not of success” (my italics). Sadly, external support is now increasingly required to rescue the most microcredit-saturated communities from their folly. Perhaps nowhere has the mistaken belief in microcredit, and the calculations of its main supporters in Latin America (notably Peruvian economist Hernando de Soto), been proved more wrong than in Bolivia. Here, the US government (USAID) supported push for microfinance has thrown an economy once slowly and painfully industrialising under Import Substitution Industrialisation (ISI) policies, after having carefully built up a raft of efficient industrial SMEs, into reverse. Its industrial sector contracted while its informal microenterprise and self-employment sector has exploded since the 1980s. With nearly 40% of Bolivia’s financial resources now mobilised and intermediated through the powerful microfinance sector, including through the hugely celebrated BancoSol, the country has established a major economic headwind that is frustrating many other (oil and gas industry-financed) moves to upgrade Bolivia’s industrial structure and promote more sustainable enterprises capable of paying decent wages, promoting training, innovating, deploying some new technologies and exporting. If even the former head of the Chamber of Commerce in La Paz now laments the great damage done to his country thanks to the growing dominance of microcredit, and by extension to every other country in which the microcredit movement has made serious inroads, we see that even some of those traditionally ideologically predisposed to support the market-driven microcredit concept are finally seeing its real effects in practice. It is surely time to ditch pointless and misleading impact evaluations of the microcredit model, such as the four evaluations discussed above, and either come up with better methodologies, or much more critically engage with existing methodologies (such as RCTs), in order to assess the important issue of the real long-term impact of microcredit upon local economies everywhere. Footnotes: [1] According to their own statements, AKZ was funded by CompartamosBanco (the largest Mexican MFI), the Bill and Melinda Gates Foundation (a major global funder) and the National Science Foundation (unrelated to microfinance); AHHM was funded by the European Bank for Reconstruction and Development (EBRD; a major funder) and two authors are EBRD employees; KS was produced for and published by the World Bank; DBGK was funded by the Vanguard Charitable Endowment Program, ICICI bank (a wholesale funder of MFIs) and Spandana (a big Indian MFI). …