The debate at the Australian Aid conference is now available as a podcast so you can listen. But I wanted to point out the second reason debating about RCTs is fun: the examples that advocate use of RCTs are often self-refuting as to their importance.
For instance, in the debate the main proponent of the idea that RCTs were an important innovation in development economics used the example of TOMS shoes. He said (roughly, and I am just trying to summarize what he said, I have no first hand knowledge of this topic) that the owner or CEO of TOMS shoes had been in a developing country and seen kids without shoes and so had decided to donate a pair of shoes for every pair sold. Then, after some time, they had done a RCT of the impact. He said that from the RCT they had learned that the shoe donations had little or no impact. Moreover he (the RCT proponent) said the RCT taught them they were giving the wrong kind of shoe and, if they were going to give shoes, they should give sneakers, not loafers. This, he said was a good example of the way in which RCTs contribute to development and development economics.
The table below shows, from standard World Bank sources, the headcount poverty rates at PPP$ 5.5 and 1.9 per day and the table is sorted by the absolute number of people poor at the P$5.5 poverty line among the world’s 30 most populated countries. I (and a number of other economists) argue for much higher poverty lines (“who is not poor“) for measuring global poverty, more like P$10 or P$15 so let’s take P$5.5 as a “split the difference” between low bar and high bar poverty). (Even if we take the penurious “dollar a day” line (updated for inflation) it is roughly the same set of countries with just more weight to the poorest large countries).
These countries cumulatively have 2.7 billion P$5.5 a day poor people and hence, if one were going to address global poverty one would have to do so by addressing it in these countries.
Conversely, if your development issue/tool/learning is not addressing the important development concerns of these countries it is not really an important item on the development agenda.
Now, just imagine you had an opportunity to make a presentation to the leadership of any of these developing countries (where leadership could be political, intellectual, civil society). And here is your pitch: “Development economics has an important new tool, RCTs, and with that tool a good example of what we have learned about development is that giving away free shoes doesn’t really work, and secondarily, if you do give free shoes anyway, give the kinds of shoes that kids really want to wear.”
I can imagine two responses.
One. “Really? That is what you have to offer? This is what you think development is about? You have come to our country (India/Indonesia/Nigeria/Ethiopia/etc.) and what you think we care about, our vision of the future of our nation and people, our goals, dreams and ideals, our vision of national development, hinges on the effectiveness of the charity of an American shoe company? Is there any way you could have been more condescending to us, the leadership of this country, than taking up our time talking about giving away shoes?”
Two, suspicions about your own judgment, in two senses. First, how did you personally come to be spending your time talking about something so trivial? Second, in what way did you think economics or development economics “learned” from this RCT? The standard economics (from theory and long empirics of the impact of in-kind transfers) would have expected the impact of a pair of free shoes to be roughly the re-sale value, which, even for a poor person, is likely small as even for a household at the P$1.9 per day poverty line annual income is P$2777 so a pair of shoes in a small increment so what is new in a Bayesian sense? Second, standard economics (from theory and empirics) would have suggested that if one were going to give something in-kind as charity choosing something the household wanted would be best (if only to reduce the losses from transactions costs of re-sale).
| Country Name | Total population | Headcount | People Poor | P$1.9 per day rank | ||
| P$5.5 per day | P$1.9 per day | P$5.5 per day | P$1.9 per day | |||
| Total | 5093 | 2764.4 | 506.7 | |||
| India | 1311 | 82.3% | 13.4% | 1078.6 | 176.0 | 1 |
| China | 1376 | 27.2% | 0.7% | 374.4 | 10.0 | 11 |
| Indonesia | 257 | 67.0% | 7.2% | 172.1 | 18.5 | 7 |
| Nigeria | 182 | 90.0% | 47.8% | 163.8 | 86.9 | 2 |
| Pakistan | 188 | 78.0% | 5.2% | 146.6 | 9.8 | 12 |
| Bangladesh | 160 | 84.8% | 15.2% | 135.8 | 24.2 | 5 |
| Ethiopia | 99 | 85.5% | 27.0% | 84.6 | 26.7 | 4 |
| Congo, Democratic Republic of | 77 | 97.1% | 72.3% | 74.8 | 55.7 | 3 |
| Philippines | 100 | 64.2% | 8.3% | 64.2 | 8.3 | 13 |
| Egypt, Arab Republic of | 91 | 61.9% | 1.3% | 56.3 | 1.2 | 20 |
| Tanzania | 53 | 91.0% | 40.7% | 48.2 | 21.6 | 6 |
| Mexico | 127 | 37.9% | 3.3% | 48.1 | 4.2 | 15 |
| Brazil | 207 | 19.4% | 3.4% | 40.1 | 7.0 | 14 |
| Kenya | 46 | 86.6% | 37.3% | 39.8 | 17.2 | 8 |
| Myanmar | 53 | 67.6% | 6.4% | 35.8 | 3.4 | 16 |
| Uganda | 39 | 87.4% | 39.2% | 34.1 | 15.3 | 9 |
| South Africa | 54 | 57.1% | 18.9% | 30.8 | 10.2 | 10 |
| Vietnam | 93 | 31.6% | 2.3% | 29.3 | 2.1 | 19 |
| Sudan | 40 | 59.1% | 7.7% | 23.6 | 3.1 | 17 |
| Iraq | 36 | 56.0% | 2.2% | 20.2 | 0.8 | 22 |
| Colombia | 48 | 28.7% | 4.5% | 13.8 | 2.2 | 18 |
| Morocco | 34 | 30.0% | 0.9% | 10.2 | 0.3 | 23 |
| Iran, Islamic Republic of | 79 | 11.8% | 0.4% | 9.4 | 0.3 | 24 |
| Turkey | 78 | 11.5% | 0.3% | 9.0 | 0.2 | 26 |
| Peru | 31 | 24.3% | 3.6% | 7.5 | 1.1 | 21 |
| Thailand | 67 | 7.1% | 0.0% | 4.8 | 0.0 | 29 |
| Argentina | 43 | 8.1% | 0.6% | 3.5 | 0.3 | 25 |
| Ukraine | 44 | 7.8% | 0.1% | 3.4 | 0.1 | 28 |
| Malaysia | 30 | 2.9% | 0.0% | 0.9 | 0.0 | 30 |
| Korea, Republic of | 50 | 1.2% | 0.2% | 0.6 | 0.1 | 27 |
As in the previous entry, if I had chosen the TOMS shoes give-away impact evaluation as an example of an RCT and what they can do and why they are a contribution I would have been rightly criticized as concocting a straw man. But I didn’t.