“Random trials are undemocratic in nature”

Princeton University economist Angus Deaton on how to evaluate welfare schemes

trithesh

Trithesh Nandan | April 8, 2013


Prof Angus Deaton
Prof Angus Deaton

For the development economists, the randomised control trials (RCTs) are the latest fad. Imported from medical research into the field of economics, many economists including Abhijit Banerjee and Esther Duflo have been heavily using RCTs into their work. Banerjee and Duflo’s book, ‘Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty’, highlighted the reliance on RCTs to find out if poverty interventions programmes are effective or not.
But
Angus Deaton, the Dwight D. Eisenhower professor of international affairs and economics at Princeton University, is critical of this method. During a lecture in New Delhi last year, he hit out at the ‘randomistas’. Talking to Trithesh Nandan, he says that RCTs pose potential pitfalls for research. Prof. Deaton also questions the assumptions behind a pilot project and a survey in a Delhi locality that formed the basis for cash transfer across the capital and a forerunner to the direct benefit transfer (DBT).
Edited excerpts from the interview:

Is India evaluating the impact of its welfare initiatives properly?

I don’t know much about evaluation here. My main work is more about studying poverty in India. In the last few years, I have questioned the randomised control trials (RCTs), the latest tool of development economists. In a way, we are getting too much dependent on the RCTs. It gives an average result. We should avoid doing this.
I would say that RCTs may identify a causal connection in one situation, but the cause might be specific to that trial and not a general principle. Blind trust in RCTs leads to overconfidence and lack of sufficient scrutiny of potentially bad evidence.

Also see the Howard White interview: Get more bang for development bucks

Let’s take a concrete example: the debate on public distribution system (PDS) vs. cash transfers. The Delhi government did a pilot project, and now the central government too is rolling out the direct benefit transfer in phases. How do you see such pilots and trials?
Policymakers also give the logic that the cash transfer mechanism has worked wonderfully in the Latin American countries. The sample size at one small place (Raghubir Nagar) can’t form the basis for whole Delhi. There's no validity to generalisations about the data. It is not the ‘gold standard’ for the whole population of Delhi.
My experience says that RCTs are often conducted on a convenient sample. It is not random at all. It’s like designing a better lawnmower (and who wouldn’t want that unless you’re in a country with no grass or where the government dumps waste on your lawn?). There is no guarantee that a perfectly designed RCT will work in any other context. I have also argued with Prof Abhijit Banerjee of MIT on NYU Development Research Institute website on the same lines. In a Rube Goldberg machine, flying a kite sharpens a pencil, but kite flying does not normally cause pencil sharpening. 
In order to estimate the standard error, the RCT could be bootstrapped to calculate its distribution and, therefore, its standard error. You can’t substitute machine with people. My concern on RCTs is all about this.

Why do politicians here favour such surveys and pilots?
Politicians also tend to base policy decisions on the outcome of RCT research. If politicians do things we don't like with research based on RCTs, that's another argument for being careful about which RCTs we use or which questions we ask.

Development economists are also increasingly using the method.
In development economics, there is a movement towards the use of RCTs to accumulate credible knowledge of what works, without over-reliance on questionable theories or statistical methods. When RCTs are not possible, this movement advocates quasi-randomisation through instrumental variable techniques or natural experiments.

You had a debate on this with Prof Banerjee.

Yes, the debate has been going on. Banerjee could not debate his points properly. However, there is no one-on-one discussion on this subject and there is no need to do it. Past development practice is seen as a succession of fads, with one supposed magic bullet replacing another—from planning to infrastructure to human capital to structural adjustment to health and social capital to the environment and back to infrastructure—a process that seems not to be guided by progressive learning. The economists of the Abdul Latif Jameel poverty action lab (J-PAL) at MIT are more often advocating the use of RCTs.
Duflo of the J-PAL argued that randomised trials of projects would generate knowledge that could be used elsewhere, an international public good. The J-PAL lists dozens of completed and ongoing projects in a large number of countries, many of which are project evaluations. The World Bank is now using substantial numbers of randomised trials, and the methodology is sometimes explicitly requested by governments, who supply the World Bank with funds for this purpose.

Duflo even compared development economics before the advent of RCTs to medieval doctors using leeches.

It is an extremely ridiculous argument. I will tell you why with this example. When two independent but identical RCTs in two cities in India find that children’s scores improved less in Mumbai than in Vadodara, Duflo and other authors state that this is likely related to the fact that over 80 percent of the children in Mumbai had already mastered the basic language skills the programme was covering. My question is that how ‘likely’ is established here, and there is certainly no evidence that conforms to the ‘gold standard’ that is seen as one of the central justifications for the RCTs.

Banerjee also described RCT as “a new economics being born”.

I have already argued against RCTs in my answers. The RCT is basically undemocratic in nature. The technique is never a substitute for the business of doing economics.

Is it Princeton vs. MIT?
No. J-PAL is home to this experiment. My point is that it is not a panacea.

So would you also advise as Chris Blattman goes so far as to advising young development economists to move away from RCTs?
Yes, of course. The RCTs is not at all a magic bullet for development economics.

You have said that pharmaceuticals trials are corrupt in nature. How?
RCTs are widely used to test different medicines or procedures. Nonetheless, it is argued that this method is unethical because the mean treatment effect estimated is not very useful to predict individual outcomes. For a specific doctor facing a specific patient, the average outcome of a randomised controlled trial will often be unhelpful. The physician usually has some theory of how the drug works and also an understanding of patient, who might, for example, be elderly, frail, overweight, and ex-smoker, with a history of responding to some medications and not others. Therefore the physician will often not prescribe a drug that passed its randomised controlled trial with flying colours and instead prescribe one that did less well but that is a better fit for the patient. At the same time there is much concern that those who sponsor trials and those who analyse them have large financial stakes in the outcome, which sometimes casts doubts on the results. Much of medicine is not ‘evidence-based’, for good reason. That is why I say these trials are corrupt.

What would you suggest as an alternative?
I shall argue that the analysis of projects needs to be refocused towards the investigation of potentially generalisable mechanisms that explain why and in what contexts projects can be expected to work. Since the RCTs are not very good at ‘mechanisms’ (developing theory), instrumental variables or natural experiments seem to be a better approach.

When did you decide to take on ‘randomistas’?
It is now a decade, I think, the same time when J-PAL economists started spreading their work based on RCTs. I felt it was not a genuine work. My main concern with current practice in development economics is that these links are weak and that much empirical work makes no attempt to investigate mechanisms. In the last few years, I have become more vocal about their work.

You wrote the book, “Data and Dogma: The Great Indian Poverty Debate”, in 2005. The debate is still on. What is your take on this issue?
It is no longer the great Indian poverty debate. There are still more people living below the poverty line than what the planning commission estimates. There have been discrepancies between surveys and national accounts; the effects of questionnaire design, reporting periods, survey non-response, repair of imperfect data, choice of poverty lines, and interplay between statistics and politics. So coming up with a more reliable estimate of India’s poor goes a long way towards getting a better estimate of poverty rate. India is home to the most undernourished people in the world, so the policy makers have to act assertively.

Before the announcement of Nobel Prize in Economics 2012, media reports said your name was in consideration. Any regrets?
I am now relieved. The press made a small issue mountain out of a molehill. There were several contenders for the award.

Comments

 

Other News

Vanni: In struggle of memory, pictures complement words

Gotabaya Rajapaksa, the new president of Sri Lanka, won the November elections that were held amid continued polarization in the island nation. As defence secretary, he had led the government’s decisive fight against the LTTE rebels in 2009, in which most of the guerrilla leaders were killed. Thus en

How Sahitya Akademi promotes India lit abroad

Sahitya Akademi, the autonomous organization under the ministry of culture, has been silently promoting Indian literature abroad through a host of activities, including promotion of translations. The Akademi has helped translate much-talked about Bangla novel ‘Herbert’ by Nabarun

Hotel responsible for vehicle in valet parking: supreme court

Once a guest hands over the car keys to the valet, the vehicle is under the hotel’s responsibility, and it will have to compensate any damage to it, the supreme court has ruled. In a judgment under the contract law, a bench of justices Mohan M Shantanagoudar and Ajay Rastogi has upheld

Homebuyers have filed 1,821 cases against builders

Amid oversupply, slackening demand and financial troubles, many builders have not been able to deliver homes to buyers, leaving the latter in the lurch. The buyers in turn have taken recourse to filing cases against builders: a total of 1,821 cases under the Insolvency and Bankruptcy Code (IBC) 2016 were p

Rajya Sabha looks back and forward on historic occasion

The beginning of its historic 250th session gave the upper house of parliament an opportunity to take stock of its past and its future, the challenges ahead and the ways to meet them. The house on Monday took up a discussion on ‘The Role of Rajya Sabha in Indian Polity and the Way Forw

Kartarpur Sahib: The corrdidor can show the way to peace

When the corridor to the Gurudwara Darbar Sahib, in Kartarpur (Pakistan), was opened for the pilgrims from India on November 9, prime minister Narendra Modi recalled the fall of Berlin Wall, which took place on the same date 30 years ago. The Berlin Wall collapse not only reunited the two Ge



Archives

Current Issue

Video

CM Nitish’s convoy attacked in Buxar

Opinion

Facebook    Twitter    Google Plus    Linkedin    Subscribe Newsletter

Twitter