India has no dearth of government schemes and programmes aimed at delivering development in general and welfare to the needy. But have they delivered development and made a difference in the life of a poor family? More importantly, do we raise these questions to begin with – questions of evaluating the impact of development/welfare initiatives?

Howard White, an economist and Delhi-based executive director of International Initiative for Impact Evaluation (3ie), says simply monitoring outcomes is not enough, what is needed is impact evaluation. In an interaction with Trithesh Nandan, White explains this model and what it can do for India. Edited excerpts from the interview:

How does impact evaluation of development programmes matter for a country like India?
Impact evaluation matters enormously for all countries developing or developed, particularly for the countries with large numbers of social programmes. In India, the state and central governments launch social programmes to try and help different targeted groups. The rationale or, say, motivation for these programmes is to alleviate poverty and improve the conditions of these groups.

What impact evaluations seek to know is whether these programme are helping the targeted group or not. The government of India and state governments have been always commissioning evaluations, outcome monitoring or studies of certain social programmes. But these studies have not been designed to answer the crucial questions like what difference the programmes make to people’s lives. It is only impact evaluation that can answer this particular question.

So, there is evaluation and then there is impact evaluation. What is the difference?
Impact evaluation uses a particular method to assess the programmes. It also seeks to know what happens in the absence of the programmes. Then you compare two things to know the difference between intervention and non-intervention due to the programmes. Technically, we say that what happens is the factual description and what happens in the absence of the intervention is the counter-factual. And it is the key to impact evaluation. Normal evaluation is just factual analysis of the programme and different kinds of problems. But with counter-factual, you can compare what happened with the programme and what would have happened in the absence with the programme. And the comparison allows you to say what difference the programme made. Forty years ago the value of having good-quality measures of intervention benefits would not have been questioned. Now it is more questioned. So you need more scientific tools like impact evaluation.

Can you give a few examples?
There have been lots of impact evaluations done in the last decade or so around the world. Some of them definitely happened to be in India; for example, those concerning reservation in gram panchayats for women and whether women as the heads of gram panchayats make a difference in the allocation of village resources. The survey was done randomly. When you do random assignment of the programme, then you know that you are comparing two different sets of gram panchayats. To know the counter-factual you need to place those gram panchayats with other gram panchayats where such thing was not there. Then you come to know the voting behaviour of these gram panchayats.

Who did that study?
This study was done by the Abdul Latif Jameel Poverty Action Lab (J-PAL) of the Massachusetts Institute of Technology (MIT). This group does such impact evaluations throughout the world. The survey was done across the country, I believe.

Do you mean to say that through impact evaluations, you know exactly what you want from the programme and what kind of beneficiaries you want to target?
It is more fundamental. Through impact evaluation, you know the programme is working or not or say the particular programme making difference. In other evaluations, you don’t ask such kinds of questions: Is this programme working or not? Is it doing what it does? Just to understand, take the case study of the community-related sanitation programme. Why do you want to do that? You have poor sanitation practices which result in diseases among young children. Experts would look at child diarrhoea. They would say that the disease has decreased with the practice of improved sanitation. If you did not use the counter-factual approach, you would just see what happened in the child diarrhoea programme. The instance of child diarrhoea goes up and down throughout the year. You can’t just particularly look at what happened in the programme and say that it changed because of the programme. How could you find that because of the change of the programme? You can do so with the impact evaluation; you would compare the village where you have the programme and comparable villages where you don’t have the programme. Then you find out if these programmes really make a difference. Such studies are going on in Odisha. It is carried out by 3ie, and not yet completed.

The other one that 3ie is doing on the health insurance in Maharashtra and Gujarat. We are looking if health insurance is improving maternal and child health. We are doing 20-odd studies in India on a range of sectors.

Is impact evaluation based on ‘randomised control trial’ (RCT)?
Absolutely yes. It is not only randomised controlled trials but also a particular approach to designing evaluation. The approach is that if you can’t do it, you do it this way. The way the RCT works is that you chose programmes in 10 districts of the state that can be divided into 50 or 60 sub-districts. You chose random, you chose half of the districts of the programme and the other half not for now. Then you can use it for comparison group. You select randomly which one you do it and which one you don’t. That’s how RCT works. Many of our programmes are RCT.

But of late economists like Angus Deaton of Princeton University have been criticising RCT.
I have great respect for Prof Deaton but here he is simply wrong. In my view, RCTs can give powerful messages in terms of effective development programmes. The best programme of impact evaluation was the conditional cash transfer in Mexico. The programme started in the early 1990s and money was given to poor households. With the introduction of cash transfer, girls started attending schools and pregnant women went to hospitals. When the government decided to do impact evaluation, it decided to do it through RCT. The success of the programme led the Mexican government to roll it out nationally. Most Latin American countries copied the programme. Now African and Asian countries are doing similar programmes on the basis of impact evaluation. It is a social programme and shows that this design worked.

In India too, the debate on the Latin American experience is picking up with the introduction of direct benefit transfer.
The critics of impact evaluation and other evaluation have a point that you can’t necessarily generalise the issue. Mexico and India or Mexico and Cambodia are very different. Because the programme worked in one place doesn’t necessarily mean it will work in another place. You have to develop the context or you have to develop a context that is sufficiently similar so that if the programme worked here it will also work there. In the Latin American countries, the large number of impact evaluation on cash transfer and findings are very similar. But if you go to Zambia, Malawi or Zimbabwe, you will find something different. I won’t say if CCT worked in Mexico; it will definitely work in Delhi. Or I won’t say that the CCT is better than PDS on the same evidence. The PDS is well established for last several decades. If the new system works better, you have to experiment with. You have to be very clear that these are two different sets of programmes with the same objective. Then you try to find out which is the most cost effective. And here impact evaluation can give you the answer.

What I can tell you here is that the first generation impact evaluation questions whether it works or not. Because the second generation impact evaluation questions are about changing design of the programme like how it works better. Through CCT, you want to see how it has helped in school enrolment. Perhaps it makes a difference: instead of giving cash through the year; you give bulk of money at the time when school fees are asked. That’s more likely to help in development.

How would impact evaluation look at MNREGS, one of our leading welfare initiatives?
Think of some programme design modification. Then you can randomly assign different districts. It might be about information on right documentation, government officials on how to administer the programme. You think of indentifying the barrier to access and put in different schemes to overcome those barriers.
 
Which countries are using impact evaluation to improve development programmes?
The Latin American countries have been successfully using the method. In Mexico and Brazil they make tough decisions based on impact evaluations. Social programmes in countries like Mexico and Colombia legally require impact evaluations to secure funding.

On using impact evaluation, where would you place India on a scale of 1 to 10?
I would put India at 4.

Why?
You have to ask the government of India. In 2010, India approved establishment of an independent evaluation office (IEO) to undertake impartial and objective assessment of the various public programmes and improve the effectiveness of public interventions. Sadly, there is no progress after that.

India is a huge country with large and diverse groups, can impact evaluation work in diverse country like India?
The Indian government should seize the opportunity to become international opinion leaders by kicking (out) some of the flagship programmes that seem to have international relevance and substituting those with high-quality impact evaluation to contribute to international knowledge about which programmes work. It will benefit not only India but also internationally.

Can aspect of governance measured through impact evaluation?
Yes, it can. Governance has different layers. At the local level in India, you do social audits of gram panchayats. You can always bring in accountability measures by using impact evaluation for proper use of public funds. The next level is voting behaviour. We are financing a programme in Rajasthan and Delhi which is a voters’ education programme. It is a scorecard approach to inform voters about performance of their elected officials in terms of participating in debates and votes and how elected officials work in gram panchayats and comparisons with other gram panchayats.

At the national level, the impact evaluation can be done on voters’ perceptions of MPs’ behaviour. A lot of governance issues can be addressed through impact evaluation.