Whither M &E in IFAD projects? – A Modest Proposal

By Steven Schonberger

Where I come from, in Oakland, California in the US, if you want to raise the temperature in a room you just have to mention the “immaculate reception”; the moment in 1972 when the Pittsburgh Steelers football team scored a winning touchdown against the Oakland Raiders in what we Raiders fans are sure was an illegal play which eliminated them from the playoffs and a potential Super Bowl victory (similar to Diego Maradona’s “mano de Dios” against the English in the 1986 World Cup quarterfinals).  In IFAD, I have noticed that a similar catalyst of passions is the mention amongst CPMs of the Results Indicators Management System, or RIMS which can turn any meeting into an exchange of frustrations and complaints.  My sense, however, is that this is not necessarily a response to RIMS, per se, but rather to the difficulties of implementing M&E systems in general in donor financed projects.  In this case, IFAD’s RIMS is perhaps “the best of the bad”.  Fortunately, in M&E, as in the case of football, there are always more games to play and opportunities to do better if we can improve our game.  Following are a few ideas of how we might do that.

In July, we benefitted from a visit by  Rachel Glennerster, the Executive Director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachussetts Institute of Technology which has gained international renown for implementing and advocating the use of Randomized Control Trials (RCTs) to assess the impact of development activities ranging from the use and willingness to pay for improved mosquito nets to incentives to keep girls in school to the household impacts of access to microfinance.  Drawing on this experience, Rachel emphasized some key points of importance to IFAD, including:

·         The trend is that RCTs are becoming more common as the methodological standard expected by funding agencies;

·         The good newsis that improved approaches to establish control groups are broadening the situations where RCT can be applied;

·         The bad newsis that some funders are insisting on more use of RCTs than is either useful or feasible;

·         The challengeis to know when it makes sense to use rigorous, RCT-type approaches and what other types of approaches should be used based on what information is needed and what information can be realistically generated.

A key outcome of the subsequent discussion was that donors, including IFAD, invest a good deal of funding and effort in what are called ‘impact evaluations’ (RIMS level 3)  but which do not provide credible assessments of impacts.  At the same time, there is a great deal more that can be done to strengthen the basic output reporting and more qualitative assessments of how outputs are delivered and how the recipients respond to this. 

Regarding the first point on impact evaluation, the consensus in the room was moving towards what Chris Blattman, a Professor at Yale University advocates:  ‘Do R&D, not M&E’.  Essentially the idea is that impact assessment (level 3) should be focused on broader learning to determine what approaches generally work and under what conditions.  This requires a focused research program which develops carefully defined initiatives and employs rigorous, RCT-type evaluations of these initiatives conducted under different conditions in different regions (but not all projects!).  This  also requires an open attitude which does not focus on success or failure but rather on learning from the results whether they are what was expected or not (in fact, most of the great scientific advances in history were the unexpected, not predicted, results of experiments, but that is grist for another blog…) The objective is to generate something approaching generalizable conclusions regarding questions such as ‘ what is the best way to encourage increased use of fertilizer by smallholders to improve yields?’, or ‘what types of institutional conditions result in sustained empowerment of women in local, rural governance?’.   The result of this effort is an enhanced toolkit of ‘tested’ approaches and the conditions under which they should work which we can apply with some confidence of success.

The second point, regarding reporting, is that we need to focus more on monitoring to ensure that projects are doing the basics of transforming IFAD funding into goods and services for smallholder farmers, and, perhaps even more importantly, ensuring that these are of the right quality and delivered in a manner which meets the real needs of the target group.  Geoffrey Livingston, Sara Delaney and I wrote a paper about this for IFAD’s Smallholder Agriculture Conference in February of this year where we argued for much more attention to the challenges of spatial and temporal coordination of development activities, or more colloquially what we called ‘Right place, right time development’.  

So now for the modest proposal:  Let’s reorient our project M&E to  levels 1 and 2 where we can make a difference and use the information – specifically in monitoring the timely conversion of funding into goods and services for the target group through normal project reporting and due diligence.  Let’s then ensure that local universities or NGOs are hired to do annual ‘client satisfaction’ surveys regarding farmer’s views regarding the timeliness and quality of what they receive as well as how they are using the assistance provided.  As Geoffrey, Sara and I argued, simply addressing this part of the log frame or results chain would already likely lead to much better outcomes.

And what about impact studies?  Let’s ask those RCT-oriented members of the Board (and perhaps some interested foundations) to finance an internal IFAD fund for carrying out RCT and similar, high quality research on key issues for IFAD programs.  The IFAD fund could plan to provide finance and direct technical support for 2 to 4 RCT evaluations per year per Region which respond to priority issues for IFAD as a whole.  We can then share these results with partners, such as J-PAL, IFPRI, FAO, the World Bank and others in building up the collective rural development tool kit which program design could draw upon.  Of course we will continue to innovate and experiment beyond what is in the toolkit, but with more attention to designing these innovations in ways that facilitates evaluating if they work, and under what circumstances, we can strengthen the basis for the type of scaling up which is emphasized under IFAD 9.

And what will be the indicator for monitoring and evaluating this new M&E approach?  Simple - unlike Oakland Raider fans, we will ‘get over’ our frustrations with M&E and focus that energy on something more useful, like how we get effective agricultural finance working…. What do you think?