Measuring impact, maximizing resources: A strategy for effective development

We all learn from experience. In fact, real learning is impossible without some degree of trial and error. But when IFAD and its partners invest in projects to help rural households lift themselves out of extreme poverty, mistakes can be costly in both financial and human terms. That’s why it’s crucial, from the start, to have tools at hand that effectively measure the impact of every intervention.


Herder reviews pasture rotation map with a project
evaluation officer in Mongolia. ©IFAD/Susan Beccio

Impact evaluation was in the spotlight at IFAD headquarters this week during a seminar organized by IFAD and the International Food Policy Research Institute (IFPRI). It was the second in a series of research-oriented events co-sponsored by the two organizations.

While the content of the seminar was highly technical, even non-experts could grasp the importance of its key question: How can we make the best possible use of the resources we spend on development?

A credible case for support

Speaking by videoconference from Washington, DC, Maximo Torero, Director of IFPRI’s Markets, Trades and Institutions Division, offered a detailed strategy in response to that question. Answering it, he said, means assessing not only how cost-effective programmes are, but also how well they generate economic development and reduce poverty for beneficiaries on the ground.

Only by gathering evidence-based knowledge about what works, Torero argued, can aid agencies make a credible case for more donor support.

“It’s important to identify which interventions make sense and which ones don’t, and why,” he said.

Objective and systematic
Torero’s rapid-fire explanation of impact evaluation covered a lot of ground, as did the ensuing question-and-answer period. (Watch the full seminar here.) Following are a few of the presentation’s main themes.

• Causal links. Over the past decade – and especially amidst global recession and fiscal constraints – governments and other donors have increasingly demanded proof of the impact made by development policies and programmes. To meet these demands, aid organizations must clearly identify the causal pathways between interventions and results, and systematically gauge the impact of their work.
• Before and after. Objective impact evaluation hinges on a difficult task: determining what would have happened if the project in question never existed. This requires a comparison of indicators for two separate groups – beneficiaries and non-beneficiaries – both before and after the intervention. 
• Control groups. Controlled trials are one approach to making such comparisons. The trials compare randomly selected members of a beneficiary group with their counterparts in a non-beneficiary control group. There are other methods of comparison, as well. No approach is perfect, but each can be helpful in addressing the issue of attribution – that is, finding out whether changes in the status of beneficiaries are attributable to a particular intervention.
• Outcomes vs. impacts. When choosing indicators to be used for evaluation, it’s important to recognize the distinction between the outcomes and the impacts of a development project. A rural electrification initiative, for example, might result in improved indoor air quality, as households use fewer polluting fuels for cooking, lighting and heating. In this case, better air quality is the outcome. From a human development standpoint, however, the relevant impact could be a reduction in respiratory diseases and deaths among children under five.
• Design for success. To ensure an accurate assessment and to reduce costs, the impact evaluation should be designed at the inception of a project, not imposed after the project is already operational. Optimally, interventions can be evaluated at the pilot stage and then brought to scale on the basis of a rigorous analysis of lessons learned early on.

Torero returned to this last point several times during the seminar, stressing the need for “external validity” in project design and implementation. Even with large, complex projects, he said, “you need to have some testing at the pilot level to understand which interventions have the greatest impact. We can try things and see if they’re working, and once we’re sure, scale them up.”

Evaluation is, of course, simply a means to an end. For IFAD, the ultimate goal is to make a tangible, positive impact on the lives of poor smallholder farmers in the developing world. As Torero put it: “We’re not looking for a silver bullet. We’re trying to get people out of poverty on a sustainable basis.”

Missed the seminar? Watch the recording.

PowerPoint presentation

Beyond the Arab Awakening: Food security and conflict in the Middle East and North Africa

As popular discontent swept across much of the Arab world over the past year, some observers were puzzled. “How is it,” asked the World Bank’s World Development Report 2011, “that countries in the Middle East and North Africa could face explosions of popular grievances despite, in some cases, sustained high growth and improvement in social indicators?”

Last week, experts from IFAD and the International Food Policy Research Institute (IFPRI) took a stab at answering that question in a panel discussion at IFAD headquarters in Rome. Their observations were based on early results from a three-year research project, co-funded by IFAD and IFPRI, examining the nexus of rural development, food security and conflict.

The study’s initial phase covers 24 “Arab-plus” nations, comprising all the members of the Arab League plus Iran and Turkey. The panellists emphasized that it is a work in progress and said the presentation was, in part, an opportunity to solicit feedback from colleagues who brought their own expertise to the table.

Those caveats aside, the researchers’ preliminary conclusions were provocative. While conditions vary from country to country, they said, indicators of well-being in the region are not as positive as official figures suggest. What’s more, Arab nations may be exceptionally vulnerable to conflicts associated with food insecurity and poverty.

A proxy for poverty
Given the severe rates of youth unemployment and rising commodity prices in many Arab states, there’s little doubt that economic grievances triggered at least some of the unrest they’ve experienced. Yet the official statistics issued by governments show a steady decline in poverty amidst economic growth.

At the panel discussion, IFPRI research analyst Perrihan Al-Riffai presented evidence that the government data may be flawed and incomplete. To set an accurate baseline for analysis, she explained, the IFPRI researchers have selected a proxy for official poverty figures. Rather than income, they’re looking at the rate of stunting among children under five – because stunting is a clear marker for vulnerability and food insecurity at the household level.

“The nutritional indicator is more comprehensive than the poverty indicator,” said Al-Riffai. “It really paints a different picture. When we plot the nutrition-growth relationship, we find that the Arab-plus countries are not doing as well as the rest of the world in reducing poverty through economic growth.”

This analysis matters, Al-Riffai and other panellists argued, because reliable data provide the only sound basis for evidence-based budget and policy decisions that will reduce poverty and promote stability.

Risk of food insecurity
The figures on child stunting also shed new light on food insecurity in the Arab world. Based on the proportion of revenues that each country spends to import food, the researchers found that the risk of food insecurity ranges from “low” to “serious” across the region. When the child-nutrition data are factored in, however, the risk shifts toward “alarming.”

Three countries in the Horn of Africa – Sudan, Somalia and Yemen – are among those registering the highest risk. And the rural poor are the most vulnerable population overall.

As IFPRI senior researcher Clemens Breisinger pointed out, investments in agriculture usually reduce poverty more cost-effectively than any other form of development. Yet smallholder farming has not thrived in the Arab-plus countries. In fact, farming is no longer the main source of income for the region’s poorest rural households, who now earn more than 80 per cent of their income from non-agricultural activities.

“Agriculture is globally the most pro-poor sector, but not in the Arab world,” said IFPRI senior researcher Clemens Breisinger. “There are some serious issues of growth not reaching the poor.”

Links to conflict
Another panellist, IFPRI senior researcher Jean-François Maystadt, went on to highlight some of the links between lagging agricultural development and conflict.

In general, said Maystadt, periods of political transition heighten the risk of violence between groups within a society, and recent events around the Middle East and North Africa surely seem to verify that point. More broadly, he said, the Arab world faces an ongoing risk of conflict stemming from myriad factors. These include weak institutions and a demographic ”youth bubble,” as well as dependency on oil exports for foreign exchange.

In some places – such as Somalia, the subject of a case study by the research team – climate change also fosters conflict by depriving rural populations of their traditional livelihoods.

But in many Arab countries, the disconnect between overall economic growth and persistent rural poverty may be the most inflammatory factor of all. “Rising food prices increase food insecurity,” said Maystadt, “and this increases the risk of conflict.”

Basis for investments
These finding suggest that a broad-based approach to food security, focused not only on agricultural development but also on stimulating the wider rural economy, could mitigate the risk of future conflicts in Arab nations. More study is needed to determine which policies will have the greatest impact.

To that end, the IFAD-IFPRI project team will supplement its initial research with visits to IFAD programmes and interviews in the field. In addition, the team will use its newly developed Atlas for Rural Development and Conflict, a comprehensive digital database, for further analysis and modelling. Their on-going work will concentrate mainly on Egypt, Gaza and the West Bank, Iraq, Jordan, Lebanon, Somalia, Sudan, Syria, Tunisia and Yemen.

Ultimately, the goal of the research project is more practical than academic. “It will give us a more quantified and empirical basis for our investments,” said IFAD Country Programme Manager Omer Zafar. Over time, informed by solid research, those investments should make a real difference in rural lives across the Arab world.

• Missed the presentation? Watch the recording
• Video interview with Perrihan Al-Riffai, IFPRI research analyst
• Video interview with Jean-François Maystadt, IFPRI senior researcher
• Powerpoint presentation