Measuring impact, maximizing resources: A strategy for effective development

We all learn from experience. In fact, real learning is impossible without some degree of trial and error. But when IFAD and its partners invest in projects to help rural households lift themselves out of extreme poverty, mistakes can be costly in both financial and human terms. That’s why it’s crucial, from the start, to have tools at hand that effectively measure the impact of every intervention.


Herder reviews pasture rotation map with a project
evaluation officer in Mongolia. ©IFAD/Susan Beccio

Impact evaluation was in the spotlight at IFAD headquarters this week during a seminar organized by IFAD and the International Food Policy Research Institute (IFPRI). It was the second in a series of research-oriented events co-sponsored by the two organizations.

While the content of the seminar was highly technical, even non-experts could grasp the importance of its key question: How can we make the best possible use of the resources we spend on development?

A credible case for support

Speaking by videoconference from Washington, DC, Maximo Torero, Director of IFPRI’s Markets, Trades and Institutions Division, offered a detailed strategy in response to that question. Answering it, he said, means assessing not only how cost-effective programmes are, but also how well they generate economic development and reduce poverty for beneficiaries on the ground.

Only by gathering evidence-based knowledge about what works, Torero argued, can aid agencies make a credible case for more donor support.

“It’s important to identify which interventions make sense and which ones don’t, and why,” he said.

Objective and systematic
Torero’s rapid-fire explanation of impact evaluation covered a lot of ground, as did the ensuing question-and-answer period. (Watch the full seminar here.) Following are a few of the presentation’s main themes.

• Causal links. Over the past decade – and especially amidst global recession and fiscal constraints – governments and other donors have increasingly demanded proof of the impact made by development policies and programmes. To meet these demands, aid organizations must clearly identify the causal pathways between interventions and results, and systematically gauge the impact of their work.
• Before and after. Objective impact evaluation hinges on a difficult task: determining what would have happened if the project in question never existed. This requires a comparison of indicators for two separate groups – beneficiaries and non-beneficiaries – both before and after the intervention. 
• Control groups. Controlled trials are one approach to making such comparisons. The trials compare randomly selected members of a beneficiary group with their counterparts in a non-beneficiary control group. There are other methods of comparison, as well. No approach is perfect, but each can be helpful in addressing the issue of attribution – that is, finding out whether changes in the status of beneficiaries are attributable to a particular intervention.
• Outcomes vs. impacts. When choosing indicators to be used for evaluation, it’s important to recognize the distinction between the outcomes and the impacts of a development project. A rural electrification initiative, for example, might result in improved indoor air quality, as households use fewer polluting fuels for cooking, lighting and heating. In this case, better air quality is the outcome. From a human development standpoint, however, the relevant impact could be a reduction in respiratory diseases and deaths among children under five.
• Design for success. To ensure an accurate assessment and to reduce costs, the impact evaluation should be designed at the inception of a project, not imposed after the project is already operational. Optimally, interventions can be evaluated at the pilot stage and then brought to scale on the basis of a rigorous analysis of lessons learned early on.

Torero returned to this last point several times during the seminar, stressing the need for “external validity” in project design and implementation. Even with large, complex projects, he said, “you need to have some testing at the pilot level to understand which interventions have the greatest impact. We can try things and see if they’re working, and once we’re sure, scale them up.”

Evaluation is, of course, simply a means to an end. For IFAD, the ultimate goal is to make a tangible, positive impact on the lives of poor smallholder farmers in the developing world. As Torero put it: “We’re not looking for a silver bullet. We’re trying to get people out of poverty on a sustainable basis.”

Missed the seminar? Watch the recording.

PowerPoint presentation

“Happiest man in Africa…….and change is possible….”.

By Willem Bettink

I participated in the annual regional  implementation workshop of the East and Southern Africa  region. A full week immersion in Africa, to be more precise in northern Tanzania,  at the foot of mount Kilamanjaro. It was an  exciting and dynamic 4 days, not in the least thanks to “veteran” Edward Chumamoto. He facilitated -189 people- with grand style, a sense of humour and enormous skill –a true pleasure!

The project coordinators in the region decided last year to radically change the format as they felt there was not enough interaction and added value generated.  The workshop embraced the  open space methodology which generated many unexpected sessions about a variety topics from learning routes, communication and social media, gender equality and many more.

The theme of this workshop was: Managing for Impact-one of those development  concepts that appear clear at face value, but are not . As a participants said : “over these 4 days we have unpacked the concept and it has become clearer for me what it means to me in my work in our project”. 

Over the days we unpacked the managing for impact into its key elements from the perspective of a project team . One of the  results was a practical diagnostic tool referred to as the learning wheel for managing for impact. It brings together  12 elements that enable a project team to diagnose its performance.

The outcome of this diagnosis enables a team to discuss and agree on corrective actions to improve its delivery to achieve impact. One of these elements is “ continuous experimentation with new ideas and approaches as a source of innovation and performance improvement”.   

Almost all- if not all- development programmes are change management programmes. Research and our own experience  has proven  that a typical organizational change programme has a 20-30% success rate . Or if we  look at innovation and start –up businesses: only 10-15% make it through year  1 and go on to achieve sustainable profits.

If we truly believe in “managing for impact",  it implies that project teams, given the change and innovativeness of development programmes,  continuously need to perform at the top of their toes. Delivery has to be of an exceptional quality while it is conditioned by known obstacles , unplanned interferences, unexpected natural disasters and what have you.

This implies that a project team  needs to have  a strong predisposition to openly reflect upon its challenges and mistakes , harvest the learning (in particular from failed attempts ) and share  this learning with its stakeholders.

In my opinion , managing for impact is closely linked to return on investment. If we do not focus persistently on achieving impact, creating value,  we are not ensuring the pay-off to the rural poor for what they invest to engage with the changes and innovations promoted by development programmes.

A final personal note: that change is possible whether at  personal, process, technological level  I am strongly convinced off. While I was in Arusha  I witnessed the transformation of a colleague to be the happiest man in Africa.  

Managing for impact: learning from successes and failures

The East and Southern Africa (ESA) annual workshop focusing on managing for impact kicked off on 14 November in Arusha, Tanzania. The event brings together 200 participants from 17 countries. 

This is a special event on many fronts. To start with, the participants warmly welcomed colleagues from South Sudan. Secondly, they bid farewell to Ides de Willebois, who is moving on to lead the West and Central Africa division.  And last but not least, this is a unique event because for the first time, participants have embraced moving away their from traditional way of doing workshops and have embraced knowledge sharing methods.

Ides de Willebois in his opening remarks reiterated the fact this is a learning and sharing event. “Without learning and sharing we cannot improve and we cannot have impact”. He also went on to say that “we need to have a better understanding of what we do, so that we can do it better.”

In concluding his remarks, de Willebois mentioned that he is planning to start an exchange visit between East and Southern Africa and West and Central Africa. “I hope that West and Central Africa colleagues will join next year’s annual event, so that they can learn from your experience”. 

Breaking the mould

Whenever you decide to innovate or break away from business as usual, there is some apprehension. You feel gratified when breaking the mould ends up being successful and resonates with the expectations. And this is exactly what happened at the ESA event.

It was so refreshing to see 200+ participants embrace knowledge sharing methods such as openspace, spectogramme and world cafe type discussions. Thanks to these participatory methods, they started to mingle, bond and in no time they got to know each other.  

In the process, they heard many stories, including one about how ESA implementation workshops evolved from a being top-down loan administration events to more learning events owned and organized by the projects.

The majestic mount Kilimanjaro and the Kalali women dairy cooperative

As a child, when school was  over, I would be counting the days so that I could join my grandmother in Mount Damavand. Damavand is Persia’s tallest mountain in the Alborz mountain range. Grandma had these wonderful stories about mount Everest, the far way mount Fuji and then a majestic mountain in Africa - mount Kilimanjaro. 

I was intrigued by mount Kilimanjaro, because as a I child I found it difficult to fathom that Africa had mountains. So, on Monday when we had to sign-up for the field trips, I could not believe that finally I would be able to see this far far away mountain.

Unfortunately the majestic mount Kilimanjaro decided to remain pretty veiled and the clouds did not cooperate. But nonetheless, seeing it was an emotional moment and brought back many fond childhood memories.

We then moved on to visit the Kalali women dairy cooperative. This cooperative is part of the Agricultural Sector Development Programme where IFAD is investing loan funds in supplementing financing to the nationwide development programme, to counteract the depletion of the initial funds committed by development partners as part of a basket fund arrangement.

The cooperative is located in Machame division, Hai district in Kilimanjaro region. Inspired by a local daily community business, this cooperative was established with 132 members in 1988. The cooperative’s goal is to:

• increase income of poor rural households, particularly women in the area through a number of different income generating activities
• improve lives of orphans and provide assistance to poor families who were unable to provide education and sound nutrition for their children

“To get going the cooperative members contributed 1000 shillings each and payed a membership fee of 50 shillings,” explained Nancy Manasseh Kidin, the cooperative’s chairperson.

Considering their scarce income, these monies were paid in 4 installments. The cooperative started with 8 dairy cows. In 1995 they had 41 dairy cows.

Considering the vital role women play in rural societies, the cooperative decided that women headed-households were the ones to receive the cows.  They agreed that the person who receives the cow for a period of seven months would:

• give one litre of milk a day to the orphanage
• give the calves to other women headed-household and members of the cooperative 

As a result in a short time, 24 women who had signed up to this deal managed to provide dairy cows to 117 cooperative members.

Today the cooperative has a total of 260 members.

Diversifying activities

These industrious women used their income to diversify their business. In the 90s they bought themselves a milling machine.

“We provided milling service in the local market and expanded our business by buying a container to store the maize”, said Kidin.

“We also used the profits from the milling business to buy seeds.”

In 1993, they went one step further and bought themselves an electric machine and donated their diesel operated milling machine to another women group.

With the profit from their milling business they bought feed for the cows.

A terrible blow

They had a flourishing business, produced milk in abundance and sold this to a company in Arusha, thinking that a company is a reliable partner.

“We failed to ask for the money upfront and the company failed to pay us 12,000,000 schillings”, said Kidin.

This was a terrible blow for them. The community lost confidence in cooperatives and felt let down.

This mishap taught the cooperative an important lesson: never accept an “IOU”, always finish a transaction, sell, get your money and go on.

Despite this set back, they managed to back on their feet, thanks to the generous contribution of Italians.

A new door opens: dairy processing 

The Italians provided them processing instruments and that is how they started making butter and cheese.

Today they produce:

• 400-800 litres of milk a day
• 10-20kg of cheese per day
• 50-95 packs of butter per week
• 400-600 packs of yogurt per week

They keep track of their daily production and make some impressive graphs without using Excel. They also have a sophisticated booking keeping practice.

They package their products and have a registered trademark. “We sell the dairy products at Moshi, to grocery stores, to hotels and send our cheese and butter all the way to Zanzibar”, explains a proud Kidin.

The cooperative has the necessary certification to package its product, however, they are faced with the challenge of not having adequate technical support. This means they are unable to take their business to the next step and benefit from industry’s best practice.

One thing that we observed was the BEST BEFORE DATE on their products. Without exception their products seem to have a shelf life of one year!!!!

Kidin shared with us that the cooperative aspires to be able to:

• procure large 50 litre metal containers to store the milk instead of using plastic containers
• avail themselves of technical expertise and training so that their products are of high quality meet regional standards, have bar codes and can compete with Kenyan products
• have access to veterinary service
• have access to East African Community Market and get packaging machinery 
• have all the women of the community become members of the cooperative
• become a renown cooperative both inside and outside Tanzania

In conducting their business, the cooperative has learnt the importance of:

• providing thorough and in depth training to the machine operators, so that they do not only know how to operate the machine, but also maintain it properly
• understanding the market demand and their potential competitors
• doing a good market search before buying equipment and better understanding what is needed and how a piece of equipment can help them 

Expanding the business and facing new challenges

With a relatively good income from their dairy business, the cooperative bought a sunflower oil pressing machine and started a savings credit cooperative (SACCOS).

The SACCOS has 375 members and started with a capital of 3.2million schillings.

The older women of the community put their savings in the SACCOS and the more business oriented women use the SACCOS to get a loan. The loans have an interest rate of 3%.

Kidin shares with us that the SACCOS are faced with two challenges:

• underpayment
• corruption

This has led to a loss of 22 million schillings.  This loss made the community understand the importance of choosing right people as SACCOS board members. They now know that a board member:

• needs to be a trusted member of the community
• one who understands the ins and outs of the business and entire process
• one who has knowledge of financial management so that they can play a supervisory role, detect collusion and immediately take corrective actions

Investing in health and eduction of young people

The community is committed to provide education for the children and young people - especially to orphaned children and those from less advantaged families.

They use their profits to send children from disadvantaged families to school and are also assisted by Italians who are sponsoring a number of children and putting them through school.

Their vision is that every child has to go to school.

Their sustained programme of investing in education of their children is one of their great successes. Thanks to this initiative many young people have successfully finished school, obtained their degree and now serve as accountants, agriculture specialists, engineers and health workers.

Many have heard of WFP’s school feeding programme. Well, the Kalali women dairy cooperative runs a similar programme - they support school feeding by providing one glass of milk per class. 

The cooperative challenges

Kidin shared with us the cooperative challenges: 

• continuous power cuts which lasts days
• lack of access to adequate and state of the art machinery for packaging and conservation so that they are able to compete in the  market
• lack of access to technical expertise to take the business to its next level
• lack of local talents and expertise
• lack of access to appropriate equipment for conservation
• high taxation and cumbersome regulations
• competition with Kenyan products

The cooperatives aspirations

Kidin and the cooperative members aspire to:

• have all women in the area become members of the cooperative
• give dividends to all the members and not just provide contributions in kind
• create more employment opportunities
• send all children to school
• expand the sunflower pressing business
• innovate and continuously provide new services to the community
• provide capacity building
• build technical and leadership skills and groom local talents
• set up a milk bar so that community members understand that there are other options to drinking alcohol
• do more follow-ups and have a better feel what you can realistically achieve 
• develop robust business plans

We cannot but wish this group of hardworking and resilient women the best of luck. They are a great example of how when a community takes development in their hands, despite set back, they are able to move on and bring about change. I am sure the Kalali women dairy cooperative will go from success to success and manage to fulfill all their aspirations and be a model for many more women.

The show is on the road: Day one of the regional implementation workshop for Western and Central Africa

There is quite a bit of frenzy in the atrium of the ORCHID Hospitality Management Services (Ghana College of Physicians and Surgeons), Accra, Ghana where IFAD in conjunction with the Ministry of Food and Agriculture of Ghana, is holding a regional project implementation workshop for Western and Central Africa. These events present a wonderful opportunity for people to reconnect. You can see many smiling faces, people hugging and kissing and catching up with each other.

Adriana, Letizia, Christiane and others are helping with the registration. The last couple of days were quite hectic. Stefania and Martin, the two pillars of this workshop have diligently been organizing the minute details of the workshop. Yesterday we had a series of meeting to make sure that all the logistics was working like a swiss watch.

We spent all of yesterday setting up. Everyone, and I mean everyone, contributed to making sure we had everything. The gentlemen, Steven, Leo, Loko and others helped with assemblying the kits. David set up the exhibit and the publication desk. Stefania, Martin and I went were responsible for the signage and to make sure the breakout rooms were set up.

I am really excited, as tomorrow we'll be using a wide range of knowledge sharing methods. We'll start with three parallel chat shows, then we'll move to three parallel world cafes and we'll finish the day with a speed geeking session where colleagues will present their findings and learnings!

Kudos to all the colleagues in Western and Central Africa for embracing knowledge sharing methods. For sure, this workshop will be different and more engaging that previous editions!

This morning we'll have the honour and privilege of having His Excellency the Minister of Agriculture for the official opening. We'll then have a series presentation by the Director of Western and Central Africa and IFAD colleagues on the IFAD-funded project portfolio and the progress since the last implementation workshop.
So, let me sign-off at this point. More to come during the day. Internet access permitting we'll also be tweeting. So make sure you also check out the tweets both from @ifadnews and @rsamii.