Definition

A friend recently asked ‘what exactly is an impact report’. Good question, but with many answers. if you live in Australia (or probably any other Western democracy), google the name of your local council and search the site for the latest annual report. Near the front will be a statement called ‘Our Vision’. It will probably say something like “….a flourishing community living in harmony with the natural environment and achieving a sustainable and dynamic economic base”. The annual report itself will say how much the new library cost and the number of meals on wheels delivered, but the impact report happens at the next election when the council is either voted back or a new one given a go.

It's a statistic that surely needs to be considered. That the International Labour Organisation

It’s a statistic that surely needs to be considered. When the International Labour Organisation states that more lives are lost in the workplace annually than in war, and that doesn’t even count the number of lives made miserable at work. When will we acknowledge that ‘impact’?

At the National level, there is probably some document that talks about ‘honouring our international obligations’ which gets translated into having an overseas aid budget. A proportion of this aid budget then might be given to a Non Government Organisation (NGO), who has a vision saying something like “We will help the local community flourish, protect and enhance their environment and achieve improved economic livelihoods”. A problem is that it is not the “beneficiaries” of this gesture who vote, but the folks back home. And the Government wants to tell them that their taxes have been spent efficiently, effectively and made a difference. This is where a different form of impact report comes in, and which generally means trying to describe, define and measure something as ephemeral as a ‘flourishing community’.

The standard way to do this is through logframes (sometimes known as ‘theory of change’) and associated indicators. Indicators of achieving a ‘flourishing community’ could be fewer people getting sick, or increased life expectancy. This is fine as long as the situation is relatively simple. Collect the records from the local clinic, construct a borehole, check the clinic records again. Where it gets tricky is working within a complex environment. In other words, how do you claim attribution (or responsibility) that it has been your bore hole that has reduced diarrhoea when there are three other NGO’s doing the same thing and there is nobody in the clinic actually reporting why people are being admitted other than being ‘sick’, and/or another NGO has been training local people to treat simple ailments at home and the weather has been seriously wet?

I think I've heard this mantra too many times to take it seriously. Unless it means helping people have an enquiring mind, supporting them to try new things, celebrating failure.

I think I’ve heard this mantra too many times to take it seriously. Unless it means helping people have an enquiring mind, supporting them to try new things, celebrating failure?

A question for today regarding both situations is whether the ‘impact report’ provided then results in any real learning and change. Or do we just keep trying to do the same thing (albeit with computers and smart phones) whilst hoping for a different result? That different result being not just about bore holes and libraries, but genuinely achieving a ‘flourishing community with a stable economy living in a sustainable natural environment’? And if this is the case, then what do we need to change for us to make a real difference in how we live, work and play? My own tentative suggestion is to enable more people in on both the decision making and the subsequent assessment process of the impact of that decision. “WE made this decision and on reflection it’s sort of worked here, but could have been better there”.

Brick Wall

Travelling too fast (possibly even recklessly!), come to a corner and slide into a brick wall, only to be left picking up the pieces – metaphorically speaking. Describing the ‘impact’ that an organisations activities have through a staff perspective proved to be a bit like hitting a brick wall at speed. Not the desired impact.

The staple food in Malawi is ground maize meal, cooked into a thick paste called nsima. It’s sometimes referred to as ‘food’ by the locals, because anything else is considered merely ‘relish’, or an accompaniment. Unfortunately whilst nsima is incredibly filling, it is almost completely nutrition free. The consequences of this cultural phenomena are dire. The UNDP writes “Malnutrition remains a challenge and the single biggest contributor to underweight children under the five years of age and child mortality. If the current trend continues, about 32 percent of children will be underweight by 2015 which is 18 per cent more than the (Millenium Development Goal) target.” These children are malnourished because they’re fed an exclusive diet of nsima. Food production and eating are cultural activities, and people only change accepted cultural practice or tradition at their own pace.

Grace scooping nsima out of the pot. Each glutenous pod being a serving.

Grace scooping nsima out of the pot. Each glutenous pod being a serving.

How we describe the ‘truth’ is equally subject to cultural tradition. Wikipedia states that ‘There are differing claims on such questions as what constitutes truth: what things are truthbearers capable of being true or false; how to define and identify truth; the roles that faith-based and empirically based knowledge play; and whether truth is subjective or objective, relative or absolute.’ In other words whether one prefers the truth as described by a science undergrad or that of Hunter S Thompson is in itself one of cultural tradition. In a complex world with multiple truths, both are more or less right. Difficulties generally arise when the belief is that there is only one truth, in much the same way as believing there is only one food. Despite being a truism, what one believes to be true, is true. Unfortunately a critical factor in todays world is who has the power to define the truth, – even when describing how to identify and describe truth itself.

One story, a type of truth. Livelyhood project staff were asked 'what legacy of your project would you hope to find 2yrs after you've left an area?'. They were then asked 'what chance do you expect your project would achieve each aspect?'. Across the projects, answers were: Economically Empowered Households 70% chance; Effective Community Lead Advocacy 50%; Improved Well Being 70%; Effective local institutions 50%.

Multiple stories leading to an agreed truth. In a workshop I asked Livelyhood project staff  ‘What legacy of your project would you hope to find 2yrs after you’ve left an area?’. After grouping their answers, I  then asked ‘what chance do you expect your project would achieve each aspect?’. Of those in this photo, answers were: Economically Empowered Households 70% chance; Effective Community Lead Advocacy 50%; Improved Well Being 70%; Effective local institutions 50%.

So, whether an impact report based on staff perceptions of ‘impact’ is more or less true than something with figures and an impression of objectivity is in itself subjective. I say ‘impression’ of objectivity because whilst ‘objectivity’ is the dominant culture of describing ‘truth’, it is still only a cultural tradition, and possibly only relatively appropriate when describing change in a highly complex environment with infinite factors and parameters. So, a bit like telling a Malawian that nsima isn’t that good for them, I’m left surveying the wreckage caused by attempting to describe an organisations ‘impact’ in a different manner to the accepted tradition. Which leaves me wondering how and where to start putting things back together.

Learning Workshop

Last week I co-facilitated a ‘learning workshop’ with Adam the new Deputy Country Director, for the four livelihoods projects in CU.

We structured the program through looking at four of the OECD measures of program success: Efficiency, Effectiveness, Sustainability and Relevence.

The definition of sustainability caused problems, for the OECD definition relates quite specifically to environmental sustainability during the project lifetime, but the common usage in the organisation defines it more as ‘legacy after the project is completed’. We went for the latter.

I introduced World Cafe as a technique for the session on effectiveness, causing shocked silence when I demonstrated how participants should write on the table cloths (they presumed that the white clothes were the venues, but relaxed when assured that we had bought them for the purpose).

Tenthema (of the sugar out growers capacity building project) giving the discussion summary of the Effectiveness Successes World Cafe table.

Tenthema (of the sugar out growers capacity building project) giving the discussion summary of the Effectiveness Successes World Cafe table.

At the end of the first day, Adam did an outside exercise of brainstorming and ranking peoples big learnings for the day. He was so impressed with World Cafe that he wanted to use it again on the second day for participants to explore how the top four learnings should be incorporated into new project proposals. I was really impressed with what participants came up with for one of these; and I particularly liked the last sentence of point 10, which I think sums it all up for any development organisation.

Adam directing participants on the brainstorm & ranking exercise used at the end of the first day

Adam directing participants on the brainstorm & ranking exercise used at the end of the first day

EMPOWERING LOAL STRUCTURES TO CARRY OUT MOST OF THE ACTIVITIES – LESS PARTICIPATION BY CU STAFF

  •  CU to play an advisory/mentorship role in assisting beneficiaries or local organisations through awareness, sensitization, training instead of doing things for them

  • Identify local organisations/beneficiaries, social groups to work with

  • The identified local organisations need to participate in the need assessment e.g PRA (participatory research & analysis)

  • Need to assess their capacities e.g. financial, technical, social, literacy etc

  • Build local organisations capacity to address their  identified gaps

  • Promote self assessment of local organisations

  • Provide strategic sub-grants to local organisations

  • Develop coherent M&E to track the management of sub-grants

  • Community Advocacy should be emphasized e.g CAI

  • Focus entry point should be through CBOs (community based organisations) as they are more focused on what they are doing. Will require change of CU system

Impact Report 2

I have just run the second workshop to gain staff input to the Impact report, this time with half of the WASH members. There are seven WASH (Water Sanitation and Hygiene) projects implemented by CU, which account for 23% of the total project expenditure, so I was pleased to have a few hours of their time as part of a Whole of WASH (WoW) get together.

WASH staff look at posters describing activities and outcomes across the whole of CU

WASH staff look at posters describing activities and outcomes across the whole of CU

I again used the poster process, but this time remembered how successful it was in ’19 points of connection’ to have sticky dot likert scales associated with each poster. These sticker dot surveys have given some Interesting insights into the development world.

It looks as if there is more  support for water provision than sewage.. funny that.

It looks as if there is more support for water provision than sewage.. funny that.

Junaid Ahmad is senior director for the water global practice of the World Bank Group. He maintains that interventions that focus on collective behavior change and shifting social norms are vital in this war on open defecation. ‘In rural India, there are places with signs that say: “We will not give our daughters in marriage to someone from a village that doesn’t practice total sanitation.” This is the type of social movement that is required. In addition, political will at the highest level is absolutely fundamental…..’ Unfortunately Malawi is unlikely to make it’s Millennium Development Goals in this area, mainly because according to UNDP, there is not sufficient high level political will in the country.

OK, so it was WASH staff commenting on agriculture, but almost everyone in the country grows maize.

OK, so it was WASH staff commenting on agriculture, but almost everyone in the country grows maize.

I am pleased to see that there are some attempts at innovation, which when I consider the lack of political will to enact any change to carbon producing behaviour in other parts of the world it all makes Malawi look positively dynamic.

 

Impact Report 1

All aid agencies want to demonstrate that they’ve made an impact. My current big job is to write the 2013 -14 Concern Universal Malawi Impact Report, which has been (and continuous to be) an interesting exercise. The problem is……..multitudinous.

Concern Universal Malawi is a complex organisation – a few points to illustrate:

  • CU Malawi’s goal is (as is normal today) an outcome viz: ‘to have made a lasting improvement in peoples lives’.
  • It currently implements approximately 23 diverse projects at various stages of completion.
  • It has eleven operating centres (offices) and works in nine Regions across Malawi
  • Funding for projects and programs comes from approx. 21 different donors. Each donor has it’s own agenda and reporting requirements.
  • There is a combination of different projects and sub-projects, sometimes with different donors.
  • There is encouragement to work with ‘partners’. With no definition of what a partnership is, there are multiple working relationships with other bodies.

With each project having it’s own goals, M&E and reporting system I have approached the problem using grounded theory, looking at the data available and seeing what emerges. I now have information and stories across nine different themes from all projects.

  • Advocacy and Social Inclusion
  • Agricultural Production
  • Economic Development
  • Environmental Sustainability
  • Farm and other inputs
  • Health and Nutrition
  • WASH (Water, Sanitation & Hygiene)
  • Working with Others

In addition, ’Impact Reports’ are necessarily subjective (as they report on the broader system, which is by definition value laden), and dependant on not just the authors but also the intended recipients. The main intended audience of such a report are the donors, but with a desire that it should be ‘rigorous’ and ‘not just a PR exercise’. I am thus also running it as a development exercise for staff, which I hope will allow them to better understand the organisation they work in, and also provide the opportunity for them to say what they want to say with regard to ‘impact’.

Using posters developed for each theme (and the ‘poster process’) I have now taken it to one team (Sugar Producers Capacity Building) for their reflections and input. Along with others, their input will then contribute to the ‘learnings’ for a document later tailored for the donors. Maybe… one day… we could get staff and donors in the same room…….

Staff comments on Environmental Sustainability activities: Reduction in tree cutting because of using solar lamps and CM stoves

Staff comments on Environmental Sustainability activities impact: Reduction in tree cutting because of promoting solar lamps and CM stoves

Staff comments on ‘Pro-poor’ Economic Development activities - Fairly good impact on linking farmers to lucrative markets as witnessed by the involvement and contribution of Fairtrade:  Excellent impact on Village Savings & Loans (the numbers speak for themselves and they are screaming): More projects featuring economic development activities. Great pickup on stoves by communities

Staff comments on ‘Pro-poor’ Economic Development activities – Fairly good impact on linking farmers to lucrative markets as witnessed by the involvement and contribution of Fairtrade: Excellent impact on Village Savings & Loans (the numbers speak for themselves and they are screaming): More projects featuring economic development activities. Great pickup on stoves by communities

 

Thumbs up by the Sugar Project staff for the poster process (carried out after they’d had a long day in Fair Trade training)

Thumbs up by the Sugar Project staff for the poster process (carried out after they’d had a long day in Fair Trade training). Note 3 of 13 posters on wall behind.