Impromptu Unofficial Impact study

I have recently been taken on a trip to Dowa & Kasunga Districts with the CU Water Sanitation & Hygene (WASH) team. One stop was at a village where CU have just finished rehabilitating a bore hole that had been out of action for 8yrs. On our arrival we were met by what soon became quite a crowd. I was expected to say or do something.

A reflection on how far away they lived from the borehole. On the left - not far, those on the right - a long way

A reflection on how far away they lived from the borehole. On the left – not far, those on the right – a long way

I asked them to illustrate how far they had to walk to this borehole (Short way, Medium and Long Way). On enquiry a short way was about 3-5 mins, medium about 10mins and a long way 20 -30mins. I then asked them if thinking back to eight years ago whether they had noticed any difference in their health without the borehole. I was somewhat surprised when there was quite a proportion who indicated not a lot of difference. On further explanation though, it was because that group had been going to the next village to get water, whilst the group that had experienced an increase in diarrhoea & cholera were far from there and had been drawing water from a closer unprotected water source.

Measuring difference in health. Those on the left had not noticed much difference, those on the right, considerable difference.

Measuring difference in health. Those on the left had not noticed much difference, those on the right, considerable difference.

I then asked them if they thought the rehabilitated bore hole would make much difference to their lives in terms of time or health. Again, there was a spread. We then numbered them off, and asked the small groups to consider what impact the bore hole would have and what they might do with the improved time and health.

Small groups considering what they might do with the improved health and time in their lives.

Small groups considering what they might do with the improved health and time in their lives.

The answers across the groups were generally similar, comprising:

  • Cleaner clothes & latrines
  • Improved marital relationships (this was with much laughter, but also agreement)
  • Able to make more things to sell (mud bricks, donuts etc)
  • More time for cultural events
  • HIV +ve people better able to survive

We then asked the groups to determine what they might be able to do to ensure that the borehole continues working. Answers were:

  • Make a door to keep kids and goats out.
  • Lock the pump with specified opening times
  • Work with the village leadership to have a bye law to keep kids out
  • Make sure the bore hole management committee has funds for spare parts

The Management Committee comprises of 5M and 5F representatives from the seven villages that would use the bore hole and that they had decided that there should be a common maize field that all should work on, with the proceeds being sold to fund maintenance costs. Given the variety of answers from the small groups, I’m left wondering how much support this decision actually has.

A PAR Q

Improving/Coordinating Monitoring & Evaluation (M&E) in CU Malwi is all about working in a complex system that IMO requires a Participatory Action Research (PAR) approach.

There are approximately 22 people involved in M&E over roughly 16 projects with around 10 partner agencies and 23 donor organisations. The approximations are because there are projects with separately funded sub projects, projects funded by multi-donors and people are always on the move. My first step was to interview as many of those involved in M&E as I could regarding the state of M&E in their project. My questions were framed around six elements of M&E (LogFrame, Indicators, Data base, Data Collection tools, Data Collection Process and Reporting). There was an additional three relating to performance (Challenges, Strengths and Fitness for Purpose). I took all the answers to each of these questions, stripped the authors & projects, and roughly ranked them from ‘good’ to ‘bad’.

An extract of the list of responses to the question "'Tell me about data collection in the project"

An extract of the list of responses to the question “‘Tell me about data collection in the project”

I then sent this out asking for impressions about the state of each M&E element in CU. I got 2 responses. Unfortunately I didn’t realise that this was a really busy reporting time of year, thus when I ran a workshop to get more responses, despite attendance confirmations only 4 people attended. Furthermore, the non-attendees included the three most senior project M&E staff.  In the workshop I included the comments from the first two, and thus effectively I got a response that involved all six. This means my engagement is from only 6/22 stakeholders.

Tigwirizani & Fredson responding to the lists of answers. Each table was printed out at A3 size & stapled to butchers paper.

Tigwirizani & Fredson responding to the lists of answers to each of the elements of M&E. Each table was printed out at A3 size & stapled to butchers paper, then taped to the wall.

So…. an Action Research Question.

The participants have provided what looks to me a like a quite realistic and sensible analysis of the ‘State of M&E in CU’ but it has been generated by only a minority of stakeholders. What to do next? My inclination is to ‘publish and be damned’ i.e. publish the results and see what happens. My rational mind though says that I need to keep it all in the ‘groan zone’ a bit more to allow greater dialogue and inclusion. Options I’ve been considering:

1) I could try and organise a repeat meeting & invite only those that did not contribute before.

2) Maybe there is some intermediate step I could invent around which I could invite all to attend (at a more convenient time) which still honours the work of the earlier ones whilst enabling some backtracking for the others.

3) I could approach individually the three key people to ensure I get their individual responses, and somehow amalgamate what they say with the earlier ones into a coherent response. With this option, would I include none of the work of the first group? all their work? or only their combined reflections?

Some considerations:

All four ‘key respondents’ are at different locations approx 300k from each other. Internet and telecoms systems are not capable of conference calls or Skype.

Any ideas & suggestions would be welcome.

 

Emergent Understandings

We have recently conducted a one and a half day workshop on ‘mainstreaming’ with 25 reps from six of the major projects that CU implement. ‘Mainstreaming’ is the term used to describe how the ‘cross cutting issues’ (Rights advocacy, HIV AIDS, Equality & Environment) are integrated into all CU projects. The participants were those responsible for incorporating such activities into the projects and the workshop was an opportunity for them to share practice and ideas.

Two of the learnings captured that resonated for me were:

  • Voting and verification (used in the Targeted Food Distribution) process- can it be adopted to other situations (e.g.: livelihood projects) rather than the ‘community leaders’ making decisions?
  • There is a need to collect cross-cutting stories and document them (after the event) as part of the impact report, not try to set up indicators in advance.

I was really reminded of the conceptual similarities between the voting and verification process used to identify vulnerable households (femalie or child headed, aged, HIV/AIDS etc ) and the Participatory Budgeting movement that is emerging in the developed world. Essentially, that engaging those that have a stake in  a discretionary budget gives a better result than relying on the judgement of a few ‘leaders’.

Using a simple mould, villagers can make about 10 fuel efficient stoves a day. They can then sell these for about $1 each.

Using a simple mould, a villager will take about half a day to fully make a fuel efficient stove. They can then sell these for about $1.50 each. Part of the ‘mainstreaming’ is to also encourage rehabilitation of the clay pits. 

The second observation, regarding the impact report really struck a chord with my emerging sense that while it is really necessary to establish indicators that can monitor project outputs, trying to monitor outcomes (change in the broader system) in the same way overcomplicates things. Keep the monitoring simple for implementation purposes, and check for impact by collecting stories at the end.

The Mbaula fuel efficient stove uses about one third to a quarter of the amount of firewood to cook a meal. Unfortunately if there are plenty of trees around, it is easier to continue with an open fire than find $1.50 for a stove.

The Mbaula fuel efficient stove uses about one third to a quarter of the amount of firewood to cook a meal. Unfortunately if there are plenty of trees around, it is easier to continue with an open fire than find $1.50 for a stove.j

There was another passing comment, that the workshop did not expand on, but which is dear to my heart, which concerned the efforts made to advocate environmental regeneration of clay pits used for constructing the more fuel efficient Chitetezo Mbaula stoves, but how the destruction caused by this activity pales into insignificance when compared to that caused by the road industry.

The complexity of a complex system

The last few months has been a full on presidential election campaign here in Malawi. There are around 12 presidential candidates, one of whom is the incumbent, Joyce Banda. Voting took place this last Tuesday, 20th May.  More than one African friend has expressed concern that such elections are an imposed system, and that the results do not warrant the process.

Politics is far from being the only known unknown

Politics is far from being the only known unknown. With climate change, the sign might fall over.

About three weeks ago an email went around Concern Universal advising all staff that the Tuesday was a public holiday, and that the office would also be closed on Wednesday. Staff were though encouraged to take off the whole week as part of their holidays. Monday 19th was a quiet day, though there was a palpable sense of anticipation around town. The traffic that evening was the worst I’ve seen it, verging on gridlock. The lady who takes cheese orders on a Tuesday and delivers on a Thursday sent a text to say she was going to deliver a day early this week. On Tuesday (polling day) there were some disturbances in an area about 5km from us due to administrative foul ups at the polling booths. A place near to an AVI persons workplace was stoned, and thus they decided to shut up business for the week.  Wednesday was calm, likewise this Thursday morning. The visiting Australian Commissioner sends an email inviting any AVI people in town to a catch up.

Last night one of the leading candidates house was raided with allegations of electoral fraud being raised. The newspaper headlines say the result is 50-50. Concern Universal is open again, and we planned to return to work, however AVI have instructed us to remain at the house. Apparently a press conference is due to be called at any time and the results announced. There is plenty of speculation as to what might be the outcome of whatever announcement is made, but with so many intangibles and indeterminates, nobody knows. It all reminds me of being in Victoria on a Code Red day, the only certainty is that something might happen.

Impact Reporting

My big project for the next 6 months is to develop the Concern Universal 2014 Impact Report. The purpose of most aid programs is to ‘make an impact’, and CU Malawi is one of the few NGO’s aiming to publish an Impact Report on a regular basis. The impact referred to being a change in the broader system, not a large hole in the ground as from a meteorite.

A new (to me) concept being used in the aid industry is the idea of having a ‘Theory of Change’. My impression is that this has superseded LogFrames to describe the link between what can be physically done (outputs) and the intended outcomes (or impact) in many programs. Thus my personal ‘Theory of Change’ is that positive effects in the broader system occur when individuals work together to solve tangible problems (1). This is augmented by the many others who have researched and written on the technicalities of how to help people work better together when finding solutions to common problems (2).

Putting in a bore & pump is the easy bit. What is harder is ensuring that people can access it, that there is a management committee of those that use it, that they can raise money for spare parts, that spare parts are available etc. etc.

Putting in a bore & pump is the easy bit. What is harder is ensuring that people can access it, that there is a management committee of those that use it, that they can raise money for spare parts, that spare parts are available etc. etc.

 

A  ‘Theory of Change’ that seems prevalent with many aid donors in Malawi is that WASH (Water Sanitation & Hygiene) programs have a positive impact on the broader system. This is based on considerable evidence that improved water, sanitation & hygiene practice (not surprisingly) contributes to improved community health (3). Unfortunately ‘impact’ is dependent on the many variables of a complex system.

Some donors require a large range of ‘indicators’ of impact, which might be as many as 70 questions that have to be reported on monthly throughout the length of the project. An example being “Reduction  in  reported  cases  on waterborne  diseases (cholera, diarrhoea  and dysentery)’ measured by “#  of  waterborne  diseases  reported cases”. Another being  “students with access to an adequate number of school hand washing facilities with soap” measured monthly by “#  of  additional students with access to an adequate number of school hand washing facilities with soap”. Others take a different approach, maintaining that the linkage is a given and that due to the high number of confounding factors within each individual case of improved health, monitoring this aspect is not cost effective. Some donors require nothing more than a record of the number of water points created, whilst others focus on measuring sustainability indicators, such as water point management structures, financial management capabilities, maintenance, access to repair services, and spare parts.

To report on the ‘impact’ of the work by Concern Universal (who currently implement seven different donors WASH projects, all with different indicators) is in itself complex. Thus my intended approach is to engage the CU WASH teams in a workshop process that draws on their collective data, the processes they use in implementing the projects, their understanding of ‘what works’ and document stories of success and failure. We will then use this data to compile a broad picture of CU’s impact in WASH. Stay tuned!

1: Rittell, H & Webber, M 1973, ‘Dilemmas in a general theory of planning’, Policy Sciences, vol. 4.

2: Butcher, M 2001, ‘LogFrames Made Easy’, PLA Notes, no. 41.

Department of Sustainability and Environment 2005, Effective Engagement: Building Relationships with Community and Other Stakeholders, 3 vols., The Community Engagement Project, Resources and Regional Services Division, Victorian Government Department of Sustainability and Environment, Melbourne.

Dick, B 1991, Helping Groups to be Effective, 2 edn, Interchange, Chapel Hill.

Kaner, S 1996, Facilitator’s Guide to Participatory Decision-Making, 11 edn, New Society Publishers, Gabriola Island.

NSW Premiers Department 2006, Strengthening Rural Communities Resource Kit, pdf version edn, NSW Government, Sydney.

Pretty, J, Guijt, I, Thompson, J & Scoones, I 1995, Participatory Learning and Action : A Trainers Guide, IIED Participatory Methodology Series, International Institute for Environment and Development, London.

Rosenberg, M 2003, Nonviolent Communication: A Language of Life, 2 edn, Puddledancer Press

Schwartz, R 2002, The skilled facilitator, 2 edn, Jossey-Bass.

Southern Cross University 2003, Action Research Resources, Southern Cross University, viewed 04 03 2006 2006, <http://www.scu.edu.au/schools/gcm/ar/arp/arphome.html>.

Stanfield, B 1997, The Art of Focused Conversation, The Canadian Institute of Cultural Affairs (ICA Canada).

—- 2002, The Workshop Book: from Individual Creativity to Group Action, New Society Publishers and The Canadian Institute of Cultural Affairs, Gabriola Island.

United Nations Centre for Human Settlements (Habitat) 2001, Tools to Support Participatory Urban Decision Making.

Williams, RB 1996, More Than 50 Ways to Build Team Consensus, Hawker Brownlow.

World Bank 1996, The World Bank Participation Source Book, Adobe Acrobat PDF Version edn, World Bank, Washington, D.C.

3: Esrey,S 1985,1991; Fewtrell 2005; Clasen et al. 2009,2010, Waddington 2009; Norman et al. 2010