RBM and Theories of Change

by Russell Gasser - view pdf

Figure 1. The results-based methods pocket toolkit.All graphics courtesy of author via resultsbased.org.
Figure 1. The results-based methods pocket toolkit.
All graphics courtesy of author via resultsbased.org.

“Give me your money; I’m busy doing things” is not the most convincing fundraising appeal. Instead, “Look at the difference our program has made to the lives of the people that were helped” is far more likely to get a positive response. The overall purpose of mine action is to improve people’s lives and livelihoods, to reduce casualties, and increase compliance with political commitments like the Anti-Personnel Mine Ban Convention (APMBC). Although this is widely known, standard reporting excessively includes information on how many people received risk education, how many square meters of land were cleared, or how many people attended a training session. In other words, they report activities instead of outcomes. This information tells the donor that the mine action organization was busy but does not tell donors whether their money made positive, long-term differences in beneficiaries’ lives, nor does it help donors understand if the work was good value for money overall. To make a difference, implementing organizations need to know what success looks like and how to report it correctly. Even more important is the need to learn from experience, avoid repeating errors, and identify good practices and clever solutions for future use. Successful mine clearance is not measured by how many mines were removed but by the overall impact on the beneficiaries, the local communities and nations where they live, and how much the organizations involved were able to learn and implement continual improvement.

It’s not enough to be busy. Risk education only makes a difference if as a result people change their everyday behavior. Clearing farmland only makes a difference if local people productively use the land once cleared. It’s tempting to measure time spent, but what really matters are results. This means that success can only be measured well after a mine action intervention is completed.

Figure 2. Mine action results chain diagram.
Figure 2. Mine action results chain diagram.

This is not a new issue. Results-based management (RBM) started about 50 years ago. In the 1960s and 1970s, when national governments realized that it was not useful to measure social welfare programs by how much money they spent; New Public Management began measuring the results instead. RBM uses the same ideas and is experiencing a donor-driven resurgence. Unless donors continuously insist on using RBM, a cycle occurs where RBM is promoted with great enthusiasm and then gradually forgotten over the space of several years. Once time, money, and effort are spent on programs to ensure that RBM is successfully implemented, organizations tend to return to business as usual, and the focus on RBM is reduced or lost without pressure from donors.1

RBM is a method of ensuring quality management (QM) for donor funding. The very widely used International Organisation for Standardisation ISO 9001 approach is based on QM for customer-supplier relationships. Quality is defined as ”making sure that customer expectations are met or exceeded.”2 The underlying thinking is that a business without satisfied customers has two choices: improve quality or go bankrupt. Going out of business because local beneficiaries are unhappy is not often part of the donor-funded approach. In donor-supported programs, implementers normally spend other people’s money according to an overall plan that the field-partner did not write (perhaps a national plan) in order to deliver goods or services that will help people whom the donor does not know and will never meet. No single customer-supplier relationship exists that can be implemented for overall QM or to measure value for money. In other respects, the ISO 9001 method and the seven core principles of the 2015 version can be used to improve the quality and value of donor-funded work through RBM and QM.

One of the new phrases for RBM is theories of change, which describes and summarizes how the intervention will bring positive changes to people’s lives. It describes why spending donor money will lead to activities in the field, and why this will improve the lives of local people, whereas the work plan outlines how this will happen.

AlAlthough there is an argument among academics about the exact definition, theories of change for mine action as well as weapons management and destruction activities are evidence-based and have three, widely-accepted core parts:

  1. Results chains describe the elements that links inputs to results and are often drawn as a set of five or six boxes in a row (i.e., a results chain diagram).
  2. Causal links represent “if this, then that” cause and effect evidence of why change will happen.
  3. Assumptions about the context of the intervention that are necessary for success.3

All three must be included in a theory of change. A change model diagram is not a theory of change by itself.

Table 1. Key indicators in mine action.
Result level names Intended changes Indicator difficulty
Impact or Strategic Objective Changes to beneficiaries, lives at a community or society level Hard.
Long delays.
Often very indirect information.
External factors very important.
Unrealistic in less than three years.
Intermediate Outcome or
Medium-term Outcome
Change in behavior by people
influenced indirectly
Moderate to hard.
Can be indirect or very indirect.
Delay after activities end.
External factors will be important (attribution issues).
May include qualitative and hard to analyze.
Important for determining success.
Immediate Outcome Change in behavior by people
directly in contact with outputs
Moderate.
Short delay.
Usually directly available.
External factors quite important.
Often quantitative or pseudo-quantitative.
Important for determining success.
Output Anything we make or buy Easy or moderate.
Needs to be designed into activities.
Equivalent to Kirkpatrick level 2 evaluation.
Activity What we do Directly available from project management
information - in well organized activities,
minimal effort needed.
Does not indicate success.
Inputs Quantity of resources used Standard accountancy and audit.

Historically, planners wrote about the hypothesis of the intervention.4,5,6 Most donor-funded programs are, in essence, social science experiments. However, instead of working with volunteers (or rats) in a lab, we are spending real donor money to achieve behavioral changes in the lives of real people. Ideally, if the project delivers the planned activities, then people will change their behavior. This is so familiar that planners and implementers sometimes forget it is an experiment. If we deliver risk education, then people will change their high-risk behavior. If we clear this land, then people will use it to grow food. If we train people in planning skills, then a better national plan will not only be written but also implemented. Sometimes we make a false assumption that project inputs will automatically lead to the desired results. For example, a project must actually verify that people change their behavior after receiving risk education in order to show that the theory of change is correct. Typically, this can only be done long after the end of a short-term project, which is often a problem when donors want evidence of success immediately. A full year of an agricultural planting and harvesting cycle may be needed to show that the cleared land was put into productive use. The deeply ingrained idea that mine clearance has an impact as soon as the land is handed over needs to change. Similarly, donors should understand that the evidence of success they need cannot immediately be produced as soon as funding ends.

Table 2. Difficulty of collecting data for mine action indicators
Typical
IMPACT
Economic
Political
APMBC, CMC
Health
Wellbeing
Health
Economic
Political
APMBC, CMC
Health
Economic
Economic
Typical
OUTCOME
INDICATOR
Land in
productive use Value of crops
Injuries averted Anxiety
reduced
Injuries averted Number of UXO reported Injuries averted Anxiety
reduced
Number of
decisions/plans
improved by IM
GIS information
Typical
OUTPUT
INDICATOR
Square meters
of safe land
UXO destroyed Number of
people informed
Number of
tons (or items) safely stored
More and/or
better
information
ACTIVITY Land release Explosive
Ordnance
Disposal - EOD
Risk Education - MRE Ammunition
Safety
Management
ASM (PSSM)
Information
management
and GIS

Training projects that teach planning skills are a good example. Considerable time and many different people outside of the project may be needed to achieve the desired result, such as a well written and adopted national plan. Implementing the plan after it’s adopted is even further removed from the objective of the training. Because someone finishes the initial training does not guarantee that they will eventually write a plan that is implemented. Maybe the person finds another job soon afterward, and the plan is never written. Maybe the plan is written but rejected by the government. In these cases, the impact of the training project will be zero despite delivery of the planned activities.

Similarly, recording the number of people attending an RBM course does not indicate that an organization is following an RBM approach. Success occurs when the organization yields behavior change by using RBM (outcome). As a result, the organization is transformed into a more efficient, responsive system that learns and improves (long-term impact).

Actual projects are never simple and are often unable to deliver exactly what was intended in the planning phase. To account for this, we identify the key assumptions on which the planning logic depends. These external assumptions are issues that are beyond the control of the project. Typical assumptions include a stable political situation, the cessation of armed conflict, and the provision of adequate office space, salaries, and computers for planning staff by the national authorities so that staff do not seek other jobs. The availability of the necessary tools and supplies to start food production on cleared land is another common assumption.

Some RBM specialists argue that knowing how to measure the success of a result should be the starting point of any planning process. Unless everyone involved from donors to beneficiaries can agree on the definition of success, and the project has the means to recognize long-term success, there is a risk that the intervention will accidentally support failure. Unless success is clearly understood, defined, and measured, it cannot be properly separated from failure. Being busy is not a measure of success. Before starting field work and even before making detailed plans, project participants and funders need to define success and measure the situation. Before and after measurements are essential in order to illustrate that the project has made a difference. A single measurement at the end of a project cannot show that there has been a change. If the benefits cannot be identified or if the project does not include the necessary resources to analyze the benefits after activities have ended, it is not possible to justify claims of success. Hard work and optimism for good results is not a strategy. Positive changes have to be attributable to the activities that were funded, not just a fortunate coincidence.

Measuring the inputs to a project, the activities completed, and the resulting outputs is a matter of administering, accounting, and auditing, and should be straightforward in a well-run organization. In contrast, measuring outcomes (behavior changes in other people) and impacts (long-term societal level changes) cannot be done directly and requires the use of indicators to demonstrate what is happening. The purpose of indicators is to understand why the project developed as it did, to improve upon the project, learn from the experience, and report the project’s results. John Mayne, a leading RBM specialist, wrote “the aim of measurement is to acquire some insight and develop some comfort that the program is actually having an impact.”7 Understanding and improvement is the main goal, not justifying activities to a donor.

Identifying the difference that was made to the lives of beneficiaries and the value for money that donors can expect is essential; planning begins by first defining success. Donors increasingly expect this information and will prefer to fund implementers who include it. However, the biggest advantage of RBM is learning how to work better, how to avoid or solve problems, and how to learn and reuse solutions and skills. c

 

Biographies

Ian McLeanRussell Gasser started working in mine action 20 years ago. He has been an official of the European Commission in Brussels, an independent evaluation consultant for eight years, and until recently, was on the staff of the GICHD. His current focus is on results-based management, theories of change, and evidence-based evaluation, as well as technology for mine action and the Cynefin framework.

 

Endnotes

  1. “Easier Said Than Done: A Review of Results-based Management in Multilateral Development Institutions.” Michael Flint and Partners, March 2003. Accessed 6 June 2016. http://bit.ly/2eJGMxp.
  2. “ISO 9001:2015.” ISO. Accessed 23 September 2016. http://bit.ly/1pI39yw.
  3. Vogel, Isabel. “Review of the Use of ‘Theory of Change’ in International Development.” UK Department of International Development, April 2012. Accessed 6 June 2016. http://bit.ly/213qiz7.
  4. “Project Evaluation and the Project Appraisal Reporting System, Vol. 1.” Fry Consultants Incorporated, 24 July 1970. Accessed 6 June 2016. http://1.usa.gov/1WC8iNJ.
  5. “Project Evaluation and the Project Appraisal Reporting System, Vol.2.” Fry Consultants Incorporated, 24 July 1970. Accessed 6 June 2016. http://1.usa.gov/1U4cUZw.
  6. “Project Evaluation and the Project Appraisal Reporting System, Vol.3.” Fry Consultants Incorporated, 24 July 1970. Accessed 6 June 2016. http://1.usa.gov/1PeDnz6.
  7. Mayne, John. “Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly,” June 1999. Accessed 6 June 2016. http://bit.ly/1XxFJBT.