Counting the Uncountable: Measuring the Benefits of MRE

by Robert Keeley [ Danish Demining Group ] - view pdf

The aim of this article is to propose concepts that define MRE in terms of its outcomes and beneficiaries and to open a discussion on developing a means that may then be useful in measuring the efficacy of MRE. This article defines efficacy as the ability to produce a desired or intended result.

A mine risk education session led by Danish Demining Group.All photos courtesy of Willaim Vist-Lilles√łe, Danish Demining Group.
A mine risk education session led by Danish Demining Group.
All photos courtesy of Willaim Vist-Lillesǿe, Danish Demining Group.

Mine risk education (MRE) is an integral component of mine action and the International Mine Action Standards (IMAS) defines it as “Activities which seek to reduce the risk of injury from mines/ERW [explosive remnants of war] by raising awareness of men, women, and children in accordance with their different vulnerabilities, roles and needs, and promoting behavioural change.”1

Under this definition, MRE is (1) an educational process and (2) intended to reduce casualties through behavior modification. This is a simplified version of the comprehensive definition included in IMAS but is useful as it frames some of the concepts necessary for measuring the benefits of MRE, which is commonly measured in terms of its activities (e.g., the number of posters printed) and outputs (i.e., the number of people provided with MRE training).

In measuring the benefits of MRE, there is a set of necessary ground rules. These rules should include a definition of the outcomes of MRE as a part of mine action, and a clear understanding of how the number of beneficiaries (direct and indirect) of MRE interventions will be measured.

The terminology in change modelling, logframe analysis, etc., is sometimes confusing. An outcome can sometimes be called a result (as in results-based management), and in the development evaluation criteria created by the Organisation of Economic Cooperation and Development (OECD) and commonly used by mine action donors (and sometimes by practitioners) the outcomes are considered under the term impact. The term used throughout this article is outcome.

This article refers to MRE, but the principles can also extend to all forms of risk education as practiced in mine action, including small arms and light weapons risk education.

Definitions

The following outcomes are proposed as general definitions of mine action outcomes:

As an educational process intended to reduce casualties through the modification of behavior, MRE can be expected to contribute to outcome one as set out previously. However, MRE does not contribute to outcomes two and three (except as a potential source of data). Any measurement of efficacy should therefore concentrate on an MRE activity’s contribution to a reduction in casualties.

Outputs

It is suggested that there are three main groups of activities in mine action, namely

This allows the definition of outputs for all MRE activities in terms of people reached—and not, as mentioned previously—based on materials produced. MRE is about people, not t-shirts.

Estimating the Number of Beneficiaries

Direct and indirect beneficiaries are used to estimate the total number of beneficiaries of mine action activities. While more accurate measurement techniques could be devised, additional calculations would be unwieldy and would result in diminishing marginal returns of information.

Direct beneficiaries. There are three kinds of direct beneficiaries:

Indirect beneficiaries. Indirect beneficiaries are those indirectly impacted by mine action activities and include

A warning sign illustrating the risk of ERW.
A warning sign illustrating the risk of ERW.

Measuring Efficacy in MRE

One of the problems with social science is that accurately aggregating human data is impossible without spending lots of money on big sample sizes, and there is a point in the data sampling that results in what social scientists call diminishing marginal returns of information. Alternatively, sampling and studies provide best estimates of behavior that researchers then associate with a certain degree of confidence based on statistical rules. MRE is no different.

Nevertheless, by using the concept of Theory of Change, project designers are required to show how an activity links to an output and how an output links to an outcome. As discussed previously, defining and measuring the outputs of MRE is comparatively easy, whereas defining and measuring MRE in terms of its outcomes is more difficult. However, as a form of behavior change communication, MRE’s outcome becomes easier to understand in terms of safer behavior and reduced casualties.

Knowledge, attitude and practice (KAP) surveys are used in MRE, although the effectiveness of the KAP process has varied. Some general principles from the discipline of epidemiology can be used to good effect:

A KAP survey is limited in terms of measurement efficacy. Rather, KAP surveys (conducted scientifically) are most useful for assessing a population’s knowledge while being less useful for assessing attitudes and practices. This is due to the phenomenon of people striving to give the right answer to survey questions.

Economists define survey results as being stated preferences as opposed to revealed preferences, which are determined by other means (counting people entering a public toilet and later measuring the amount of soap used over a week). When measuring the efficacy of MRE, revealed data yields more accurate results than stated data in terms of attitudes and practices, while surveys are the best means of measuring knowledge.

Measuring Attitudes and Practices

A handout with pictures of different types of ERW.
A handout with pictures of different types of ERW.

Monitoring and measuring any increase in the reporting rates of landmines or other ERW avoids the problems of diminishing marginal returns of information and stated preference, and is a comparatively simpler way of measuring the efficacy of MRE in terms of attitude. Therefore, if the MRE is efficacious, it will result in more people understanding the need to and the process of reporting items, and will deliver an increase in the level of reporting. This can be easily measured by epidemiological processes, such as the use of control groups (areas that have not been subjected to the same MRE activities). It also allows researchers to recognize that all MRE content is not created equal, and that the content of one MRE package can have different results than another. Furthermore, an increase in reported data is revealed rather than in stated data, and is hence more reliable.

It is difficult to stand at the edge of a minefield and count people conducting unsafe behavior, but we have a measure of efficacy in terms of the casualty numbers. This is a proxy indicator (and does not measure behavior directly); however, given that the intended outcome is a reduction of casualties, this seems a reasonable one to use. The question is then one of results attribution—to what extent is the reduction in casualties attributable to a particular MRE project, to what extent is the reduction attributable to other MRE activities, and to what extent might a reduction in casualties be attributed to other interventions?

Mine action programs would benefit from the ability to establish casualty rates at a national and provincial level. Using casualty data for the areas where MRE intervention takes place, organizations could make a correlation between the intervention and any reduction in casualties.

This is, however, a best-case estimate because a correlation does not necessarily mean a causal relationship, and there are other things that can significantly affect casualty rates outside of providing MRE. Similarly, poor MRE delivery is unlikely to be responsible for behavioral changes; yet, if there is a reduction in casualties, the poor MRE project is likely to get credit for the reduction. Thus, a survey of landmine casualties is a useful way of determining whether or not victims had received MRE before their accident as well as what MRE they may have received.

In such circumstances, it may be possible to infer a causal relationship using areas that have not been exposed to MRE as a control. For example, in two districts where there was no reduction in casualties, and in a third district where there has been MRE and there was a fall in casualties, it is possible to suggest a relationship between the fall in casualty rates and the provision of MRE.

This is a work in progress and Danish Demining Group is in the process of developing an Efficacy Tool Kit to help measure the efficacy of MRE. The development and use of this tool kit will be the subject of a subsequent article.c

 

Biography

Bob KeeleyBob Keeley is a former British Army Bomb Disposal Officer who has worked in humanitarian mine action since 1991. He has worked in a number of countries at both a technical and program management level. His Ph.D. was on “The Economics of Landmine Clearance” and was completed in 2006. Since June of 2014, he has worked as the chief technical advisor for Danish Demining Group, where amongst other projects, such as the design of an mine risk education (MRE) project in Ukraine, he has also been responsible for the re-drafting of DDG’s MRE standard operating procedures.


Contact Information

Dr. Robert Keeley
Chief Technical Advisor
Danish Demining Group
Division of Emergency Safety and Supply DRC/DDG International Department
Borgergade, 3rd floor
Copenhagen / Denmark
Tel: +44 (0)787 999 0700
Email: bob.keeley@drc.dk
Website: http://www.danishdemininggroup.dk

 

Endnotes

  1. See IMAS 04.10 at http://bit.ly/1fQj93Z.