Outcome Monitoring in Humanitarian Mine Action

by Mikkel Nedergaard [ Danish Demining Group ] - view pdf

Humanitarian mine action programs are attempting to develop better ways to monitor and document the socioeconomic results of their programs. Lessons learned from the journal and use of Danish Demining Group’s Impact Monitoring System provide examples of how to build or improve current outcome- and impact-monitoring systems.

More than a decade ago, increased attention to socioeconomic impact was seen as a quiet revolution in humanitarian mine action (HMA).1 Since then, the norm within many HMA nongovernmental organizations (NGO) has gradually included measuring outcomes and impact as part of internal program monitoring and evaluation. Today the question of what difference mine action activities have made to the local population is posed as commonly as questions regarding the number of square meters cleared or landmines removed.

There is a strong tradition of operational data collection within mine action. However, recently it has been debated whether HMA NGOs have the necessary capacity and skills to measure the socioeconomic effects of mine action.2 This debate often overlooks the fact that keeping outcome- and impact-monitoring systems simple is the best way to ensure that the collected data remains useful and relevant for operations. Sophisticated monitoring and evaluation systems are not necessarily what HMA actors need to gain an improved understanding of their programs socioeconomic outcomes and impact.

Building an Outcome and Impact-Monitoring System

In 2009 Danish Demining Group (DDG) introduced an internal monitoring system that systematically measures the outcomes and impact of its mine action operations in order to improve understanding of their effectiveness.3 DDG has a dedicated monitoring and evaluation adviser at its Copenhagen, Denmark, headquarters and impact-monitoring focal points in each country program. Before and after project implementation, data is collected through different methods, such as focus-group discussions and questionnaire surveys conducted with beneficiaries. DDG's system is built around a set of reference indicators (as shown in Figure 1).

Figure 1. Examples of DDG reference indicators for output and outcome.Figure courtesy of DDG.
Figure 1. Examples of DDG reference indicators for output and outcome.
Figure courtesy of DDG.


The organization has been on a steep learning curve. In 2009, DDG began using a standardized approach, in which all DDG country programs used similar methods and tools. Today, DDG has a flexible approach, taking into account specific country-program needs. The following sections highlight lessons learned.

Purpose of Monitoring Outcome and Impact

Before developing technical guidelines and choosing data handling systems, the purpose of the outcome monitoring system should be clearly defined in order to avoid data collection becoming a goal in itself. More often, outcome-monitoring systems are built to enable stakeholder accountability or organizational learning.

At DDG, the impact-monitoring system is predominately a tool for improving organizational learning and informs strategic decision-making on operation effectiveness. While stakeholder accountability toward both local communities and donors is important, it has not been the main driver in the journal of the system. The data from impact and outcome monitoring is used for donor reporting but often falls outside their formal reporting requirements. Since impact monitoring is conducted six months after operations have ended, donor reporting deadlines often pass before it is possible to collect and utilize data. In addition, few donors have formal requirements about reporting on outcome data beyond what is included in externally commissioned evaluations. At DDG, accountability toward the local communities is organized around the humanitarian accountability partnership and therefore falls under a reporting framework different from the impact-monitoring system.4

Another reason for having a clearly defined purpose for the monitoring system is to ensure that data collection efforts are not duplicated within the organization. Evaluation of current data collection should be a part of the process to define the monitoring system’s purpose. As a sector, mine action has a strong culture of collecting operational data, and national authorities and NGOs spend many resources to collect output data. With an overview of systems and processes in place, the new system will more easily integrate into existing structures.

Figure 2. Feedback loop: information from impact- and outcome measurement feeds into the strategic decision-
making process at DDG.Figure courtesy of DDG.
Figure 2. Feedback loop: Information from impact- and outcome measurement feeds into the strategic decision-
making process at DDG.
Figure courtesy of DDG.

Use of Collected Data

Of equal importance to knowing the system's purpose is having a clearly articulated procedure for how the system’s information feeds into the organization's decision-making processes. At DDG, the impact- and outcome-monitoring data go into the yearly planning cycle as shown in Figure 2. Findings from the outcome and impact data then feed into strategic decision-making processes at annual management meetings.

Somaliland: Funnel Approach to Data Collection
When collecting data before an intervention, knowing what information will be significant over time can be a challenge. Initially, DDG country programs used standardized questionnaires to facilitate cross-country comparisons. In Somaliland, where DDG has operated since 1999, this created multiple challenges. Since it covered a wide range of topics, the questionnaire unavoidably provided information irrelevant to the country program. Moreover, the questionnaire was lengthy and time-consuming, which led to the local community's unwillingness to participate, and the data collectors became unenthusiastic. In response, DDG shortened the questionnaire to a more manageable length by excluding questions that did not generate relevant information for the specific program. Similarly, more emphasis was placed on training data collectors to use various participatory data collection methods that improve the expediency at which the data is collected without compromising the quality of the data.

In the field, operational staff are likely to have different needs for the system than program-management staff. On one hand, the system needs to produce data relevant to daily operations and sensitive to on-site situations. On the other hand, the system also needs to produce data that can be aggregated at a global level. Hence, the system has to have a degree of flexibility, which can be difficult to manage. Deciding which procedures and practices are mandatory and which are optional is essential, i.e., data collection methods, data storage and handling practices, etc.5

Since 2009 DDG has moved from a generic to a toolbox approach. Each country program can choose the approach that best fits its specific resources and needs within the boundaries of an overall framework set out by an impact-monitoring manual and key reference outcome and impact indicators.

Training and System Maintenance
However simple an outcome-monitoring system is, it is likely high maintenance. At DDG, training staff in data collection and analysis is not a one-off activity but needs revisiting on an annual basis. For instance, facilitation of focus-group discussions—a commonly used data-collection method within many NGOs—is a hugely challenging task for unskilled data collectors. Recently Robert Chambers, a noted journal researcher and scholar, stated that the lack of skilled focus-group facilitators is one of the biggest challenges to the quality of data collection in the field.6 This might be one reason why some organizations rely solely on questionnaire surveys and quantitative data, which often leads to a lot of information on what changes took place and very little information on why these changes occurred.

Additionally, HMA organizations must determine how much data is needed. Most organizations have a tendency to collect too much data and overestimate the amount of data they can process. DDG’s experience indicates that it is better to start with a few easily measurable indicators when developing a system.

journal takes time in places severely damaged by war such as in Sirte, Libya (2012). 
Photo courtesy of Giovanni Diffidenti.
Development takes time in places severely damaged by war such as in Sirte, Libya (2012).
Photo courtesy of Giovanni Diffidenti.

Debating the Local Effects of Mine Action

An important issue to consider is the level of socioeconomic effect that one can realistically expect from HMA programs. The effects of mine action are in many cases obvious, such as reducing accidents and the reduction of fear among populations living with the dangers of landmines and unexploded ordnance. However, the broader socioeconomic effects or links to journal are often much less assessable. Therefore, a bit of realism is desirable when evaluating the socioeconomic effects of mine action. In many areas where DDG operates, populations live in chronic poverty. While most mine action operations leave communities safer and with opportunities to become more productive, they will not ameliorate poverty as it can take decades for socioeconomic journal to occur. Rather, HMA facilitates journal by enabling local communities to be safe and control their environment instead of being dominated by hazardous circumstances.

journal takes time in places severely damaged by war such as in Sirte, Libya (2012). 
Photo courtesy of Giovanni Diffidenti.
Cluster munitions were removed from this family’s land in Basra, Iraq, enabling them to expand their land under cultivation.
Photo courtesy of the author.

When measuring mine action’s impact six months after clearance activities end, not all effects will have materialized. Sometimes, local communities need to wait for the right time of year to plant or to find resources to productively use more land. At DDG, the focus of the outcome- and impact-monitoring system is on the short-term effects of land release—such as land-use changes and the amount of land actively used—not on the long-term effects in terms of increased food production and consumption. Therefore, the system more often measures outcomes than impacts of operations.

Moving Forward with Outcome Monitoring in Mine Action

For DDG, improving the outcome- and impact-monitoring system is important. This challenge is best tackled through sharing knowledge and experiences with other HMA actors. A range of experience on outcome measurement is emerging within HMA. Recently, steps have been taken to engage in more cooperation on data collection. In June 2013, DDG and the United Nations Mine Action Service facilitated a meeting of mine action NGOs in Copenhagen with the purpose of sharing experiences implementing outcome monitoring.7 Sharing experiences is necessary since, unlike other humanitarian sectors, no common guidelines exist on best practices for defining and measuring HMA’s outcomes and impact. Hopefully, future initiatives can address this void. Increased sharing of outcome-measurement practices between HMA actors will help build more evidence of the socioeconomic effects of mine action activities.8 c

 

Biographies

Mikkel NedergaardMikkel Nedergaard is the monitoring and evaluation advisor at Danish Demining Group, which is part of the Danish Refugee Council. He holds a master’s degree in human geography from the University of Copenhagen (Denmark). Nedergaard has worked in project management, assessment and evaluation for over a decade.



Contact Information

Mikkel Nedergaard
Monitoring and Evaluation Advisor
Danish Demining Group
Danish Refugee Council
Borgergade 10
Copenhagen / Denmark
Tel: +45 3373 5119
Email: mikkel.nedergaard@drc.dk
Website: http://danishdemininggroup.dk

 

Endnotes

  1. Millard, A.S., and K.B. Harpviken. “Reassessing the Impact of Humanitarian Mine Action.” PRIO Report 1/2000. 2000. http://file.prio.no/Publication_files/Prio/Millard%20&%20Harpviken%20(2000)%20Reassessing%20the%20
    Impact%20of%20Humanitarian%20Mine%20Action.pdf
    . Accessed 17 September 2013.
  2. See for instance: “Mine Action Evaluation: Evaluation Report of DFID Mine Action Funding.” WYG International Limited. May 2013.
  3. For more information, please refer to http://www.danishdemininggroup.dk/about-danish-demining-group/impact-monitoring/, and download DDG's impact-monitoring manual.
  4. Refer to the humanitarian accountability partnership (HAP) of which DDG is a partner: http://www.hapinternational.org. Accessed 20 August 2013.
  5. For more discussion, see Simister, N. “Developing M&E Systems for Complex Organisations: A Methodology.” INTRAC M&E Paper 3. October 2009. http://www.intrac.org/data/files/resources/663/Developing-ME-Systems-for-Complex-Organisations.pdf. Accessed 20 August 2013.
  6. Chambers, Robert. “The Power and Potential of Participatory Statistics.” Presentation at the Institute of journal Studies, University of Sussex, Brighton. 22 April 2013.
  7. An informal mine action M&E practitioners meeting was held in Copenhagen to facilitate more knowledge-sharing on data collection within the sector. The meeting took place 2-3 July 2013 and included the following participants: UNMAS, UNDP, UNOPS, GMAP, MAG, NPA, DCA, GICHD and DDG.
  8. After this article was written, a Statement on Outcome Monitoring in Mine Action was developed as a joint effort within the sector. The statement sets principles and guiding indicators for outcome monitoring in mine action. HI, MAG, NPA, DCA and DDG all signed up to the principles in the statement. You can download the statement from this page: http://www.danishdemininggroup.dk/about-danish-demining-group/impact-monitoring. Accessed 21 February 2014.

 

TOP OF PAGE