Certainly, the international mine action community has learned a great deal over the past 15 years. But has mine action really made a significant difference to the people it is trying to help? Few things in mine action cause more frustration and misunderstanding among donors, recipient governments and programme managers than the question, "What results has the programme achieved?" Studies have decried that, despite data detailing the number of landmines destroyed, the area of land cleared and the number of people receiving mine awareness training, there are few data allowing an assessment of whether these achievements have enhanced the well-being of mine-afflicted communities.1
This means we can assess the efficiency and safety of mine action activities—"Has the job been done right?"—but we fall short when assessing the more difficult questions that determine "Has the right job been done?" The topics in question are as follows:
- Relevance. Are the objectives set for the programme consistent with government and donor policies, as well as the requirements of the beneficiaries?
- Effectiveness. Have we achieved the planned objectives and enhanced the well-being of people in mine-afflicted communities?
- Sustainability. Will the benefits to these people and communities last?
- Impact. What are all the consequences, intended and unintended, for better or for worse, of our mine action activities?2
Numerous attempts are under way to rectify this shortcoming, but for the most part these represent only partial measures. In fairness, the mine action puzzle is unusually complex. The community faces the challenges that arise when focusing more tightly on results, but it also faces challenges because many mine-afflicted countries represent difficult and rapidly changing environments.
The First Decade: 1988–97
Large mine action programmes were first established in the midst of, or just emerging from, anarchic "complex emergencies."3 In these emergencies, military engineers were responsible for controlling landmine contamination. When mine action addressed humanitarian concerns, the people involved understood the problem principally in terms of numbers of landmines and contaminated areas. Notable exceptions were "mine centred" (according priority to the most heavily mined areas) rather than "people centred" (investigating which hazards posed the greatest danger to people and their livelihoods).4 Basic mine awareness training was provided to at-risk civilians and aid workers. With a people-centred perspective, the clearance of hazards and the destruction of landmines are seen as means to an end, not as ends in themselves.
Making Progress? Mine Action Since 1998
In 1998, the United Nations stated, "It is not so much about mines as it is about people and their interactions with a mine-infested environment."5 At about the same time, a group of international non-governmental organizations (NGOs) active in mine action endorsed The Bad Honnef Guidelines.6 By mid-2001, mine clearance seemed to be delivering results in terms of socio-economic benefits and the scale of these benefits appeared to outweigh the high costs; however, a number of issues remained unresolved.
We can assess the efficiency and safety of mine action activities—"Has the job been done right?"—but we fall short when assessing the more difficult questions that determine "Has the right job been done?"
First, the gap between social scientists and mine action practitioners is unclear. For example, the human toll exacted by landmines and UXO was the principal impetus behind the international movement to ban landmines. But what do we know about the contributions made by clearance, marking and mine awareness to a reduction in the number of deaths and disabilities? We know very little, at least in quantitative terms.
The numbers of deaths and injuries from landmines often rise after a conflict as refugees return on potentially contaminated routes. Accidents then decline with awareness of the hazardous areas. Knowledge may come from mine risk education (MRE) or marked minefields, but more generally from seeing landmines, hearing from other community members or because some unfortunate person or animal has detonated a mine. With these factors influencing the level and trend of accidents, it is extremely difficult to isolate which factor has led to what portion of the decline in numbers. This information could conceivably be acquired with abundant data of excellent quality, but in most countries, data on landmine accidents is incomplete and often of poor quality. There is not a single study that demonstrates any statistically meaningful link between the numbers of landmine accidents and any component of mine action or mine action in general.
Working Smarter, Not Harder: What's So Hard About That?
The concept that good management is all about achieving results is far from new. Results-based management has been used for years in the public sector to improve performance because of the following:
- What gets measured gets done.
- If you don't measure results, you can't tell success from failure.
- If you can't see success, you're probably rewarding failure.
- If you can't see success, you can't learn from it.
- If you can't recognise failure, you can't correct it.
- If you can demonstrate success, you can win public support.7
With no clear bottom line, there is no simple measure to unambiguously gauge performance. Rather, a set of measures is required covering each stage in the "results chain" whereby inputs allow mine action activities to produce outputs that reach beneficiaries who use them to achieve intermediate and (ultimately) final outcomes.
Although programme managers have a fair degree of control in producing the outputs, they exert progressively less control over outcomes in the short-term, medium-term and final outcomes. First, they ensure that the outputs reach the intended beneficiaries and are used by those people in some worthwhile fashion. Second, a range of outside factors influences future outcomes. Therefore, planning for the achievement of outcomes needs to be based on these outside factors. Success depends on understanding the influence of outside factors to ensure the efficiency and safety with which outputs are produced.
If a programme clears agricultural land on the assumption that this land will be used for crops but the land is not so used, the assumption is faulty and research is warranted to determine why. In Angola it was found that "demining land for cultivation in a village surviving on subsistence farming provides little improvement if the farmers, after decades of war, have no seeds and tools left. They will go hungry despite the clearance of many hectares of productive land to a cost of maybe tens of thousands of dollars."8
Pitfalls to Overcome
Measurable results offer distinct advantages but also introduce dangers such as an over-concentration on indicators that are easily measured. These dangers may introduce "perverse incentives" leading to unintended and undesirable outcomes. A well-known example comes from education. The desire to enhance education quality has led many governments to measure school performance based on the pass rates on standardised tests. However, most schools "teach to the test," and students miss out on fundamental skills.
Similarly, in mine action, performance is judged on a narrow range of indicators. But there is growing evidence that this narrow focus on efficiency leads to less effective performance. Prime examples are the reports from many countries that minefields have been selected for clearance because they were easy to clear, allowing programmes and individual demining organisations to report that their costs were falling.
Also, proposals in a number of countries to introduce mobile teams to clear small hazards that are blocking vital community resources have faced resistance from "traditional" clearance organisations, who argue that these mobile teams are inefficient.9 Undoubtedly, such teams are inefficient if measured against the area cleared, but there is growing evidence they can be extremely effective in enhancing the well-being of target beneficiaries.10
Another problem is progress toward objectives can take time to determine. For example, Technical Advisors (TAs) are often mandated to get something up-and-running and to develop local capacities to keep it working over the long term. It takes time to determine whether local capacities have been developed to manage mine action effectively. Success in developing indigenous explosive ordnance disposal (EOD) supervision and management skills will only be apparent after the TA has departed.
Failure to address this type of problem implies that performance can only be measured against indicators that show measurable changes in the short term. In our TA example, this concern creates incentives for an adviser to focus on establishing the EOD teams even at the expense of developing indigenous capacities.
Standing Still is Not an Option
Demonstration of a programme's performance pays dividends in terms of bolstering support for the programme and learning to improve performance. These dividends multiply over time and the initial effort is rewarded.
Mine action programmes do not come to grips with how to set performance targets and to document the results they are achieving. Neglecting these actions, they will have no credible defence if and when negative consequences of mine action come to light. In Cambodia, for example, allegations arose in April 1999 concerning falsified payroll records in the Cambodian Mine Action Centre (CMAC). These allegations led to further scandals culminating in claims that the agency was clearing land for military commanders, logging firms and a former leader in the Khmer Rouge. As it had not set clear performance targets, CMAC was not collecting and reporting data about who the ultimate beneficiaries were and had little evidence to demonstrate whether most of the cleared land went to poor peasants. The perception quickly spread that abuse was the norm.11 Mine action organisations operating in Cambodia have since begun to document and report on the end-use of land they have cleared,12 but tremendous damage was done to CMAC's reputation and to the level of donor support to Cambodia's mine action programme.
There are twin morals to this and similar tales. If programmes do not clarify for what results they are aiming in terms of what benefits will reach which beneficiaries for what purposes (and then monitor, document and report on their achievements of these beneficial results) then only the bad results can be reported and bad results are more likely because opportunists will take advantage of lax monitoring.
Mine action programmes in heavily contaminated countries will not be able to declare victory in the short- to medium-term. Therefore, they need to equip themselves adequately for the long haul, which implies learning how to learn. This ability is required if programmes are to assess their performance in terms of results that make a difference to people in mine-afflicted communities, which is necessary to maintain the support of donors and, increasingly, of host governments. Assessment of performance in terms of meaningful results is necessary to improve such performance over time. Accordingly, the principles below should be considered:
- Mine action programmes in heavily contaminated countries should do the
- Provide for a comprehensive needs assessment (such as a Landmine Impact Survey) to provide a coarse-grain picture of how contamination is impacting communities throughout their country.
- Provide for more targeted surveys of heavily impacted regions and communities to provide a progressively more detailed picture.
- Plans for all programmes and projects in heavily contaminated countries
should do the following:
- Specify objectives at the intermediate and final outcome levels and set performance targets for each.
- Specify relevant performance indicators for each result or, where it is unclear what indicators should be used or where data on an indicator are lacking, incorporate a clear plan for monitoring the results achieved during implementation and for reporting these results.
- Provide for periodic evaluations.
- All mine action programmes should initiate a system of post-clearance surveys to determine whether the land and facilities cleared are being used in the expected manner by the intended beneficiaries without undue delays.
- The international mine action community should establish an action
research network to promote learning about the social and economic dimensions
of landmine contamination and of mine action, with a particular focus on
narrowing the gap between social scientists and mine action practitioners, by
doing the following:
- Identifying common problems and emerging issues affecting major mine action programmes.
- Recommending issues that warrant study because of their practical potential for improving the results being achieved by mine action programmes.
- Promoting the rapid dissemination of key findings from studies, evaluations, pilot projects and other action research activities designed to generate new information about the interaction between people and landmines.
- Recommending new pilot projects to accelerate the practical application of key findings generated by research and evaluations.
The information in this article is extracted from the soon-to-be-released Geneva International Centre for Humanitarian Demining study Mine Action Lessons and Challenges.
- See for example, Andersson, et al., 2001. Another observer has aptly termed debates over the number of mines as "a persistent distraction." (Horwood,  p. 3)
- The standard criteria for evaluating development programmes are efficiency, effectiveness, sustainability, impact and relevance. (Principles for Evaluation of Development Assistance, OCDE/GD (91)208, OECD: Paris)
- This term also was coined in the late 1980s to describe the situation in Mozambique. Such emergencies feature intra-state conflict, a blurring of lines between combatants and civilians, violence directed largely against civilians, fluidity in terms of conflict zones and populations on the run, and a breakdown of the state and loss of legitimate authority over increasing swaths of the country. (DAC, 1999, Guidance for Evaluating Humanitarian Assistance in Complex Emergencies, OECD, Paris)
- For an example from Mozambique, "ADP was originally criticised for concentrating on mine-dense fields at the exclusion of examining socio-economic impact. Early tasks were in logistically favourable areas..." GICHD/UNDP, 2001, Socio-Economic Approaches to Mine Action: An Operational Handbook. p. 152. (available online at http://www.gichd.ch/pdf/publications/S_E_ApproachesMineAction.pdf; retrieved June 13, 2005.) Reports from Bosnia claimed that "...much of the effort is focused on farmland and pastures, not built-up areas, which take longer to rid of mines...." (Aida Cerkez-Robinson, Undermining Demining: Mine Removal Snarled in Conflicting Interests, Mismanagement, Associated Press, 28 May 2000.)
- Mine Action and Effective Coordination: The United Nations Policy, par. 5. UNMAS, New York, 1998. Available online at http://www.mineaction.org/pdf file/UNMAS Policy.doc (Retrieved June 13, 2005).
- Formally, Guidelines for Mine Action Programmes From a Development-Oriented Point of View: The Bad Honnef Guidelines, reproduced as Annex F of UNMAS, 1998. op. cit.
- From Osborne and Gaebler, 1992, Reinventing Government, Chapter 5, "Results-Oriented Government," cited in Quinn Patton, 1999, p. 27.
- Quoted in Maslen (forthcoming), Mine Action After Diana: An Audit of International Efforts Against Landmines and Unexploded Ordnance, Pluto Press: London.
- Such resistance has been reported in a number of countries such as Afghanistan, Cambodia, and Mozambique. Over-reliance on efficiency measures puts more flexible and responsive approaches at a disadvantage even though they may deliver higher—but hard to quantify—benefits.
- In many cases, the critics have also claimed the mobile teams are ineffective, but on close examination such claims represent confusion concerning the meaning of "effectiveness." For example, a recent critique of mine action in Bosnia (Banks, 2003, and Banks and Banks, 2003), which otherwise contains a good deal of valuable information, is marred by this confusion between efficiency and effectiveness.
- Information obtained from http://www.ngoforum.org.kh/, the NGO Forum on Cambodia website, accessed on 2 October 2003.
- See for example, HALO Trust (http://www.halotrust.org/cambodia.html).
85 Chemin de la Riviere
Tel: (819) 827-6663
Fax: (819) 827-5612
Study Group Socio-Economics
Avenue de la Paix, 7bis
P.O. Box 1300
CH-1211 Geneva 1
Tel: +41 22-906-1668
Fax: +41 22-906-1690