Non-technical Survey: A Model for Evidence-based Assessment

by Aron Larsson and Love Ekenberg [ Stockholm University ] and
Åsa Wessel and Håvard Bach [ GICHD ]

In an ongoing effort to improve the Non-technical Survey, the Geneva International Centre for Humanitarian Demining teamed with Stockholm University to create an enhanced version of the Cambodia Mine Action Centre’s Evidence Assessment Model. The aim of the project was to make the existing model more user-friendly and modify the current standards for assessment of mine-affected land. CMAC is testing a revised model to ensure that it meets the needs of their Non-technical Survey teams.

The Geneva International Centre for Humanitarian Demining has developed an Evidence Assessment Model that may form part of a wider Non-technical Survey and enable decisions about when it is appropriate to release land by Non-technical Survey and when/how much Technical Survey is required. The first model was created in collaboration with Norwegian People’s Aid to enhance NPA’s land-release approach in Angola. A second, similar model was developed in support of the Cambodian Mine Action Centre’s land-release approach in Cambodia.

Although these models are in use and working fairly well, the GICHD wanted to test the quality of the model to ensure the validity of its logic procedures and develop an improved interface. The primary objective was to devise a credible, practical and user-friendly model for Non-technical Survey by August 2009. The project was a joint effort between the Department of Computer and Systems Sciences of Stockholm University and the GICHD, and it was partially funded by the Swedish Program for Information and Communication Technology in Developing Regions. GICHD asked the decision-analysis experts of the DECIDE Research Group at Stockholm University to assist with the project.

The project was initiated in March 2009, and the first phase was completed in September 2009,with the delivery of a revised CMAC Evidence Assessment Model to be used in pilot cases in Cambodia.


(Click image to enlarge)
Figure 1: Excerpt from the user sheet. A 'Y' in the cell is an added statement.

The Context and Work Process

The model is designed as a complementary tool in the existing process in which a team of field operators collects and analyzes information about an area suspected to be contaminated by landmines. Traditionally, the decision on whether an area can be released from suspicion of mines without any further mine-action support has been made by the field operator, based on personal experience and conviction. This method has often caused conservative decisions because it has been far easier and less risky for the survey teams to classify land as mine-suspected areas as opposed to mine free areas. A credible evidence-assessment model that shifts liability from the operator to the model, or the underlying concept, will encourage more appropriate decisions.

The model described in this article rates the importance, or value, of each individual piece of evidence about the mine threat provided by various informants. The model further contemplates the degree of trust in, or credibility of each source of information. If the credibility of an informant is low, the evidence weight will be reduced and will consequently contribute less to the final survey conclusion.

When organizations use the model, the burden of making the final decision rests less on the experience of the individual field operator and more on the embodied model assessment and recommendation. Using the model further ensures that every step of the survey process is thoroughly analyzed, evaluated and documented. A clear order trail is crucial and will further enable appropriate quality assurance and other follow-up if required.

Prerequisites and model requirements. An evidence-assessment tool needs to fulfill a number of requirements:

The challenge is thus to select a suitable approach for the assessment. T. Denoeux discusses one approach that employs methods from the area of pattern classification in Analysis of Evidence-Theoretic Decision Rules for Pattern Classification.1 It does not currently seem to be applicable in the context of operational mine action, although it would be theoretically appealing to explore it. We argue that a more applicable method should enable the elicitation of knowledge and experience from the local population, the military, land users and mine-action experts, and should allow user-friendly adaptations. A user-friendly model with a user sheet that accommodates individual evidence from informants has thus been developed in Microsoft Excel, which is easily available and will only require basic computer skills.

The proposed model. The proposed Evidence Assessment Model is based on traditional input- output assessment and classification, employing a formalized recommendation rule that proposes the next minimum mine-action requirement from a set of predefined recommendations. The model thus uses a well- established and easy-to-understand methodology. GICHD, Stockholm University and CMAC agreed to employ methods from multi-attribute decision theory, since the current approach for Non-technical Survey conformed to the use of numerical weights representing the relative importance of different sources of information. Typical for multi-attribute decision-making is the use of several attribute-specific values being aggregated using additive attribute weights.2

In the CMAC model, the aim has been to associate each sector Si with a sector value v(Si) used for the classification of sectors.3 We propose a method for the classification of sectors that are defined in an attribute-value space, where the sector value should be the result of an aggregation of values on evidence attributes. This method would enable simple and useful means for sensitivity analysis and model adaptations while maintaining the look and feel to which CMAC has been accustomed, with some improvements where needed. The simplicity of the model will thus be maintained, but the transparency and usefulness would improve. The model has also been designed to accommodate more advanced future methods for coping with uncertain information and vague assessments and risks.

Input statements. Input statements are entered into the model by field operators or by personnel who receive the reports from the field. Each statement is the result of obtained external information and information from field interviews with the population, police, military or other sources. A statement will either support the conclusion that an area is mined (hazardous) or mine-free (not hazardous), i.e., a statement will either belong to the statement set pro-mines M+ (supporting the presence of mines) or the statement set con-mines M- (supporting the absence of mines). Each statement corresponds to a certain sector of interest, an evidence attribute and a confidence assessment.

An evidence attribute is associated with a numerical weight, reflecting the strength (or importance) of this evidence, such as to what extent evidence supports the presence of mines in the area or not. The value of the different weights depends on whether the statement belongs to M+ or M-. Mine maps showing the presence of mines could, for example, be regarded more important (evidence of presence of mines) than mine maps showing no mines in certain areas (evidence of absence of mines).

An evidence attribute can be adjusted by the field operator based on the perceived credibility of the information source. One statement does, for example, address mine maps/records from military or police. If the investigator has access to maps that indicate an area is mined and the maps are considered accurate and reliable, this evidence attribute is either marked as high credibility or low credibilitythe latter if the map is inaccurate but yet exists. In summary, each statement (sj):

Sector value and model output. The aggregation of information that provides a sector value is straightforward. Each pair (ii, sj) is assigned a weight wij E [0, 10]. The weights are then subject to normalization so that

equation 1

or the sum of all wij will add up to 1. The aggregated value (V(S)) of a sector (S) is obtained from the difference between the weighted sum of confidence values belonging to M+ and M- respectively. It follows that V(S) is within the interval [-10, 10].

equation b

Figure 2: Threshholds used in current model.
(Click image to enlarge)
Figure 2: Threshholds used in current model.

The output is ultimately a recommendation for the next step required in the land-release process, typically a level of Technical Survey or clearance. In the current model, the recommendation rule is based on a set of thresholds for different intervals, each representing conclusions from the Non-technical Survey Evidence Assessment. The current interval thresholds are shown in Figure 2.

The recommendation may also be dependent on other criteria besides the sector value alone. For instance, a recommendation stemming from a high-confidence conclusion may be dependent on whether the field operator has entered a sufficient number of high-confidence statements in the model. In other words, a recommendation cannot be based upon a high-confidence conclusion if there are too few high-confidence statements entered into the model by the field operator.

Figure 3: Example of a CHA with proposed classified sectors.
(Click image to enlarge)

Figure 3: Example of a CHA with proposed classified sectors.

Conclusion

The model is currently being subjected to pilot testing by CMAC. If the pilot proves successful, the underlying idea of employing formal methods for making more appropriate Non-technical and Technical Survey decisions will be further refined. CMAC is one of several organizations that currently conducts a Base Line Survey with the aim of resurveying all areas suspected to be mined in Cambodia for better allocation of demining assets. The model is now used by all 13 CMAC Non-technical Survey teams as an integrated part of the BLS. Pending the results from Cambodia, the concept may well be introduced in other mine-affected countries.

There are, however, some challenges that need to be addressed, including the following issues and questions:


Preliminary results from the pilots have highlighted a need to ensure that:

Biographies

Aron LarssonAron Larsson is a researcher at Stockholm University and Mid Sweden University. He has a Ph.D. in computer and systems sciences and a Master of Business Administration. Larsson has been working on the development and application of methods and tools for evaluation of decisions under multiple objectives, uncertainties and risk. He has been developing the DecideIT decision tool and has experience applying decision theories in both public and private sectors such as environmental planning and investment decision-making.


Love EkenbergLove Ekenberg is a Professor at Stockholm University, the Swedish Royal Institute of Technology and Mid Sweden University. He has a Ph.D. in computer and systems sciences and a Ph.D. in mathematics. Ekenberg has been working with risk and decision analysis for more than 15 years. He has also worked with logic verification of complex industrial systems for several years at Swedish nuclear power plants.


Asa WesselÅsa Wessel is the Land Release Research Officer at the GICHD. She previously worked as a Quality Assurance Officer at the Mine Action Coordination Centre in South Lebanon with the
United Nations Office for Project Services. Wessel worked as Explosives Ordnance Disposal Coordinator at the Regional Mine Action Coordination Centre in Sudan, employed by the Swedish Rescue Services Agency, and as Instructor at the Swedish Armed Forces EOD and De-mining Centre. She holds a Bachelor of Science in mechanical engineering


Havard BachHåvard Bach heads the Operational Support Unit for the GICHD. He manages projects that currently include studies on mine-detection dogs, mechanical mine clearance, manual mine clearance and risk management. Bach also worked as a Norwegian Military Engineering Officer before being employed by Norwegian Peoples Aid and managing several mine-action programs worldwide.


Endnotes

  1. Denoeux, T. 1997. Analysis of Evidence-Theoretic Decision Rules for Pattern Classification. Pattern Recognition 30(7), pp. 10951107.
  2. For a comprehensive treatment of this theory, see Chankong, Vera and Haimes, Yacov Y. Multiobjective Decision Making: Theory and Methodology 1983. North-Holland, Amsterdam or Keeney, R.L, and Raiffa, L. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley. New York. http://www.amazon.com/Decisions-Multiple-Objectives-Preferences-Trade-Offs/dp/0521438837. Accessed 10 March 2010.
  3. Minefields are partitioned into disjoint sub-areas known as sectors.
  4. This may be interpreted as the default value if there is no particular reason to discredit the credibility of an informant a piece of information.

Contact Information

Aron Larsson
Dept. of Information Technology and Media
Mid Sweden University
SE-851 70 Sundsvall / Sweden
Tel: +46 60 148616
E-mail: aron.larsson@miun.se
Web site: http://www.miun.se/personal/aron.larsson

Love Ekenberg
Dept. of Computer and Systems Sciences
Stockholm University
Forum 100
SE-164 40 Kista / Sweden
Tel: +46 8 161679
E-mail: lovek@dsv.su.se
Web site: http://www.dsv.su.se/~lovek/

Åsa Wessel
Land Release Research Officer
Geneva International Center for
Humanitarian Demining
7 bis, avenue de la Paix
P.O. Box 1300
CH-1211 Geneva 1 / Switzerland
Tel: +41 7982 88752
E-mail: a.wessel@gichd.org
Web site: http://www.gichd.org