Issue 7.3, December 2003
Information within this issue may be outdated. Click here to view the most recent issue
At a four-day conference in Brussels, scientists and demining practitioners exchanged findings from their studies of old and new demining technologies. The following article discusses some of the key developments and implications from the conference.
by Ian G. McLean, GICHD
An old and persistent theme was revisited at the 2nd European Union in Humanitarian Demining and Society of Counter Ordnance (EUDEM2-SCOT) conference held in Brussels (September 15–18, 2003). Researchers on new demining technologies have promised much, yet have delivered little. No silver bullet has sprung from the millions of dollars invested on technology research. Deminers today are still using the same techniques that have been applied for almost a century.
Have things improved? Yes! Standards have been developed, metal detectors are more discriminatory, personal protective equipment (PPE) is infinitely better, machines are more reliable, the demining workplace is safer, the procedures are more efficient and the equipment is more effective. But these are new versions of old technologies. What was the point of all that other investment? Surely the job would already be finished if the money had been spent on getting mines out of the ground instead of being wasted on grandiose but unworkable ideas. Scientists may be dedicated and imaginative, but they seem incapable of transforming their dreams into practical tools.
Research and Development Implications
Let us dispense with one sensible argument. Research and development (R&D) is an early step in a tool-making process that becomes increasingly expensive at each step. Transforming a prototype into a practical tool requires significant investment, considerable time and a willingness to take a chance. Several speakers at EUDEM noted that the small size of the humanitarian demining industry simply does not justify the investment required for commercialization of a new tool. There are two practical choices—convince a large military organisation of the viability of the tool or seek investment from sponsors who are not motivated by profit. A third possibility is that the tool will have applications outside the demining industry and investment can be sought there, although usually it will have been co-opted by the demining industry after development for other purposes. The reality is that some working prototypes are not yet available as demining tools because nobody was willing to make the post-R&D investment that would turn the tool into a commercial product. The researchers did their job and it is not their responsibility to support commercialization. Of course, one is still left wondering why the original R&D was supported, but that is a side issue to the main theme here.
|There appears to be a misunderstanding of what science can actually supply and what practitioners believe it should supply. Many disagreements between practitioners and scientists can be reduced to the simple statement: “That’s not the experiment you should have done.”|
There is at least one other reason why technology researchers appear not to have delivered on their promises, and it has to do with the nature of the scientific process. A simplistic view is that there are two kinds of demining technology out there:
There is an important difference between these technologies. The former technologies may have been well tried, but were never properly tested. The latter technologies are being properly tested, but unfortunately, as a result, are never tried.
Comparison of Old and New Demining Technologies
Most research on new technology is necessarily prospective. The technology exists as a concept but will not be used until its effectiveness has been demonstrated. The researchers may get close, but they never achieve complete satisfaction with the output. The question is not about whether the technology works, because it works alright. The problem is that the experiments do not yet show that the technologies are working reliably, effectively or consistently. Sometimes more investment is required to take that last fine-tuning step. But sponsors lose interest, or the complaint that “all researchers ever do is create more research” may start to be heard, as it was heard at the EUDEM conference.
Research on old technology, if it is done at all, is necessarily retrospective. Most practitioners wonder why anybody would bother because surely the technology is “tried and true.” For example, the Geneva International Center for Humanitarian Demining (GICHD) has already been advised by several potential participants that its proposed study on manual demining is a “waste of time and resources.” As a result of long experience, demining practitioners believe in the old technologies and what they can deliver, and procedures are already in place to deal with known inadequacies. The old technologies are effective and reliable, which is why they are preferred.
Several papers at the EUDEM conference reported new tests of old technologies. To the surprise of some, those reports did not reinforce belief in the quality and effectiveness of much-loved tools. Mueller et al.1 reported success rates with metal detectors beginning at about 50 percent and rarely reaching above 80 percent. Fjellanger et al.2 reported detection success in a pilot Remote Explosive Scent Tracing (REST) study of 68 percent. Trevelyan3 reported success rates with prodders down to 50 percent, and he concluded rather provocatively that prodders should not be used at all.
These results for the metal detectors and prodders were not well-received. First, the members of the audience insisted that the results were not consistent with their experience. Second, they asked what else they were supposed to use if the old techniques were not available. The comments underline a well-known phenomenon in psychology built on a group of effects such as the “mere-exposure effect” (mere exposure to something new produces increased belief in its value), and encapsulated in the notion of “seeing is believing.” Practitioners believe in prodders and metal detectors, so they discount a demonstration that these tools are less effective. However, they have no experience with REST, ground-penetrating radar or thermal imaging, and willingly accept a demonstration that these technologies are less than 100-percent effective.
What actually is going on here? Is this retrospective research on old technology just plain silly, or is it trying to tell us something? Might it be sending us a message about the prospective research on new technologies? What sort of information does science really provide? The last question can at least be answered sensibly and the answer gives insights into the other questions. In its simplest form, the scientific process involves setting up a comparison between two or more versions of a situation (usually called treatments). Something will be measured in a standard way for each treatment and most other things will be held constant. The comparison itself is achieved using objective and highly structured procedures (statistics, graphs). It is the difference between the treatments that scientists are interested in. Less interesting or even irrelevant is some form of absolute measurement because the primary result is to be found in the comparison.
The art of practicing science involves designing experiments that allow treatments to be separated using data. Scientists are always concerned about ceiling effects (e.g., it is impossible to obtain a result beyond 100 percent, so 100 percent is a ceiling). If the results for both treatments approach 100 percent, there is a good chance that they will not be statistically distinguishable even if they really are different. Thus, a good experimental design will include treatments that give results well below ceilings in order to spread the results. Put another way, scientists would have done the wrong experiment if the results for both treatments were close to 100 percent. Of course, for a demining tool, detection rates much less than 100 percent immediately cause people to become worried.
A second important issue can be termed appropriateness. Investigations of issues related to mines can be thought of as being conducted at the following three levels:
When retrospective experiments using standard demining tools are designed, clearly there is likely to be some tension between the two requirements of ensuring a spread of data to allow effective statistical comparison and realism. Disagreement about the balance between these requirements is central to some misunderstandings between scientists and operations personnel. In theory, the old tools are extremely effective and should be pushing hard against the 100-percent ceiling under realistic treatment conditions. However, operational people are accustomed to using their standard tools in a wide variety of conditions. Therefore, it is disconcerting to see results showing that those tools are not working well under even some conditions, and it is hardly surprising that the results are greeted with disbelief.
Demining Technology in the Future
What if the results are real (i.e., they are a realistic representation of the effectiveness of standard demining tools)? After all, those tools were never thoroughly tested before implementation, and even today, belief in their effectiveness is really just that—belief. When researchers are attempting to develop a new demining tool, what kind of effectiveness should they be aiming for? Presumably, they should be getting pretty close to the desired 100 percent, because anything less than 100 percent rapidly becomes unacceptable in a demining tool. But the results presented at EUDEM suggested that well-known and accepted demining tools often give less than 80-percent effectiveness in standardized tests or in experimental situations in which different treatments are being compared. Trevelyan concluded at EUDEM that prodders should not be used. But perhaps a better conclusion is that prodders are no better than new technologies that are not yet in use because they are achieving significantly less than 100-percent detection success. Machines are not yet accepted as a clearance tool, but standardized tests often show them doing better than 90 percent.5 Results of tests of new technologies are not generally published, but if they are achieving 70-percent or more, they may be performing as well on standardized tests as traditional tools. Some may be performing better.
Despite the mantra that “there are lies, damned lies and statistics,” scientific tests do not lie. Certainly, the data can be manipulated, and worse, are regularly misquoted out of context by politicians and others. But if the methodology is clearly described and the statistical analysis is appropriate, then the results tell their own story. Scientists design experiments that ask very specific questions. Some extrapolation from their results and conclusions is appropriate, but should not be taken too far. If a scientist says “under treatment X, I obtained 80 percent effectiveness, and under treatment Y, I obtained a significantly lower effectiveness of 60 percent,” it does not follow that the tool was operating at 70-percent effectiveness. What follows is that there is something to be learned from the difference between the two treatments. Readers should also treat very cautiously the implication that 80 percent and 60 percent are absolute measurements of effectiveness under operational conditions similar to the test conditions.
A recurrent theme at the EUDEM conference was that it is time to move away from an emphasis on getting every mine out of the ground, and start addressing demining problems using risk assessment procedures. No demining tool gives 100-percent effectiveness all of the time, so we should not be too surprised when scientists get the sorts of results reported by Mueller et al., Fjellanger et al. and Trevelyan. It is refreshing to see such studies being reported because they should have been done years ago. They make an important contribution by allowing the demining industry to refine its risk analyses, and may also cast more sensible light on the effectiveness required of new technologies before implementation is considered.
Thanks to the organizers and sponsors of the EUDEM2-SCOT conference, who provided a valuable forum for communication between scientists and demining practitioners. This commentary has benefited from the comments of G. Rhodes, R. Sargisson, H. Bach and A. Griffiths.