ITEP Evaluation of Metal Detectors and Dual-sensor Detectors

by Kazunori Takahashi [ Leibniz Institute for Applied Geophysics ] and Dieter Gülle [ Federal Office of Defense Technology and Procurement ]

Since its development in the early 1970s, scientists from an array of disciplines have found reason to utilize ground-penetrating radar to create radar images of the subsurface. The following article examines how GPR use in combination with standard metal detectors could aid workers in the field of demining.

An ITEP dual-sensor detector test, led by the Bundesamt für Wehrtechnik und Beschaffu (BWB), Germany’s Federal Office of Defense Technology and Procurement, took place September–October 2009 in Germany.1,2 Analysis of the test results clearly confirmed that the tested dual-sensor detectors reduce false alarms and that their metal-detector parts are not deteriorated, in comparison to the base model of a stand-alone metal detector used in combination with a stand-alone GPR device.

In this article, a dual-sensor detector refers to a combination of a metal detector and GPR. The combination allows the detection and identification of metal-containing objects; this combination is expected to contribute to the reduction of false alarms and, consequently, improve clearance-operation efficiency. This article provides an analysis and overview of the test results. The test’s detailed descriptions, as well as the results, can be found in the test report which will be available online soon.3

Test Conditions

A test site was constructed at a BWB facility in Oberjettenberg, Bavaria, Germany. Three types of soil were prepared: laterite, magnetite and humus. Laterite is a reddish clay loam with low stone (basalt) content. The soil has a very high magnetic susceptibility and is frequency dependent.4 Thus, it often causes metal detectors to give false alarms. Magnetite, the second soil type, is coarse sand mixed with engineered magnetite. The soil has a very high magnetic susceptibility but no frequency dependence. The third soil type is a loamy soil with a relatively high humus content—about 10%. Test-soil properties are described in detail in an accompanying report.5 Three types of mine-like targets, including rendered-safe mines, were planted in the soils: ERA calibration target, Gyata-64 and PPM-2. In addition, various sizes of metal pieces, such as bullets and cartridges, were buried as metal clutter. The burial depths ranged from two to 15 centimeters (0.78 to 5.90 inches).

Objects used in the test. From left to right: metal clutter (ammunition belts, cartridges, bullets) and mine-like targets (Gyata-64, PPM-2, ERA calibration target).
Objects used in the test. From left to right: metal clutter (ammunition belts, cartridges, bullets) and mine-like targets (Gyata-64, PPM-2, ERA calibration target).
Photo courtesy of BWB

An advanced landmine-imaging system developed by Tohoku University, Japan6,7 participated in the test. Cambodian deminers, who were trained by Tohoku University and attended previously conducted tests, operated the dual-sensor detector.8 For the comparison, various models of commercial metal detectors, including the base metal detector of ALIS (CEIA MIL-D1), as well as a commercial stand-alone GPR, were also tested. Operated by two scientists in the test, the stand-alone GPR system is not specially designed for demining but for general non-destructive testing purposes. Since the stand-alone GPR is not integrated with a metal detector, the system followed various models of stand-alone metal detectors and performed only discrimination. Therefore, the detection performance of the stand-alone GPR cannot be discussed, and only the discrimination performance is demonstrated.

The test was a blind test: The detector operators did not know the locations or the object types.9 Dual-sensor operators first used the metal-detector part of the device for detecting mine-suspected objects and switched over to the GPR for discriminating mines from metals. Two colors of markers were used to indicate the location and object type (mine or metal) found in the search with a dual-sensor detector. Operators of stand-alone metal detectors simply used one color of markers. After each test run, marker positions were measured with total stations.

Data Analysis

Data collected in the test was analyzed in the same way as analyzed in “Data Analysis and Performance Evaluation of Japanese Dual-Sensor Systems tested in Croatia” from The Journal of ERW and Mine Action, Issue 13.3.10 Detection capability is evaluated by calculating probability of detection and false-alarm rate, and discrimination performance is evaluated by false-alarm rate reduction and possibility-of-detection loss. The measures are defined as follows:

Possibility-of-detection rates ranging from 0% to 100% indicate how often targets are found, and a higher value indicates a better performance. False-alarm rate shows how many false positive indications (false alarms, alarms from other than target) are obtained in one square meter, and a lower value indicates better performance. False-alarm rate reduction indicates how many false alarms the GPR use decreases the number of false alarms found with the metal-detector alone. False-alarm rate reduction of 100% means that GPR use successfully discriminates and rejects all false alarms, and 0% means that no false alarms are rejected. This measure directly relates to efficiency improvements.

Possibility-of-detection loss indicates how many mines detected by the metal-detector part are falsely identified as metals and rejected by the GPR. A 0% possibility-of-detection loss means all mines are correctly recognized as mines, and a 100% possibility-of-detection loss means all mines are falsely rejected. This measure is directly related to the safety of deminers.

Possibility-of-detection and false-alarm rate can be calculated at two stages of the dual-sensor detector’s operation: after using only the metal-detector feature and after using both sensors. On the other hand, false-alarm rate reduction and possibility-of-detection loss can only be calculated after using both sensors. This means that both can be considered to be performance measures of the dual-sensor detectors’ GPR sensor.

Note that in this data analysis, unlike previously-conducted stand-alone metal-detector trials, metal pieces are considered a source of false alarms, not true positives. In this data analysis, only mine-like objects are considered the source of true positives (see Table 1 below from our earlier article).10 This is because dual-sensor detectors are supposed to discriminate mines from metals. In this article, this categorization is applied to stand-alone metal detectors as well so that their results can be directly compared to those of the dual-sensor detectors.

Table 1: Differences in categorization of sources of alarms for stand-alone metal detectors and dual sensors
(Click image to enlarge)
Table 1: Differences in categorization of sources of alarms for stand-alone metal detectors and dual sensors

Results

To demonstrate an overview of the detectors’ performance, results shown in this article are analyzed in each of the soil types. These results, as well as detailed interpretations, will be in the test report.3

Figure 1 shows possibility-of-detection versus false-alarm rate of ALIS and stand-alone metal detectors. The metal-detector part of ALIS (blue dot) achieved a result similar to its base metal detector (CEIA MIL-D1, light blue cross). This result indicates that the metal-detector performance integrated in ALIS is not deteriorated by the combined GPR, and it is still as good as the base metal detector. The metal detector part of ALIS declared approximately 2.5 false alarms per square meter (1.19 square yards), and using the GPR sensor reduces it to about 1.4 false alarms, denoting a 45% reduction. Consequently, the false-alarm rate obtained by ALIS is lower than any other stand-alone metal detector tested in the campaign. Since the metal detector is the primary sensor in ALIS, the detection performance depends entirely on the base metal detector. In the soils used in this test, the base metal detector achieved the lowest possibility-of-detection loss among all tested detectors. Therefore, the possibility-of-detection loss obtained by ALIS is also low, but this is due to the base metal detector’s performance.

Figure 1: Possibility-of-detection loss  versus false-alarm rate of ALIS and stand-alone metal detectors, in all soil types. The error bars show 95%  confidence bounds. For ALIS, the dot and circle indicate before and after  discrimination respectively.
(Click image to enlarge)
Figure 1: Possibility-of-detection loss versus false-alarm rate of ALIS and stand-alone metal detectors, in all soil types. The error bars show 95% confidence bounds. For ALIS, the dot and circle indicate before and after discrimination respectively.

False-alarm rate reduction and possibility-of-detection loss are plotted in Figure 2. The stand-alone GPR (red cross) achieved a remarkably high false-alarm rate reduction, indicating that approximately 90% of the false alarms are correctly identified. Furthermore, the false-alarm rate reduction by ALIS is much lower, meaning more metal pieces were misidentified and left as mines by ALIS as compared to the stand-alone GPR. On the other hand, the stand-alone GPR missed more mines than ALIS. It is difficult to grade the devices because the results can change with each operator. If an operator is afraid of missing mines and reports mines for all metal-containing objects the metal detector signals, no mine will be missed, but also no false alarms will be rejected, meaning possibility-of-detection loss and false-alarm rate reduction is very low. This is due to the fact that the device only provides information on the objects, and this information must be interpreted by the operator. Thus, the decision is entirely up to the operator. Nevertheless, the figure clearly shows that GPR itself is potentially capable of discriminating landmines from metal pieces. However, from the operational point of view, possibility-of-detection loss must be kept as low as possible.

Figure 2: False-alarm rate reduction versus possibility-of-detection loss found in the discrimination process in all tested soil types. The error bar shows 95% confidence bounds.
(Click image to enlarge)
Figure 2: False-alarm rate reduction versus possibility-of-detection loss found in the discrimination process in all tested soil types. The error bar shows 95% confidence bounds.

Figure 3 shows false-alarm rate reduction and possibility-of-detection loss as a function of depth. As a tendency, ALIS and the stand-alone GPR achieved lower false-alarm rate reductions and higher possibility-of-detection losses at shallow depths, which confirms the results in a former test.9 The depth dependency looks weaker for ALIS, especially at the shallowest depth range of 0–3 centimeters (0–1.18 inches) in both false-alarm rate reduction and possibility-of-detection loss. This variance might be due to the difference in signal processing employed in the systems and the GPR data’s representation to the operators. The stand-alone GPR displays almost raw data12 as a vertical slice of the subsurface, whereas ALIS constructs horizontal slices by applying a number of signal processing. As a result, ALIS may be able to obtain more robust information on targets than the stand-alone GPR through the sophisticated processing.

  Figure 3a
(Click image to enlarge)
Figure 3a
Figures 3a and 3b: False-alarm rate  reduction and possibility-of-detection loss as a function of depth in all soil  types together. The dashed lines show 95%  confidence bounds.
(Click image to enlarge)
Figures 3a and 3b: False-alarm rate reduction and possibility-of-detection loss as a function of depth in all soil types together. The dashed lines show 95% confidence bounds.

Figure 4 shows the averaged search speeds of ALIS and stand-alone metal detectors operated by newly trained vs. experienced personnel. ALIS required nearly double the metal detector’s time. In other words, ALIS was twice as slow as the stand-alone metal detectors. In this test, only detection and discrimination were performed. Excavation and confirmation of detected objects, which corresponds to the steps 4 and 5 in the Boshoff and Cresci article, ”The HALO Trust and HSTAMIDS,” were not included.13 Therefore, assessing the efficiency improvements of the entire clearance operation with a dual-sensor detector in detail is impossible based on the obtained results. However, a rough estimate can be made as follows: Let T0, the total time necessary for the entire clearance operation with a metal detector, be equal to the search time plus the time for excavation (and other processes). The search time can be expressed as the time for searching one metal-containing object (ts) multiplied by the number of objects found, x. In a similar manner, the time for excavation can be expressed as the time for excavating one object (te) multiplied by the number of objects, x.

Assuming ALIS needs twice the search time of a stand-alone metal detector for detection and identification of one object, but reduces false alarms by half, the total work time using ALIS (T1) can be expressed as:

If T0 >T1, we obtain te >2ts, which means that the clearance operation is expected to be accelerated if the excavation process for one object requires more than twice the time necessary for finding one object, under the assumption that rejected false alarms will not be excavated. For the sake of humanitarian demining, rejected false alarms may also need checking, but it can be done quickly if the detected objects are identified as non-explosive items through a device such as the Handheld Standoff Mine Detection System.13 Even taking intoaccount the rapid excavation process, the situation may be realistic, especially in heavily metal-contaminated areas.

A study shows that the most common activity at the time of an incident is excavation.14 Using a dual-sensor detector to reject metals cannot reduce the potential risk of the excavation process because detected landmines must be taken out anyway. However, the amount of this stressful work can be reduced, and it may help deminers concentrate on their tasks.

Figure 4: Average search speed of ALIS and stand-alone metal detectors in minutes per square meter. The labels “MD” and “MD mfr” indicate metal detectors operated by trained operators and the manufacturers, respectively.
(Click image to enlarge)
Figure 4: Average search speed of ALIS and stand-alone metal detectors in minutes per square meter. The labels “MD” and “MD mfr” indicate metal detectors operated by trained operators and the manufacturers, respectively.

Discussion and Conclusions

The test results confirm that dual-sensor detectors can reduce false alarms as compared to stand-alone metal detectors, which indicates potential efficiency improvements in clearance operations. However, a few issues in need of consideration came up during the test and data analysis, such as possibility-of-detection loss, search speed and training. From observation, dual-sensor detectors can correctly reject false alarms, but they also sometimes falsely reject mines. This seems to happen especially at shallow depths (see Figure 3b above), but it also appears related to the soil type.15

Investigating soil properties and matching the appropriate soil type to the proper equipment can minimize the false rejection of mines (possibility-of-detection loss) for dual-sensor detectors. The search speed is directly related to the efficiency improvements, and the higher the search speed, the more improvements can be achieved. The test results indicate that dual-sensor detectors are twice as slow as stand-alone metal detectors. Even so, the clearance operation can be accelerated if a certain number of false alarms are reduced. Furthermore, an additional attempt in this test indicated that operators of dual-sensor detectors who have more experience and knowledge working with the device can work as fast as operators using stand-alone metal detectors. However, this fact also indicates that more training and/or practice is necessary for dual-sensor detector use when compared to standard metal detectors. The advantages of experienced personnel who have trained for a short period of time appear significant in search speed and performance.

The dual-sensor test allowed us to evaluate detection and discrimination performance in a blind test. Although a very rough estimate of the efficiency improvements has been made, other factors need consideration for the detailed assessment such as excavation time, detector costs, and training and practice costs. Only a long-term field trial can evaluate these factors.

In the test campaign, stand-alone metal detectors that possess the capability of discriminating objects were also tested. The evaluation is not discussed in this article, but readers interested in these devices can find the results in the test report.3 j

Endnotes

  1. “ITEP Work Plan 2000-2009.” International Test and Evaluation Program for Humanitarian Demining.(April 2010). http://www.itep.ws/activities/itep_workplan/ITEPWorkPlan2000_2009draft.pdf. Accessed 12 October 2010.
  2. Borry, F. “An Update on the ITEP Program and Activities. The Journal of ERW and Mine Action, 12.2 (Winter 2008/2009: 98-100). http://www.jmu.edu/cisr/journal/12.2/rd/borry/borry.shtml. Accessed 12 October 2010.
  3. Takahashi, Kazunori; Gülle, Dieter. ITEP Dual-Sensor Test September/October 2009. Will be available soon at http://www.itep.ws or http://gichd.org. A summary is currently available at http://www.itep.ws/pdf/DStrial2009FinalSummary.pdf. Accessed 12 October 2010.
  4. This radar types performance capability is strongly dependent on the site’s soil electrical conductivity. If the soil conductivity is high, the radar signal’s attenuation in the soil can severely restrict the radar signal’s maximum penetration depth. For cart-mounted radars, 150 MHz is a typical center frequency; however, 300 and 500 MHz are sometimes used for shallow, high-resolution probing, and frequencies as low as 20 MHz are used for locating deep caves or mine tunnels. For more information: http://ldolphin.org/GPRLimits.html. Accessed 12 October 2010.
  5. Preetz, H., Takahashi, K., and Igel, J. “Physical Characterisation of the Test Lanes in the ITEP Dual Sensor Test. Oberjettenberg/Germany 2009. Leibniz Institute for Applied Geophysics. (February 2010).  http://www.itep.ws/pdf/OberjettenbergDStest_soilsLIAG2009.pdf. Accessed 12 October 2010.
  6. Takahashi, K., and Sato M. “ALIS A Hand Held Dual-Sensor with Imaging Capability. Mine Action Technology Newsletter. No. 9. (January 2009). http://www.gichd.org/fileadmin/pdf/technology/technology_newsletter/MA-Technology-Newsletter-No9-February2009.pdf. Accessed 12 October 2010.
  7. “ALIS: Innovative Hand Held Dual Sensor for Humanitarian Demining. (2009). http://www.alis.jp/. Accessed 12 October 2010.
  8. Sato, M. “Evaluation Test of ALIS in Cambodia for Humanitarian Demining. Proceedings of the SPIE Conference on Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XV, 7664, April 2010.
  9. “CEN Workshop Agreement, Humanitarian Mine Action Test and Evaluation Part 1: Metal Detectors. CWA 14747-1, June 2003.
  10. Takahashi, K., Gaal, M., and Gülle, D. “Data Analysis and Performance Evaluation of Japanese Dual-Sensor Systems tested in Croatia. The Journal of ERW and Mine Action, Issue 13.3 (Fall 2009: 66-70). http://www.jmu.edu/cisr/journal/13.3/randd/takahashi_etal/takahashi_etal.shtml. Accessed 12 October 2010.
  11. “Landmine Detection and Clearance. The Landmine Action Smart Book (December 2004). Mine Action Information Center. http://www.jmu.edu/cisr/research/smartbook.pdf. Accessed 12 October 2010.
  12. “Raw data" means "unprocessed data. It is very common to apply some signal processing to GPR data, but the stand-alone GPR did not apply sophisticated processing. This simple processing cannot drastically improve the data quality.
  13. Boshoff, C., and Cresci, R. “The HALO Trust and HSTAMIDS. The Journal of Mine Action, Issue 12.1 (Summer 2008: 86-89). http://www.jmu.edu/cisr/journal/12.1/rd/cresci/cresci.shtml. Accessed 12 October 2010.
  14. Smith, A. “Injuries that Occur In Humanitarian Demining. Landmine Monitor Report 2001. New York: International Campaign to Ban Landmine. http://www.the-monitor.org/index.php/publications/
    display?url=lm/2000/appendices/injuries.html
    . Accessed 9 September 2010.
  15. Takahashi, K., Preetz, H., and Igel, J. “Soil Characterisation and Performane of Demining Sensors.” 7th International Symposium Humanitarian Demining 2010, Sibenik, Croatia, April 2010.

Biography

Kazunori TakahashiKazunori Takahashi is a Research Scientist at the Leibniz Institute for Applied Geophysics in Hannover, Germany. His research activities include development and evaluation of dual-sensor systems for humanitarian demining, investigations of soil influences on demining sensors and GPR-system development, as well as signal processing for subsurface sensing.


Dieter GulleDieter Gülle is a retired colonel who has worked since 1995 in different positions connected with minefield survey, clearance operations and clearance-equipment testing. He represented Germany in the International Test and Evaluation Program for Humanitarian Demining as head of the ITEP Secretariat. He is currently a consultant to the Federal Office of Defense Technology and Procurement in Germany.

Contact Information

Kazunori Takahashi
Research Scientist
Leibniz Institute for Applied Geophysics
Stilleweg 2
30655 Hannover / Germany
Tel: +49 511 643 3572
Fax: +49 511 643 3665
E-mail: kazunori.takahashi(at)liag-hannover.de
Web site: http://www.liag-hannover.de

Dieter Gülle
Consultant
Federal Office of Defense Technology and Procurement
Stubenrauchstr 65d
Berlin / Germany
Tel: +49 306 366 418