Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in Croatia

By by Kazunori Takahashi [Leibniz Institute for Applied Geophysics], Mate Gaal [Federal Institute for Materials Research and Testing] and Dieter Gülle [Federal Office of Defense Technology and Procurement]

Two years ago, the Croatian Mine Action Center–Center for Testing Development and Training Ltd. tested two Japanese dual-sensor systems for humanitarian demining in Croatia. The test’s results show that these detection systems can potentially increase the accuracy of mine-detecting operations, but several improvements to the sensors may be required before the systems are fully effective.

Gryphon dual-sensor system
The Gryphon dual-sensor system evaluated in the test. The Gryphon team consists of two buggies: one with a metal detector (near side) and one with GPR (far side).
All photos and graphics courtesy of the authors

In October 2007, the Croatian Mine Action Center–Center for Testing Development and Training Ltd. (HCR-CTRO), with assistance from the German Federal Institute for Materials Research and Testing (BAM), tested two sensor systems: the Advanced Landmine Imaging System, developed by Tohoku University, Japan, and the Gryphon, developed by Tokyo Institute for Technology, Japan. Both systems employ commercial metal detectors (the ALIS with CEIA MIL-D1 and the Gryphon with Minelab F3) and ground-penetrating radar. The metal detector only indicates the presence of metal; it cannot determine if the metal is a mine. The GPR indicates objects with a shape that could resemble a mine. The operator of the system decides whether to reject the metal clutter. Together, the systems improve the productivity of demining operations.1,2,3 This article discusses the tests results, the systemsperformances and the data analysis.

Test Conditions

The complete report, detailing the conditions, procedures and results of the two dual-sensor systems is available online.4 The test was carried out at the Benkovac test site in Croatia where previous metal-detector trials have taken place, such as the Systematic Test and Evaluation of Metal Detector (STEMD) trial.5 Three soils are available at this site: red bauxite (Lane 1), neutral clay (Lane 3), and red bauxite with neutral stones (Lane 5, local soil), as shown in the figures

Lane 1: red bauxite. Lane 3: neutral clay. Lane 5: red bauxite with netural stones.

Blind tests were conducted in these lanes with real, rendered-safe mines (11 were PMA-2 and nine were PMA-3 mines) and metal clutter. The target layout was the same as that in the International Test and Evaluation Program for Humanitarian Demining STEMD trial5 with additional small pieces of various metals (nine per lane) placed on the ground surface. Thus, each lane comprised a total of 38 buried targets.


(Click image to enlarge)
Figure 1: Operation procedure of the dual-sensor systems.

Three deminers from the Croatian Mine Action Center served as operators of the ALIS; the developer had trained them together 10 workdays prior to the test. In the test, each deminer went through each of the three lanes once. The developers team of five to six persons operated the Gryphon.

Both dual-sensor systems employ a metal detector as a primary sensor and a GPR as a secondary sensor; the metal detector first detects all the metal objects, and then the GPR identifies objects suspected to be landmines. In the test, red markers indicated positions of objects detected by the metal detector and yellow markers indicated positions of objects confirmed as landmines by the GPR, so that those detections could be classified later. The operation procedure is schematically illustrated in Figure 1.

where anddenote the number of mines buried and the number of mines detected by the metal detector, respectively. The other POD for GPR is defined as:

where is number of mines correctly confirmed after the use of the metal detector and the GPR. Metal-detector alarms not caused by mines and GPR alarms incorrectly confirmed as mines are considered false calls for a dual sensor. Therefore, the false-alarm rates for the metal detector and for the GPR, respectively, are defined as:


(Click image to enlarge)
Table 1: Differences in categorization of sources of alarms for stand-alone metal detectors and dual sensors.

where , , and are alarm numbers caused by metal and soil, reported by the metal detector and the GPR, and A denotes an area searched. Note that the definition of the false alarm for metal detectors in this analysis is different from that of stand-alone metal detectors. Alarms from metal pieces are normally counted as true positives for stand-alone metal detectors, while they are considered false positives for dual sensors because of the detectors objective, which is to differentiate between landmines and other objects. The different categorizations of alarms are summarized in
Table 1.

In order to observe how much efficiency is improved, FAR reduction, , is introduced as follows:

If all the false alarms are rejected, takes a value 1.

The GPR could fail to detect mines. It can be acceptable to miss false alarms; however, miss-discrimination for mines threatens the lives of end-users. In order to see the frequency of missed mines, probability of detection reduction, , is defined as:

If the GPR does not reject any mines found by the metal detector, the value becomes 0. Related to the reduction, the discrimination ratio for mines is introduced to find how often mines are correctly confirmed.

The ratio is actually given by one minus POD reduction:



Figure 2: Receiver-operating-characteristic diagram with 95% confidence limits for each device.

Confidence limits of 95% of POD, FAR and their reductions are provided in Figures 2 and 3 to illustrate the accuracy of the estimations. They are calculated assuming the binomial and the Poisson distributions for POD and FAR, respectively.9,10,11

In the following section, the defined quantities above are incorporated into various figures that display the results of the dual-sensor system performance evaluations.

Results

Although performance of metal detectors and GPRs can be quite different in various types of soil, the results in the three lanes are analyzed together in this article to show the overview. An analysis of each soil can be found in the trial report.4 The ALIS operator whose results differed greatly from the others had his results excluded as an outlier.

Figure 2 shows the receiver-operating-characteristic diagram in which probability of detections are plotted against false-alarm rates and the 95% confidence limits. Each device has two plots, one from using only the metal detector (primary sensor) and one from using both sensors (metal detector and GPR). It can be observed that the FARs by metal detectors (squares) are shifted toward the left by using GPRs (circles), meaning that FARs are reduced significantly. However, at the same time, reductions of PODs also occur for both devices, which should not happen for safety reasons.


Figure 3: FAR reduction versus POD reduction for each device.

The absolute levels of POD and FAR are basically given by the metal detectors, which are commercial ones in both systems. The reductions of FAR and POD can be seen as contributions of the GPR. The reductions are plotted in Figure 3. In this figure, FAR reductions are plotted with respect to POD reductions; therefore an ideal dual-sensor system that can perfectly discriminate targets gives a plot on the upper left portion of the graph. If a system uses random chance to determine whether a mine is present, the plot lies on the diagonal line. Both the ALIS and the Gryphon give plots above the diagonal line, therefore the GPRs in both systems are contributing to the decision-making. The Gryphon gives larger FAR reduction than the ALIS; however, the POD reduction is also larger than that by the ALIS. The difference in the POD reductions is not so significant considering the 95% confidence limit, but devices for demining must avoid the POD reduction as much as possible. The results suggest that the Gryphon can reduce FAR more than the ALIS can. However, the absolute level of FARs is almost the same as shown in Figure 2 and the larger FAR reduction is due to a larger number of false alarms given by the metal detector implemented in the Gryphon. Therefore, performances of the whole system as dual sensor in terms of FAR can be characterized as almost the same.



Figures 4a and 4b: POD given by the metal detector (solid lines) and by bothe sensors (dotted lines), and discrimination ratio for mines (dotted broken lines) with respect to target burial depths, given by the ALIS (top) and the Gryphon (bottom).

Figure 4 shows probability of detections given by the metal detector and by both sensors, along with the discrimination ratio with respect to depth for each device. As the theory in the Das and McFee article 12 states and former trials verified, the PODs given by the metal detectors are decreasing with depth. Since the GPRs are always used after the metal detectors, the PODs used by the dual sensors cannot exceed those by the metal detectors. It can be observed that the PODs by both sensors positively correlate with the PODs by the metal detectors.

Furthermore, discrimination ratios tend to increase with depth in these results. This fact cannot be determined conclusively because the number of mines belonging to each depth class is small and the estimation would not be sufficiently accurate. This tendency supports a common theory that GPR has difficulties in detecting shallowly buried targets since reflections from the ground surface mask those from targets.13 However, this observed tendency is not as strong as expected; both systems achieved about 0.7 of the discrimination ratio at the depth range from 0–3cm, so the theory cannot clearly be confirmed. This may be because both sensors measured data of GPR as images in terms of horizontal slice and this type of representation may be good at depicting small changes close to the surface, unlike only one-time signals or a vertical slice.


The ALIS dual-sensor system evaluated in the test.

Conclusions and Discussion

The results of the test campaign for the dual-sensor systems tell us that those systems reduced false-alarm rates significantly by more than one-half. However, the systems also reduced probability of detections, which must be avoided in real clearance operations. Usefulness of the dual sensors may strongly depend on improvements with POD.

The full report4 stated that the three deminers who worked on the ALIS achieved different results in terms of POD, FAR and working hours. The variation may be caused by the way the deminers interpret the output of the sensor and make decisions when operating the ALIS. The visual interpretation of images and decision-making process are entirely subject to the operators themselves. In order to avoid unstable and/or unexpected results, further developments/improvements, such as an automatic-recognition algorithm, are recommended.

Unfortunately, it was not possible to use stand-alone metal detectors at the same time as a benchmark, making a direct comparison of dual sensors to stand-alone metal detectors unavailable. However, one can roughly compare the detectors to those from the STEMD trial,5 taking into account additional metals. The ALIS and the Gryphon needed approximately five and nine minutes, respectively, to survey an average of one square meter. It can be roughly estimated that the ALIS may be two to three times slower and the Gryphon may be four to five times slower than stand-alone metal detectors.14 Even if the search speed in this test is slower than for a stand-alone metal detector, it is possible that these dual sensors would accelerate the clearance operation in total, because rejected alarms from metals would reduce the need for excavation or could be rapidly excavated. Increased search speed would also multiply these benefits.

Another dual-sensor trial in Germany was carried out in September 2009 by the International Test and Evaluation Program for Humanitarian Demining and led by the German Federal Office of Defense Technology and Procurement.15 The results are being analyzed and we hope that a more detailed evaluation of dual-sensor performance will be available soon.

The authors acknowledge Mr. N. Pavković and Mr. T. V. B. Vondracek from HCR-CTRO for managing the test. We also thank the developers and deminers that participated.

Biographies

Kazunori Takahashi is a Research Scientist formerly employed with Federal Institute for Materials Research and Testing (BAM) and currently at Leibniz Institute for Applied Geophysics. His research activities include development and evaluation of dual-sensor systems for humanitarian demining, signal processing for GPR, and reliability analysis of non-destructive testing methods.


Mate Gaal is a Research Scientist at BAM who previously worked on testing metal detectors and dual sensors. He earned his doctorate on experimental design for tests of demining equipment. Currently he works on developing sensors for ultrasonic testing.


Dieter Gülle is a retired colonel who has worked since 1995 in different positions connected with minefield survey, clearance operations and testing of clearance equipment. He represented Germany in the International Test and Evaluation Program for Humanitarian Demining as head of the ITEP Secretariat. He is currently a consultant to the Federal Office of Defense Technology and Procurement in Germany.


Endnotes

  1. Sato, M., and Takahashi K. “ALIS–A Hand Held Dual-Sensor with Imaging Capability.” Mine Action Technology Newsletter. No. 9, January 2009. http://www.mineaction.org/downloads/1/MA-Technology-Newsletter-No9-February2009.pdf. Accessed 23 October 2009.
  2. Detectors and Personal Protective Equipment Catalogue 2009. Geneva International Centre for Humanitarian Demining. January 2009. http://www.gichd.org/fileadmin/pdf/publications/PPE-Catalogue-2009/PPE-Cat-2009-Complete-UpdatedJune2009.pdf. Accessed 23 October 2009.
  3. Freese, M., Fukushima, E.F., and Hirose, S. “Improved Landmine Discrimination With an Off-the-Shelf Metal Detector.Journal of Mine Action, Issue 12.1 (Summer 2008). http://www.jmu.edu/cisr/journal/12.1/rd/freese/
    freese.shtml
    . Accessed 23 October 2009.
  4. Pavković, N., Ishikawa, J., Furuta, K., Takahashi, K., Gaal, M., and Gülle, D. Test and Evaluation of Japanese GPR-EMI Dual Sensor Systems at Benkovac Test Site in Croatia: HCR-CTRO_TECH_GPR08-001. Croatian Mine Action Center-Center for Testing, Development and Training. Zagreb, Croatia, 2008. http://www.itep.ws/pdf/
    TestDualSensorJST_CTRO2007.pdf
    . Accessed 23 October 2009.
  5. Gülle, D., Gaal, M., Bertović, M., Müller, C., Scharmach, M., and Pavlović, M. South-East Europe Interim Report Field Trial Croatia (Continuation of the ITEP-Project Systematic Test and Evaluation of Metal Detectors – STEMD). Berlin, March 2007. http://www.itep.ws/pdf/STEMD_Interim_Croatia_final.pdf. Accessed 23 October 2009.
  6. Takahashi, K., Pavlović, M., Gaal, M., and Müller, C. “Soil Measurements and Modelling.” ITEP Project Report 2.4.2.1, 2007. http://www.itep.ws/pdf/SoilMeasurementsAndModelling.pdf. Accessed 20 November 2009.
  7. Preetz, H., and Igel, J. Pedological Description and Magnetic Susceptibility of the Natural Soil Nearby the Test Site of the Croatian Mine Action Center in Benkovac, Croatia. Leibniz Institute for Applied Geosciences. Hannover, November 2006. http://www.itep.ws/pdf/SoilBenkovac_11_2006.pdf. Accessed 23 October 2009.
  8. “CEN Workshop Agreement, Test and Evaluation of Metal Detectors.” CWA 14747, June 2003. http://www.itep.ws/pdf/CWA_metal_detectors.pdf. Accessed 23 October 2009.
  9. Simonson, K.M. Statistical Considerations in Designing Tests of Mine Detection Systems: I–Measures Related to the Probability of Detection. SAND98-1769/1, Sandia National Laboratories, 1998.
  10. Simonson, K.M. Statistical Considerations in Designing Tests of Mine Detection Systems: II–Measures Related to the False Alarm Rate. SAND98-1769/1, Sandia National Laboratories, 1998.
  11. Gaal, M. “Trial Design for Testing and Evaluation in Humanitarian Mine Clearance.” Ph.D. thesis, Brandenburg Technical University. Cottbus, Germany, 2007. http://www.itep.ws/pdf/abstracts/Gaal_abstract.pdf. Accessed 23 October 2009.
  12. Das, Y., McFee, J.E., Toews, J., and Stuart, G.C. “Analysis of an Electromagnetic Induction Detector for Real-Time Location of Buried Objects.” Geoscience and Remote Sensing, IEEE Transactions. Vol. 28, pp. 278–288. May 1990.
  13. Brunzell, H. “Detection of Shallowly Buried Objects Using Impulse Radar.” Geoscience and Remote Sensing, IEEE Transactions. Vol. 37, pp. 875–886. March, 1999.
  14. See document referenced in endnote 5 for the STEMD trial search speed of stand-alone metal detectors.
  15. Borry, F. “An Update on the ITEP Program and Activities.” The Journal of ERW and Mine Action, Issue 12.2 (Winter 2008/2009: 98–100). Available at http://www.jmu.edu/cisr/journal/12.2/rd/borry/borry.shtml. Accessed 20 November 2009.

Contact Information

Kazunori Takahashi
Research Scientist
Leibniz Institute for Applied Geophysics
Stilleweg 2
30655 Hannover / Germany
Tel: +49 511 643 3572
Fax: +49 511 643 3665
E-mail: kazunori.takahashi@liag-hannover.de
Web site: http://www.liag-hannover.de

Mate Gaal
Research Scientist
BAM – Federal Institute for Materials
Research and Testing
Unter den Eichen 87
12205 Berlin / Germany
Tel: +49 30 8104 3174
Fax: +49 30 8104 1845
E-mail: mate.gaal@bam.de
Web site: http://www.bam.de

Dieter Gülle
Consultant
Federal Office of Defense Technology and Procurement
Stubenrauchstr. 65d
Berlin 12487 / Germany
Tel: +49 306 366 418
E-mail: dieter.guelle@xyz.de