Developing a Hyperspectral Non-Technical Survey via UAV and Helicopter

by Milan Bajić, Ph.D., Tamara Ivelja, and Anna Brook, Ph.D. - view pdf

Figure courtesy of the authors.
Figure 1. The considered topics.
Figure courtesy of authors.
Figure 2. The Pearson mine roller. Photo courtesy of Pearson Engineering.
Figure 2a.

The main topic discussed in this paper is the research, development, and implementation of hyperspectral remote sensing technology in humanitarian mine action (HMA), mainly for non-technical survey (NTS) from aerial platforms (Figure 1).1 NTS should be conducted to determine whether landmines or unexploded ordnance (UXO) exist in considered areas and whether clearance is needed and if so, in what scope. The availability of the hyperspectral sensors (2011–2012) that are suitable for use with multi-engine unmanned aerial vehicles (UAV) enabled the development of systems for HMA. While large and costly airborne hyperspectral systems have been available for over two decades, they had little to no impact on HMA.2 Optical spectroscopy brought out interest in the relationship between vegetation and explosives, and this relationship is considered herein. Therefore, we briefly analyzed the state of landmine detection via hyperspectral technology, particularly the efforts to detect the mines due to the influence of explosives on vegetation reflectance spectra.

The Croatian Mine Action Centre for testing, development and training Ltd. (HCR-CTRO) contributed to the development and implementation of hyperspectral technology in mine action and is also discussed here. Our activity in the domain of HMA started with the European Commission FP5-IST project ARC and continued to the European Commission’s Framework Programme 7 (FP7) project: TIRAMISU.3,4 In this program, the Specim ImSpector V9 hyperspectral line scanner was integrated as a push broom mode acquisition system for surveying minefields and UXO-contaminated areas, as well as for other purposes.5,6,7,8,9,10,11,12 In 2015, a full frame hyperspectral sensor, the Cubert UHD-185, was applied.4,13 Both types of sensors (line scanner and full frame sensor) were integrated into aerial acquisition and navigation systems on board a UAV and a Bell-206 helicopter, as well as on ground vehicle platforms in technically simpler versions. These were involved in the intensive data collecting campaigns over the minefields.

Figure 2b
Figure 2b.
Figure 2c.
Figure 2c.

Figure 2 (a-c) Modes of hyperspectral data collecting at W wavelengths, typical W > 90.
Figure courtesy of the authors.
Figure 3a. The hyperspectral detection of the difference between grassy vegetation IN and OUT of the minefield. 
Figure courtesy of the authors.
Figure 3a. The hyperspectral detection of the difference between grassy vegetation IN and OUT of the minefield.
Figure courtesy of the authors.
Figure 3b. Bushes and trees are excluded from hyperspectral data. 
Figure courtesy of the authors.
Figure 3b. Bushes and trees are excluded from hyperspectral data.
Figure courtesy of the authors.
Figure 4a. The gray area of the Murgići minefield (> 504,000 sq m) near Lički Osik, Croatia, for which hyperspectral data were collected via helicopter.
 
Figure courtesy of the authors.
Figure 4a. The gray area of the Murgići minefield (> 504,000 sq m) near Lički Osik, Croatia, for which hyperspectral data were collected via helicopter.
Figure courtesy of the authors.
Figure 4b. The small subarea selected from demined one (blue delineated polygon). 
Figure courtesy of the authors.
Figure 4b. The small subarea selected from demined one (blue delineated polygon) .
Figure courtesy of the authors.
Figure 5. The spectral features were measured for a number of UXO items.
Figure 5. The spectral features were measured for a number of UXO items.
Figure 6a. Mi-8.
Figure 6a. Mi-8.
Figure 6b. Radio-controlled blimp.
Figure 6b. Radio-controlled blimp.
Figure 6c. Eight-engine unmanned aerial vehicle (UAV).
Figure 6c (left). Eight-engine unmanned aerial vehicle (UAV).
Figure 6d. Four-engine UAV with UHD-185 sensor.
Figure 6d. Four-engine UAV with UHD-185 sensor.
Figure 6e. Bell-206 with V9 line scanner and UHD-185 frame sensor. Figure 6e. Bell-206 with V9 line scanner and UHD-185 frame sensor.
Figure 6f. Sensor pod on Bell-206.
Figure 6f. Sensor pod on Bell-206.
Figure 7a.
Figure 7a.
Figure 7b.
Figure 7b.
Figure 7c.Figure 7. Ground-based vehicle platforms for collecting hyperspectral data: a) for collecting nadir data, b) for collecting oblique data, and c) geometry for collecting oblique data.9 
Figures courtesy of authors.
Figure 7c.

Figure 7. Ground-based vehicle platforms for collecting hyperspectral data: a) for collecting nadir data, b) for collecting oblique data, and c) geometry for collecting oblique data.9
Figures courtesy of authors.
Figure 8a and b.
Figure 8a and b.
Figure 8c. Figure 8 (a-c). Geometry of collecting airborne hyperspectral data a) with a line scanner, narrow rectangular areas (named - lines) are scattered due to pitch, roll, and yaw of the aerial platform. b) The parametric geocoding is needed to provide a hyperspectral images. c) The frame hyperspectral cameras produce a hyperspectral set of images for each frame (note white borders of the frames). 
 Figures courtesy of authors.
Figure 8c.

Figure 8 (a-c). Geometry of collecting airborne hyperspectral data a) with a line scanner, narrow rectangular areas (named - lines) are scattered due to pitch, roll, and yaw of the aerial platform. b) The parametric geocoding is needed to provide a hyperspectral images. c) The frame hyperspectral cameras produce a hyperspectral set of images for each frame (note white borders of the frames).
Figures courtesy of authors.
Figure 9a.
Figure 9a.
Figure 9b. Figure 9 (a-b). Sun irradiance changes in the day. a) The absolute irradiance during working hours a) at 10:00 and b) at 19:32 local time. 
Figures courtesy of authors.
Figure 9b.

Figure 9 (a-b). Sun irradiance changes in the day. a) The absolute irradiance during working hours a) at 10:00 and b) at 19:32 local time.
Figures courtesy of authors.
Figure 10a.
Figure 10a.
Figure 10b. Figure 10. Radiometry calibration of the hyperspectral data collected from UAV. a) Reflectance spectra of Spectralon at nominal levels 12%, 25%, 50% and 90% reflectance. b) Reflectance marker Spectralon. 
Figures courtesy of authors.
Figure 10b.

Figure 10. Radiometry calibration of the hyperspectral data collected from UAV. a) Reflectance spectra of Spectralon at nominal levels 12%, 25%, 50% and 90% reflectance. b) Reflectance marker Spectralon.
Figures courtesy of authors.
Figure 11a.
Figure 11a.
Figure 11b.
Figure 11b.
Figure 11c.
Figure 11c.
Figure 11d.Figure 11. a) Sensor operator can check the reflectance spectra of objects and terrain below the helicopter. b) Example of objects below the helicopter during the acquisition flight, c) reflectance spectra of areas from d). Note that the same colors are used for selection of areas on c) and d) for presenting their reflectance.
Figures courtesy of authors.
Figure 11d.

Figure 11. a) Sensor operator can check the reflectance spectra of objects and terrain below the helicopter. b) Example of objects below the helicopter during the acquisition flight, c) reflectance spectra of areas from d). Note that the same colors are used for selection of areas on c) and d) for presenting their reflectance.
Figures courtesy of authors.
Figure 12a.
Figure 12a.
Figure 12b.
Figure 12b.
Figure 12c.
Figure 12c.
Figure 12d.
Figure 12d.
Figure 12e.
Figure 12e.
Figure 12f.
Figure 12f.

Figure 12 (a-f). Application of the feature mapping and computer-assisted photo interpretation (CAPI) on the hyperspectral images at selected wavelengths for discrimination of the field road and the meadow with grassy plants from trees. Figures courtesy of authors.
Figure 13a. Hyperspectral image of a target (the mortar mine M60) visualized by narrow band (10 nm) images (central wavelengths red – 650 nm, green 550 nm, blue 450 nm).
Figure 13a. Hyperspectral image of a target (the mortar mine M60) visualized by narrow band (10 nm) images (central wavelengths red – 650 nm, green 550 nm, blue 450 nm).
Figures courtesy of authors.
Figure 13b. Spectral Angle Mapping (SAM) for angle 3, 7 degrees reconstructs the shape of the target.
Figure 13b. Spectral Angle Mapping (SAM) for angle 3, 7 degrees reconstructs the shape of the target.
Figures courtesy of authors.
Figure 14a. Radiance spectra of the target M60, soil, grass, and calibration lines of 12%, 25%, 50%, and 90% radiance.Figures courtesy of authors.
Figure 14a. Radiance spectra of the target M60, soil, grass, and calibration lines of 12%, 25%, 50%, and 90% radiance.
Figures courtesy of authors.

The basic operational features of the considered hyperspectral sensors are explained in Figure 2. Line scanners (2a) collect hyperspectral data for W wavelengths in successive lines. Technically complex, line scanners produce results that are available after time-consuming processing. Near real-time, full-frame sensors (2b) include images in visible wavelengths (blue), sharpening the spatially coarser hyperspectral image (red) at W wavelengths of the same area. The intrinsic hyperspectral spatial resolution is Q times coarser (e.g., Q = 20). Near real-time control of collected hyperspectral data is possible. Near real-time sensors (2c) provide hyperspectral data at any K of W selected wavelengths, K < W (typical K = 30, W = 300) at full spatial resolution.

Besides the development of a hyperspectral survey system suitable for NTS, our goal was to implement the methods and technology for NTS in minefields by applying spectroscopy. The initial goal was to analyze and verify the vegetation indices that show differences between the mixture of grassy vegetation inside (IN) and outside (OUT) of the minefield. For this purpose, minefields in Croatia, near Benkovac (Figure 3) and Murgii, near Lički Osik (Figure 4) were surveyed.

Hyperspectral data (Figure 3a) were collected simultaneously on the OUT (43,323 sq m) and IN (10,000 sq m) areas of the Benkovac minefield (containing 1,000 mines), where 5,900 sq m of the minefield is aimed for the blind statistical tests. Public-use data on the location of 150 mines exists for part of the minefield. Bushes and trees hyperspectral data were excluded manually from data of the IN and OUT areas.

In our approach, the features and characteristics of the particular vegetation types were not analyzed. Instead, a grassy vegetation mixture (IN and OUT) of the considered area was taken into account. This is a crucial fact on which we continued our research and development. Differences in the vegetation’s spectral reflectance (both IN and OUT) were perceived and confirmed in a case of the Benkovac minefield. After this step, vegetation indices were analyzed, and we confirmed that indices values from the IN area differ from those of the OUT areas.4,10,13

Besides the references linked to our work, a large amount of valuable information on hyperspectral topics exists. Used for military applications, this technology can analyze the variability of terrain signatures, produce hyperspectral images for data fusion, and detect anomalies.14,15,16,17,18

Hyperspectral Detection

Some of the earliest information on hyperspectral detection of buried mines via the spectral changes in vegetation was presented in 1997 (McFee, Ripley), 2007 (McFee et al.) and later in 2010 (Yoresh).19,20,21 In 2007, McFee et al. reported that “The detection of buried mines using a CASI hyperspectral imager from personnel lift at 5 m distance, under various vegetative covers and bare soil at various times from 1 2/3 to 15 1/2 months after burial. Short vegetation gave better detection results than bare soil which gave better results than medium length vegetation. Probability of detection (Pd) was typically in the range of 55 to 94 percent and False Alarm Rate (FAR) varied from about 0.17/m2 to 0.52/m2.”20 In 2010, Yoresh stated that “Processing of the imagery was applied into a mosaic of the geometrically rectified 125 channels HyMap, data which was atmospherically corrected. It allows identifying suspected minefields, in the size of 30 x 30 meters from satellite photo, and of smaller size of 1 x 1 meter from aerial photos.”21

The longwave infrared (LWIR) hyperspectral method of detecting buried mines is based on sensing soil surface changes.22 The extremely high cost of LWIR hyperspectral sensors limits their application when compared to the prices of visible, near-infrared, and shortwave wavelength sensors.

The hyperspectral method of detecting mines on the surface in visible, near-infrared light and shortwave infrared light is described by McFee et al. as “Individual mines [that] could be distinguished from backgrounds in all seasons and times of day between sunrise and sunset, even when they were partially obscured by vegetation. High spatial resolution airborne imagery of real and surrogate minefields has also been obtained from the helicopter (0.1 m x 0.1 m at 75 m altitude to 0.2 m x 0.2 m at 150 m altitudes). Analysis yielded Pd of nearly 100 % with FAR of 0.0003 per sq m and 0.004 per sq m. Real-time analysis, necessary for military applications, has been demonstrated from low speed ground vehicles and recently from airborne platforms. Down-welling radiation sensors are necessary to allow detection of mines independent of natural illumination, since calibrated panels are not practical in the field.”20

The process of using hyperspectral imagery to detect unexploded ordnance (UXO) that has been scattered by unplanned explosions of munition stockpiles is discussed in (Foley, Patterson 2007), (Bajić et al. 2013), (Yvinec et al. 2015). The wide area assessment of UXO distribution at and around ammunition depots post explosion can be obtained by combining aerial hyperspectral and very high spatial resolution visible (VHR VIS) images. A similar approach was applied where hyperspectral samples were collected by ground based system and VHR VIS sensors attached to helicopters and UAVs.8,23,24 For TIRAMISU 2012, we collected hyperspectral and VHR VIS data and images of UXO, soil, and vegetation relevant to unplanned explosions of munition stockpiles.4

Assessing former military test sites with hyperspectral imaging to detect the UXO and abandoned ordnance (AXO) on or below the surface was considered in 2007 (Foley, Patterson).8,23,24 Researchers concluded that the wide area assessment of UXO distribution in the former military test range could be obtained by combined aerial hyperspectral and VHR VIS images.23 The Geneva International Centre for Humanitarian Demining (GICHD) observed a similar approach that was applied to a former military shooting range and concluded that the statements about probability and reliability of results are not strong enough and are not supported by evidence.25

Using hyperspectral imaging to detect residual contamination on military test sites that will be used for civilians once clearance has been completed is a standard problem.26 For former military test sites (i.e., shooting ranges), clearance requires convincing evidence and verification that levels of residual contamination are at acceptable thresholds. The hyperspectral aerial survey and ground truth hyperspectral measurements provide a basis for success.

The hyperspectral survey of minefields was reported in 2010 (Yoresh), whereas the observer considers substantial limitation of the method due to vegetation’s role in the method.21,25,27

Partial analyses of spectral features of plants fed by explosives appear quite often. A very simple goal was defined: “To determine if a technique called hyperspectral imaging can be used as an accurate tool for detecting buried explosives.”28 In 2014 (Smit et al), the analysis was serious: “Indigenous trees in 20 liter pots contaminated with different TNT concentrations, plants were kept in semi-natural conditions, concentrations used were from 30 mg/kg, to 5000 mg/kg.29 Initial results show that there are impacts. Some trees showed more impact than others. The plants healthier than their neighbors could indicate the presence of a minefield : toxins from explosives leak into the surrounding environment, affecting plant health, especially herbaceous plants more than woody plants.30,31 They are considering creating an ‘Explosive Specific Index’ that will record how explosives affect various vegetation types.” In 2016, Manley reported: “Three species representing different functional types (Cyperus esculentus, a sedge; Ulmus alata, a tree; Vitis labrusca, a vine) exposed to RDX and TNT over a nine-week period.32 Woody species, exhibited changes in pigment content, leaf area, specific leaf area, dry leaf biomass, and canopy reflectance.”

The hyperspectral detection of illicit plants is a technology suitable for counter-drug survey in case of combined drug plants fields with mines and the improvised explosive devices (IED) in certain countries e.g., coca fields in Colombia.33,34 The availability of a new generation of hyperspectral sensors (full frame cameras), which provide data in near real time, is key to the implementation of this technology.

In the last several years, UAVs were frequently considered for detection purposes, although the first fully operational project based on UAV platforms for mine action was done in 2014.35 In future projects, the use of the full frame hyperspectral cameras will significantly advance results of the survey.

Methods and Materials

Experience from several airborne-based NTS projects enabled a unique and holistic approach to the problem of large suspected hazardous areas (SHA), the minefields survey and the challenges and chances for hyperspectral technology.35,36 SHA is an area where there is reasonable suspicion of mine and explosive remnants of war (ERW) contamination on the basis of indirect evidence of the presence of mines and ERW.

Objectives and Goals

Our current work in a hyperspectral domain was led by the following goals:

  1. Goal 1: Develop, test, evaluate, and operationally validate a system suitable for hyperspectral NTS based on the application of a multi-engine UAV and currently available hyperspectral sensors.
  2. Goal 2: Collect and prove that the pure spectral samples (endmembers) data for considered UXO enable their detection.
  3. Goal 3: Prove that the spectral vegetation indices enable discriminating grassy vegetation IN and OUT of the minefield.

Objectives:

The considered SHA and minefield (IN) surrounding the vegetated area (OUT) were available with the same features (soil, moisture, vegetation). This area was not cultivated from the time the mines were buried until survey. The hyperspectral data acquisition of the SHA, minefield and surrounding area were conducted simultaneously in the same acquisition flight.

Since 2002, we have used the hyperspectral system for analyses in the ecology and forestry sectors.5,11,12 The initial key influence on our goals was made in 2011 by Šestak, who supported the concept that the mixture of grassy vegetation should be the representative of the situaton in the considered minefields and SHA instead of one preselected vegetation type.37

Minefields 

Within the scope of the TIRAMISU project, two approaches were defined regarding the hyperspectral detection of minefields and SHA. The aim of the first approach was to develop, test, evaluate, and validate a system for hyperspectral data collection via UAV and helicopter that is suitable for application in NTS, to collect pure spectral samples (endmembers), to derive hyperspectral cubes for considered set of UXO, and to prove that the vegetation indices can discriminate between the grass IN and OUT of the minefield. The work on the first approach was continued, and hyperspectral survey flights have been made by UAVs and helicopters on SHAs and minefields for a total area of 514,000 sq m. Survey was done from lower altitudes, and valuable data with high spatial resolution were gained, Figure 3 and Figure 4. The second approach was large-scale, aerial, hyperspectral acquisition of SHAs and minefields in different terrain, climate, vegetation, and mine contamination. It was planned to survey around 136 sq km of SHAs and minefields with a HySpex sensor (visible, near infrared, shortwave infrared). Also, the ground truth mission for that area was planned and prepared. The main expected benefit should have been a significant set of hyperspectral data representing the above-mentioned areas of interest. Based on these data, it would be possible to differentiate land cover classes and derive reliable statistics of vegetation characteristics and features difference from IN and OUT of the contaminated areas. Unfortunately, these surveys were never conducted.

Figure 4a depicts the SHA and Murgići minefield near Lički Osik in Croatia. The image of Murgići is visualized by narrow wavelengths (8 nm) spectral images (central wavelengths: red = 650 nm, green = 550 nm, blue = 458 nm). Areas in the blue delineated polygons have been demined. This area, with a minefield near Benkovac, served as the research, development, testing, evaluation, and validation area. Figure 4b shows a small example area selected from the demined area that demonstrates the hyperspectral survey potentials for NTS. This area is hilly and has a continental climate.

Figure 3a depicts the Benkovac minefield. The red polygon outlines mined area 50 x 200 m. The black line is the border of the region of interest (ROI). The hyperspectral data and very high resolution color images were collected via helicopter and UAV. This area has a rocky terrain and a subcoastal climate. It contains 1,000 mines with very high density of 0.1 mine per sq m, and for that reason, served as the main area for our research. In the Benkovac minefield, hyperspectral acquisition was conducted via light helicopter, UAV, and ground vehicles in 2012, 2013, and 2015.

The Prototyping Model was applied in research and development of the hyperspectral acquisition system. The Prototyping Model is a systems development method in which an early approximation of a final system is built, tested, and then reworked as necessary until an acceptable prototype is finally achieved. This model works best in scenarios where not all of the project requirements are known in detail ahead of time. It is an iterative, trial-and-error process that takes place between the developers and the users. The main difficulty is the need to understand the collected hyperspectral data from the minefields, and the ability to correct and calibrate the data, analyze the results, and decide on the next step in the system development.10 This process is reflected later in the article.

Unexploded Ordnance

Addressing the on-surface issue was done by examining the exploded ammunition depot of Pađene (near the town of Knin, Croatia), in which an explosion occurred in 2011 and large amounts of UXO were scattered in an approximately 5 km radius area. The area was under ground-based and airborne survey, during which hyperspectral UXO data was collected.8,24 The main goal was to provide a reliable ground truth for discriminating UXO from its surrounding (soil, vegetation, etc.).

Sensors, Platforms, and Hyperspectral Data Acquisition

A helicopter-mounted, multi-sensor, aerial acquisition system with a hyperspectral line scanner was initially developed in the ARC project and later upgraded.3,4,5,6,7,8 It was the first non-military system for airborne NTS. Several kinds of aerial platforms have been used in research and development of hyperspectral technology for minefield surveys. To achieve quality data, the following requirements for the platform have been defined:

These requirements are largely related to survey with the V9 hyperspectral line scanner, Figure 2a, and the complex procedure of pre-processing hyperspectral images by parametric geocoding.6,7,10 Several data acquisition systems were developed for multiple platforms: light and heavy helicopters (Bell-206b, Gazela, Mi-8), a radio controlled blimp and different types of UAVs, Figure 6. The acquisition systems are always assembled from several components: acquisition computer, communication unit, GPS receiver, and electric power supply. Moreover, the V9 hyperspectral line scanner requires inertial measuring unit (IMU) on board the aerial platform for measurement of its pitch, roll, and yaw. That much equipment encumbers the aerial platform significantly. For helicopters, the payload parameter was not a problem, while for the blimp (Figure 6b) and multi-engine UAVs (Figure 6c and Figure 6d), this is the most critical issue limiting flight performances and endurance. In 2015, we introduced a new hyperspectral sensor, a full-frame hyperspectral camera UHD-185 (generation 2012), with which we substituted the V9 line scanner used before (generation 2000). Application of the new camera enabled us to finalize successful development of the system and technology for aerial NTS survey via UAV and helicopter: Figure 6c-f. Technical stability and robustness of the system was confirmed during intensive data collection missions in Croatia, 2015. Developed acquisition systems on a light helicopter Bell-206B and multi-engine UAVs were approved and validated by the Croatian Mine Action Centre (CROMAC).4

The research of the basic behavior of hyperspectral features, potentially relevant for the goals and the objectives defined previously, was conducted on the Benkovac minefield, Figure 3a (page 50), using ground-based vehicle systems, Figure 7.9 Through the ground-based systems’ nadir mode, hyperspectral data and respective color video data were collected during 2012, Figure 16. The ground-based hyperspectral survey systems show high sensitivity and very high spatial resolution of hyperspectral data of the minefield in flat terrain, but more processing is needed to overcome imaging geometry problems due to a rough terrain and to properly link the positions of buried mines with the hyperspectral data. A developed technique and technology of hyperspectral imaging from a UAV was operationally validated as a useful one, and it can completely substitute the ground-based hyperspectral survey.4

Geometry of Acquisition

Data collected by the V9 line scanner are spatially scattered due to pitch and roll of the aerial platform, see Figure 8a. After parametric geocoding, the lines are transformed to form a row of parallel lines, Figure 2a, and the result is the corresponding hyperspectral cube, Figure 8b. Data collected by hyperspectral frame sensor (model in Figure 2b) produce the hyperspectral cubes, their borders on images are white overlapping rectangles, Figure 8c.

The hyperspectral frame cameras that work in mode Figure 2c have smaller number of wavelengths (channels), K < W, but their hyperspectral images enable mosaicking due to their full spatial resolution in comparison to cameras, which work in accordance to the model in Figure 2b.

Calibration

Changes in the Sun’s irradiance, see Figure 9, have strong negative impacts on the quality of the collected hyperspectral data and consequently on the airborne hyperspectral NTS operations. The down-welling radiation sensors are necessary if the calibrated panels are not practical in the field.20 The radiation sensor FODIS of the V9 line scanner can be the source of errors, whereas the frame sensor UHD-185 counts on a calibrated panel Spectralon for near range missions if done via UAV, Figure 10. If the frame sensor is used from the helicopter, larger calibrating objects should be used, while the measurements have to be done at the take-off and landing flight phases. The supervised vicarious calibration (Brook, Ben Dor 2011) is operationally the most suitable one.38 The insolation variability limits hyperspectral survey by UAV at shorter time spans and at smaller ranges. Alternatively, helicopters are necessary platforms, as the speed is greater than the speed of the UAVs. Figure 9 shows an example of the changes of the absolute irradiance during working hours.

Near Real Time Control of Collected Hyperspectral Data

The near real-time frame hyperspectral sensors (frame sensor), which operate in accordance to Figure 2b, enable control of the data collecting from the helicopter, see Figure 11. This is the most important operational advantage for hyperspectral NTS of SHAs, as minefields in some areas are out of range for UAV platforms. The technology that is currently available does not enable near real-time control of data while flying UAVs, but it is possible after landing.

Demonstrating the Benefits of NTS and UXO Detection by Selected Processing

The hyperspectral remote sensing, data processing, and application in many domains are very developed, and basic information is available in many sources.14,15,17,18,37,39,40 While our goal is to enable aerial hyperspectral NTS of SHAs and minefields and the survey of UXO, we will select two simple examples that clearly demonstrate the convincing benefits that are operationally feasible. In these examples, we apply the following processing methods: (1) the Feature Mapping method, using the data of selected wavelengths and (2) the Spectral Angle Mapping (SAM) method of processing hyperspectral data.29,39 In these examples, we avoided thorough quantitative analyses (e.g., by confusion matrix), while the visual comparison of images verifies effects.

The Feature Mapping Method of Processing of Hyperspectral Data

The Feature Mapping method enables the visual interpretation of a reference image to guide the automated classification procedures applied to any number of images at different wavelengths. Results are the set of spectral categories or feature classes. This method takes advantage of visual interpretation skills, and the classification can be simply programmed. Another advantage is the application of the subset of any K < W hyperspectral images, W is maximum number of images in used wavelength range from λmin to λmax. Therefore this method is suitable for hyperspectral sensors that operate in modes, Figure 2c, that provide full spatial resolution of hyperspectral data in K spectral wavelengths. The example in Figure 12 demonstrates useful outcomes of Feature Mapping applied to eight wavelengths from 135 available wavelengths, Figure 12a, Figure 12d, Figure 12e, three wavelengths in Figure 12f, and only two wavelengths in Figure 12b and Figure 12c. The eight wavelengths were selected after reflectance spectra analysis in the considered hyperspectral scene from Figure 4b, the aim was to discriminate between the road, grassy meadow, and trees. The selected wavelengths were 450 nm, 502 nm, 550 nm, 602 nm, 650 nm, 702 nm, 750 nm, and 802 nm.

The Benefits for NTS

  1. The grassy vegetation in meadow Figure 12d are easily discriminated from trees Figure 12e, while in the TIRAMISU 2012 project, this was not solved.
  2. The road is partially covered by grass but can be detected from eight wavelengths, Figure 12a.
  3. The normalized difference vegetation index NDVI detects the road, Figure 12b. Here is NDVI = (NIR-R)/(NIR+R), where NIR is the image at wavelengths 750 nm, 702 nm, or 802 nm, R is the image at wavelength 650 nm. Figure 12c shows negative of NDVI from Figure 12b.
  4. If three images are selected that present red, green, and blue parts of visible spectra (650 nm, 550 nm, 450 nm), trees cannot be reliably classified, Figure 12f, whereas eight wavelengths, Figure 12e, are successful.

Note that for the Feature Mapping method to be successful, a selection of wavelengths is needed. While in the worst case, one can select more of them and gain the experience after several trials at a current scene.

Hyperspectral Detection of UXO by SAM method

SAM is a method that uses the angle between any two spectral vectors originating from a common origin, where the magnitude of the angle indicates the similarity or dissimilarity of the materials—a smaller angle correlates with a more similar spectral signature.39 The SAM method of processing was applied to a hyperspectral cube of UXO can be seen in Figure 13. Hyperspectral cube is the name for a hyperspectral images format in which consecutive images follow each other in accordance to increasing wavelengths. The spectral samples are measured from the hyperspectral cube in Figure 13a and stored. Via the SAM method, the shape of the mortar mine M60 in real environment (grass, red soil, and rocks) can be reconstructed, Figure 13b.

The diagram in Figure 14a shows measured radiance spectra of mortar mine M60 and environment, while the Figure 14b shows a radiometry marker Spectralon. Note that for SAM, the radiance data have been calibrated and transformed into reflectance.

The Benefits for UXO Detection

Implementing the Modified Johnson's Criteria for UXO Survey

The probability of successfully conducting aerial survey functions—detection, classification, recognition, and identification—of UXO can be estimated by comparing the spatial resolution of the electro-optical sensor (e.g., its ground resolving distance d) and the UXO dimensions. In the advanced Johnson’s model, the two orthogonal dimensions of UXO (e.g., its length Lx, and width Ly) are combined in their geometrical mean: Lc = (LxLy)1/2.40 Describing the situation by the available number of resolution elements, N = Lc/d, the probably of the successful survey (P) is determined by: P = (N/N50)E/[1+(N/N50)E], E = 2.7+0.7(N/N50) where the parameter N50 equals N needed for P = 0.5. Table 1 indicates the value for N50 for each of four survey functions.

The above model relates the UXO dimensions, sensor resolution and probability of the success of the survey for a given item of UXO. Typical results are illustrated in Figure 15, using the mortar mine M60 from Figure 5.

Detecting the Differences of Grassy Vegetation IN and OUT of the Minefield

The first hyperspectral data of grass inside the Benkovac minefield were collected by the V9 line scanner with a ground based nadir looking system, Figure 7a, Figure 16. Around 1,000 mines are buried in the Benkovac minefield. Locations for 150 mines are available for the public part of Figure 3a, while the mines in the blind test area are aimed at blind statistical tests and their distribution is not available. The mines are randomly distributed in 45 rows, each is 45 m long and 1 m wide. Between rows there are 3 m wide paths, and mines are buried in 1 x 1 m squares.

An interesting fact was that the reflectance spectra of grass IN and OUT of the minefield are different. The reflectance spectra IN the minefield show healthy grass in its full developmental phase, with a local maximum near 550 nm (chlorophyll), a deep minimum near 690 nm, and a step of reflectance after the red edge. The spectra of the grass OUT of the minefield had distorted form, very different from the form of grass in the minefield. In the following phase, many different vegetation indices (from ENVI software list) were tested regarding their potential to quantify results of the mentioned spectral difference. In this analysis, we used the hyperspectral data collected by the V9 line scanner from a helicopter flying at altitudes of 210 m and 400 m. The provided spatial resolution was 0.4 m, data were parametrically geocoded and the radiometric/atmospheric calibrating were applied. Among the considered vegetation indices, six were stable in discriminating grass IN and OUT of the minefield, Table 2. The best results were obtained via carotenoid, whereas the chlorophyll total and CRI indices were obtained from Gitelson 2002, therefore further analysis was done via carotenoid.42 Note that data used for these three indices were collected by near real-time frame sensor UHD-185 from a helicopter, Figure 17. Data for bushes and trees are manually excluded from analysis (white blobs). The red polygon is the border of the whole Benkovac minefield.

The spatial distribution of the carotenoid of the grassy vegetation decreases if the threshold of the carotenoid is increasing, Figure 18. These changes are presented in Figure 19. The probability density function (PDF) proves that vegetation IN and OUT of the minefield can be discriminated.

Discussion

The large and costly airborne hyperspectral systems were available for more than two decades yet without any impact on humanitarian mine action.2 Alternatively, the application of hyperspectral technology in many civilian domains is widely used and well developed and has good results and significant impacts. The availability of hyperspectral sensors, which are suitable for use via multi-engine UAVs, enables the development of systems that can be implemented in humanitarian mine action. In the scope of the 2012 TIRAMISU project, two approaches were defined with regard to hyperspectral detection of minefields and SHAs.4 The first approach was realized by three tasks. The first task was aimed at researching and developing the hyperspectral system for data collecting via multi-engine UAV and helicopter, which are suitable for hyperspectral NTS. The second task was to collect hyperspectral data of UXO inside areas where ammunition depots have exploded, including pure spectral samples (endmembers) of the area and the object of interest. The third task was to analyze and verify the vegetation indices that show the difference between grassy vegetation IN and OUT of the minefield. All three tasks were accomplished. The results, the conditions, and the limitations are presented in the current paper and elsewhere.13 The system for collecting hyperspectral data via multi-engine UAV and helicopter was operationally validated and the hyperspectral NTS functionality is included in the operational TIRAMISU Advanced Intelligence Decision Support System (T-AIDSS), which is the advanced version of AIDSS.4,36

Although the 2012 TIRAMISU project was planned to combine the top-down and bottom-up approaches to the research, we were limited to two minefields (10,000 sq m and 504,000 sq m) instead on the large area (136 sq km = 136,000,000 sq m). Regardless, a simple analysis of the reflectance spectra of the grassy vegetation in the considered minefield and its neighboring areas showed an increase of greenness and vegetation stress, visible in the deteriorated shape of the reflectance response in comparison to the shape of healty vegetation.30,31,32,43 In the analysis of the data collected by the V9 line scanner, we excluded ground (soil) contribution by using the normalized difference vegetation index, while the bushes and trees data were manually excluded for both sensors.

Six of many considered vegetation indices that are verified to be suitable for the discrimination of grassy vegetation IN and OUT of the minefield have been analyzed, whereas only the results for the carotenoid are shown. Future research should apply the fusion methods of available indices where improvement is expected.17

Conclusion

Acknowledgement

The authors are especially grateful to the pilots of the Croatian Air Force for flying the helicopters, and to the UAV pilots of the multi-engine UAV from Geoarheo Ltd. Many persons participated in the development of this hyperspectral aerial survey; the authors are particularly grateful to H. Gold, I. Šestak, T. Kičimbači, S. Šemanjski, M. Krajnović, D. Vuletić, A. Krtalić, M. Tomić, D. Gajski, I. Racetin, and G. Skelac. The research leading to these results received funding from the European Commission’s Seventh Framework Programme FP7/2012–2015 under grant agreement n° 284747, project TIRAMISU.

Biographies

Milan Bajić, Ph.D.
Ret. Lt. Col.

Milan Bajić, Ph.D., Ret. Lt. Col., former professor of remote sensing at the Faculty of Geodesy, University of Zagreb. He is a member of the Scientific Council of HCR-CTRO, Zagreb, Croatia, and was involved in a number of projects and various emergency response, technology developments, capacity building, and advanced intelligence decision support avtivities in Croatia and Bosnia and Herzegovina. He led Croatian teams in the European Commission’s research and development projects on airborne remote sensing for mine action (SMART, ARC, TIRAMISU), and has written more than 50 scientific papers about remote sensing and humanitarian mine action.

Tamara Ivelja
Faculty of Geodesy, University of Zagreb

Tamara Ivelja received a bachelor’s degree in geodesy and geoinformatics in 2009 and a postgraduate master’s degree in geoinformatics from the University of Zagreb in 2009. From 2012 to 2013 she worked for HCR-CTRO and Faculty of Geodesy in Zagreb as a researcher on hyperspectral minefield detection and unexploded explosive devices in the EU FP7 project TIRAMISU. She is currently employed as a teaching assistant at the Zagreb University of Applied Sciences.

Anna BrookAnna Brook, Ph.D.
University of Haifa, Israel

Anna Brook, Ph.D., is a senior lecturer and head of the Remote Sensing Laboratory at the University of Haifa, Israel. She received her Ph.D. in environmental sciences with a thesis on “Reflectance Spectroscopy as a Tool to Assess the Quality of Concrete in situ” from the Porter School of Environmental Studies at Tel-Aviv University in 2010. Her current research interests include image and signal processing, automation target recognition, sub-pixel detection, spatial and temporal models and pattern recognition.

Figure 14b. Hyperspectral image of the Spectralon marker for radiometry calibration. 
Figures courtesy of authors.
Figure 14b. Hyperspectral image of the Spectralon marker for radiometry calibration.
Figures courtesy of authors.
Survey Function N50
Detection 1.5
Classification 3.0
Recognition 6.0
Identification 12.0
Table 1. N50 value for 50% survey.
Table courtesy of M. Bajic.
Figure 14b. Hyperspectral image of the Spectralon marker for radiometry calibration. 
Figures courtesy of authors.
Figure 14b. Hyperspectral image of the Spectralon marker for radiometry calibration.
Figures courtesy of authors.
Figure 16a.
Figure 16a.
Figure 16b.
Figure 16b.
Figure 16c.
Figure 16c.
Figures courtesy of HCR-CTRO.
Figures 16d, e, and f (left to right)

Figure 16 (a-f). Scheme of nadir measurements in the Benkovac minefield 2013. a) Scheme of collecting hyperspectral data. b) Aerial color image of the minefield. c) Legend of ground based nadir survey. d), e), and f) Types of the objects buried in the public part, landmines PMA-2, PMA-3, parts of a rifle grenade.
Figures courtesy of HCR-CTRO.
Figure 17. Distribution of the carotenoid of grassy vegetation in ROI Benkovac. 
Figure courtesy of the authors.
Figure 17. Distribution of the carotenoid of grassy vegetation in ROI Benkovac.
Figure courtesy of the authors.
Figure courtesy of the authors.
Figure 18. Increasing the carotenoid threshold decreases the outside (OUT) area quicker than the area inside (IN). Red font indicates the threshold level, which is changed from minimum (-2) to maximum (44) and scaled in the range from 0 to 255 (8 bits).
Figure courtesy of the authors.
Figure 19. Decrease of the areas with the carotenoid outside (OUT) and inside (IN, IN_B) the minefield near Benkovac. 
Figure courtesy of the authors.
Figure 19. Decrease of the areas with the carotenoid outside (OUT) and inside (IN, IN_B) the minefield near Benkovac.
Figure courtesy of the authors .
Figure 20. Probability density function of carotenoid inside of the Benkovac minefield (PDF_IN_B, PDF_IN_T) and outside (PDF_OUT), the dependence on values of threshold level values. 
Figure courtesy of the authors.
Figure 20. Probability density function of carotenoid inside of the Benkovac minefield (PDF_IN_B, PDF_IN_T) and outside (PDF_OUT), the dependence on values of threshold level values.
Figure courtesy of the authors.


Endnotes

  1. United Nations Mine Action Service. First Edition, Amendment 2, “Non-Technical Survey: IMAS 08.10,” March 2013. International Mine Action Standard.
  2. Birk, Ronald J., and Thomas B. McCord. “Airborne Hyperspectral Sensor Systems.” IEEE AES Systems Magazine (October 1994): 26–33.
  3. European Commission. “ARC-Airborne Minefield Area Reduction: FP5-IST Project, 2001–2003.” Community Research and Development Information. Accessed 20 February 2017. http://bit.ly/2lnDuBp.
  4. TIRAMISU. “Toolbox Implementation for Removal of Anti-personnel Mines, Submunitions and UXO” European Community’s Seventh Framework Programme. Accessed 20 February 2017. http://bit.ly/2kRv9nT.
  5. Bajić, M., H. Gold, Z. Pracic, and D. Vuletic. “Airborne Sampling of the Reflectivity by the Hyper Spectral Line Scanner in A Visible and Near Infrared Bands.” Proceedings of the 24th EARSeL Symposium, New Strategies for European Remote Sensing, Dubrovnik, Croatia, 25–27 May 2004, Rotterdam: Millpress, (2005): 703–710.
  6. Semanjski, S., and D. Gajski. “Integration of Position and Orientation System and Hyperspectral Line Scanner.” Proceedings of Disaster Management and Emergency Response in the Mediterranean Region, Oluic (ed.), EARSeL Conference, Zadar, Croatia, (September 2008): 377–384.
  7. Semanjski, S., D. Gajski, and M. Bajić. “Transformation of the Hyperspectral Line Scanner into A Strip Imaging System.” Proceedings of Disaster Management and Emergency Response in the Mediterranean Region, Oluic (ed.), First EARSeL Conference, Zadar, Croatia, (September 2008): 369–375.
  8. Bajić, Milan, Tamara Ivelja, Andrija Krtalić, Mile Tomić, and Dejan Vuletić. “The Multisensor and Hyper Spectral Survey of the UXO Around the Exploded Ammunition Depot, of the Land Mines Test Site Vegetation.” Humanitarian Demining 2013 International Symposium, Šibenik, Croatia, 23–25 April 2015, Book of papers, 91–96.
  9. Bajić, Milan, Marko Krajnović, Anna Brook, and Tamara Ivelja. “Ground Vehicle Based System for Hyper Spectral Measurement of Minefields.” The 11th International Symposium ˝Humanitarian Demining 2014”, Book of Papers, pp. 11–14.
  10. Ivelja, Tamera, Milan Bajić, and Goran Skelac. “UAV Deployment in Survey with Hyperspectral Line Scanner.” 12th Mine Action International Symposium 2015, Croatia, 27–30 April 2015. Book of Papers, pp. 37–42.
  11. Ančić, Mario, Renata Pernar, Milan Bajić, Ante Seletković, and Jelena Kolić. “Detecting Mistletoe Infestation on Silver Fir Using Hyperspectral Images.” Journal Forest Biogeosciences and Forestry 7 (2014): 85–91. Accessed 20 February 2017. http://bit.ly/2lzOuxF.
  12. Bajić, Milan. “Airborne Hyperspectral Surveillance of the Ship-based Oil Pollution in Croatian Part of the Adriatic Sea.” Geodetski List 66 no.89 (June 2012) 77–100. Accessed 20 February 2017. http://bit.ly/2leLUws
  13. Bajić, Milan, and Tamara Ivelja. “New Technology for Mine Action: the Hyperspectral Non-Technical Survey from UAV and Helicopter.” The Humanitarian Demining International Symposium Biograd, Croatia, 2015. Book of papers pp.179–183. 
  14. CRC Press. Hyperspectral Remote Sensing of Vegetation. Edited by Prasad S. Thenkabail, John G. Lyon, and Alfredo Huete. Boca Raton: Taylor & Francis Group, 2011. 
  15. Ardouin, Jean-Pierre, Josee Levesque, Vincent Roy, Yves Van Chestein, and Anthony Faust. “Demonstration of Hyperspectral Image Exploitation for Military Applications” in Remote Sensing Applications, edited by Dr. Boris Escalanto, 493–516. InTech, 2012. Accessed 20 February 2017. http://bit.ly/2lzy4W8.
  16. Jones, K.F., D.K. Perovich, and G.G. Koeing. “Spatial and Temporal Variability of Hyperspectral Signatures of Terrain.” Proc. SPIE 6966, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XIV (April 2008). Accessed 20 February 2017. http://bit.ly/2kF55Bd.
  17. Subhasis and Ketan Kotwal. Hyperspectral Image Fusion. New York: Springer, 2013.
  18. Matteoli, S., M. Diani, and G. Corsini. “A Tutorial Overview of Anomaly Detection in Hyperspectral Images.” IEEE Aerospace and Electronic Systems Magazine 25, no.7 (2010): 5–28. Accessed 20 February 2017. http://bit.ly/2lf0Af4.
  19. McFee, John E. and Herb T. Ripley. “Detection of Buried Land Mines Using A CASI Hyperspectral Imager.” Proc. SPIE 3079, Detection and Remediation Technologies for Mines and Minelike Targets II 738 (July 22 1997). Accessed 20 February 20117. http://bit.ly/2mfok0x.
  20. McFee, John E., Cliff Anger, Steve Achal, and Tyler Ivanco. “Landmine Detection Using Passive Hyperspectral Imaging.” Proc. SPIE 6554, Chemical and Biological Sensing VIII, 655404 (April 2007).
  21. Avi Buzaglo Yoresh. “Mine Detection” by Air Photography, Proceedings of International Symposium Humanitarian Demining 2010, 27–29 April 2010, Šibenik, Croatia, pp. 40–42.
  22. Zare, Alina, Jeremy Bolton, Paul Gader, and Miranda Schatten. “Vegetation Mapping for Landmine Detection Using Long-Wave Hyperspectral Imagery.” IEEE Transactions on Geoscience and Remote Sensing 46, no.1 (January 2008): 172–78.
  23. Foley J., and C. Patterson. “Demonstration of Airborne Wide Area Assessment Technologies at Pueblo Precision Bombing Ranges, Colorado.” – Hyperspectral Imaging, Final Report, Project No. 200416, Prepared by Sky Research, Inc. 445 Dead Indian Memorial Road Ashland, OR 97520, September 27, 2007, Final v. 2.0. 
  24. Yvinec, Y., Y. Baudoin, G. DeCubber, M. Armada, L. Marques, J.M. Desaulniers, and M. Bajić. “TIRAMISU : FP7-Project for An Integrated Toolbox in Humanitarian Demining.” (2015). Accessed 20 February 2017. http://bit.ly/2kRNVv9.
  25. “Geomine Demonstration Test 2012/2013 Observer Report.” Geneva International Center Humanitarian Demining. (January 2014). Accessed 20 February 2017. http://bit.ly/2leXWpA.
  26. Slonecker, E.T., and G.B. Fisher. “An Evaluation of Remote Sensing Technologies For the Detection of Residual Contamination.” U.S. Geological Survey. (2014). Accessed 20 February 2017. http://bit.ly/2lnThjS.
  27. BuzagloYoresh, Avi. “Identification of Mine Fields.” Aerial Photography. UNMAS/GICHD Mine Action Technology Workshop, Geneva, 2010.
  28. Abraham, Sathya Achia. “Using Plants to Detect Buried Explosives.” VCU News, 26 March 2013. http://bit.ly/2m1giMy.
  29. Smith, Randall B., and Merri P. Skrdla. “Feature Mapping with TNTmaps: Tutorial.” Micro Images. (2014). Accessed 20 February 2017. http://bit.ly/2kFatUE.
  30. Shea, Patrick. “Using Plants to Detect Landmines.” The Journal of ERW and Mine Action 18, no.3, (December 2014).
  31. Oskin, Becky. “Leafy Bloodhounds: Plants Might Find Land Mines.” Live Science, 14 August 2014. Accessed 20 February 2017. http://bit.ly/2l0Wfd9.
  32. Manley II, Paul V. “Plant Functional Trait and Hyperspectral Reflectance Responses to Comp. B Exposure: Efficacy of Plants as Landmine Detectors.” Master’s Thesis, Virginia Commonwealth University, 2015. 
  33. Azaria, I., N. Goldshleger, E. Ben-Dor, and R. Bar-Hamburger. “Detection of Cannabis Plants by Hyper-Spectral Remote Sensing Means.” Geography Department, Remote Sensing Laboratory, Tel Aviv University, 2010. Accessed 20 February 2017. http://bit.ly/2kFdeFL.
  34. Yimmy, Alexander, Gomez Quevedo, and Milan Bajić. “The Specific Colombian Problem - the Antipersonnel Landmines and Improvised Explosive Devices in the Fields of the Illicit Crops.” International Symposium “Humanitarian Demining 2011” 26 to 28 April 2011, Šibenik, Croatia, Book of papers, pp.61–64.
  35. Bajić, Milan, Tamara Ivelja, Emina Hadžić, Haris Balta, Goran Skelac, and Zoran Grujić. “Impact of Flooding on Mine Action in Bosnia and Herzegovina, Croatia and Serbia.” The Journal of ERW and Mine Action 19, no.1 (April 2015): 43–49.
  36. Bajić, Milan. “The Advanced Intelligence Decision Support System for the Assessment of Mine-Suspected Areas.” The Journal of ERW and Mine Action 14, no.3, (Fall 2010): 69–75. 
  37. Šestak, Ivana. “Use of Field Spectroscopy for Assessment of Nitrogen Use Efficiency in Winter Wheat.” Doctoral thesis, University of Zagreb, 2011.
  38. Brook, Anna and Eyal Ben Dor. “Supervised Vicarious Calibration (SVC) of Hyperspectral Remote-Sensing Data.” Remote Sensing of Environment 115 (2011): 1543–1555.
  39. Borengasser, Marcus, William S. Hungate, and Russel Watkins. “Hyperspectral Remote Sensing: Principles and Applications.” Boca Raton: Taylor & Francis Group, 2008.
  40. Chang, Chein-I. “Hyperspectral Data Processing: Algorithm Design and Analysis.” Hoboken: JohnWiley & Sons, Inc., 2013.
  41. Harney, Robert C. “Combat Systems: Volume 1. Sensor Elements, Part I. Sensor Functional Characteristics.” Naval Postgraduate School, Monterey, CA, (2004): 427–432.
  42. Gitelson, A. A., Y. Zur, O. B. Chivkunova, and M. N. Merzlyak. “Assessing Carotenoid Content in Plant Leaves with Reflectance Spectroscopy.” Photochemistry and Photobiology 75, no.3 (2002): 272–281.
  43. Smit, Rene, Peter Schmitz, Niell du Plooy, and Antony Cooper. “The Influence of Explosives on Plants Using In–Situ Hyperspectral Data.” 5th UNMAS/GICHD Bi-Annual Technology Workshop, Pretoria, South Africa, 18–20 June 2014.