Information within this issue may be outdated. Click here to link to the most recent issue

ITEP/JMU Database of International Experiences: Supporting the Test and Evaluation Community

by Franciska Borry and Dieter Guelle, ITEP

Introduction

A new database has been added to the James Madison University (JMU) Lessons Learned database, with more specific content and aim. The specific content reflects the main tasks that the International Test and Evaluation Program (ITEP) for humanitarian demining has agreed on in its Memorandum of Understanding, namely to evaluate and standardise the process of equipment testing in the humanitarian demining industry. The JMU Mine Action Information Centre (MAIC) maintains the database, whereas the ITEP Secretariat is responsible for its content.

The ITEP/JMU Database of International Experiences in Support of the Test and Evaluation Community (DIETEC) was created in order to summarise test and evaluation (T&E) experiences that are referenced to the original test reports. The most demanding part was to work out the database structure and to discuss the concrete contributions currently included. There are fine dividing lines, when at all, between lessons learned, experiences and test results. A remaining question is whether it is opportune to also include technical questions related to the use of tested equipment (test results) or to strictly focus on the testing process. In this regard, we would like to invite the reader to share his/her opinion with the ITEP Secretariat.

Currently, the database only contains experiences related to the T&E process. It was found that the test reports contain an important amount of information, either general or specific, which could be useful for the T&E community as a whole. In order to make these hidden experiences more widely and easily available to the test community, ITEP decided to join up with the MAIC. Funding was provided by the U.S. Department of Defense.

Aim

The main aim of DIETEC is to collect and publish information derived from experiences in T&E of humanitarian demining equipment. The collected "experiences" are intended to highlight key areas of consideration in the T&E process, as well as specific observations related to the evaluation of humanitarian demining equipment in operational use. Moreover, it attempts to provide a common structural approach for T&E of equipment used in humanitarian demining. The database should provide useful information for a variety of T&E stakeholders, ranging from test engineers and entities involved in large-scale T&E campaigns to the individual user at the field level interested in evaluating his/her specific piece of equipment.

Definitions and Approach

The database, in its current version, provides a list of experiences. The experience may be positive or negative but should meet the following criteria:

  • It should be significant in that it has real or assumed impact on the T&E operation.
  • It should be valid in that it is factually and technically correct.
  • It should be applicable in that it identifies a specific process or decision that reduces or limits the potential for failures and mishaps, or reinforces a positive result.
  • It should be understandable for a skilled person not necessarily intimately familiar with the subject matter (adapted from the U.S. National Aeronautics and Space Administration's definition of lessons learned).

The experience may be generic or equipment-specific and can be submitted by any individual or organisation. However, a core set of experiences is extracted from publicly available resources (T&E reports on humanitarian demining equipment) and from T&E activities undertaken under the umbrella of the International Test and Evaluation Program for Humanitarian Demining Equipment.

The intention is that all experiences entered in the database will be periodically reviewed by an international panel of experts from the ITEP network and other organisations performing activities related to T&E, e.g., the United Nations or the Geneva International Centre for Humanitarian Demining (GICHD). During this process, the relevance of the experiences to T&E standards, technical notes and/or methodologies will be assessed.

An entry could become one of the following four things:

  1. An item added to an ITEP T&E methodology: The experience is seen as an important and/or a new way of carrying out tests. A spin-off of this could be a request for further research to be carried out to back up a new test methodology. This further research could ultimately be executed in the form of a collaborative project under the ITEP umbrella.
  2. An item added to the Lessons Learned/Experiences database: The experience is judged to have a significant impact on the testing method and is preferably backed up by at least two other experiences. An existing test methodology may be changed or updated to take into account this experience. An entry can also fall into this category if the experience reports on an unexpected event during testing and outlines a strategy for dealing with it.
  3. An item circulated to the User Community: The experience has an impact on the operational use of demining equipment/systems. It should be passed on to the User Community, for instance through a Technical Note for Mine Action (TNMA) issued by the GICHD.
  4. Dismissed: The experience does not fall into one of the above categories. It may be dismissed with reasons stated.

In its initial stage, the database will mainly include "experiences." After review, the database will be expanded with "lessons learned," referenced to a set of experiences. These lessons learned can then further be incorporated into standards and similar documents whenever relevant.

Approach

Generic Considerations to T&E

Planning

Preparation

Execution

Reporting and Control

Management

Location

Timing

Timing

Objectives/aims

Personnel

Personnel

Format

Location

Instruction

Safety

Content

Personnel

Equipment to be tested

Equipment to be tested

Access

Timing

Supporting equipment

Supporting equipment

Follow-up/-on

Financial resources

Logistics

Unexpected events

 

Coordination

 

Management

 

Supporting information

 

Data collection

 

Supporting equipment

 

 

 

External factors

 

 

 

 
Detection Manual
Tools
Mechanical
Demining
Equipment
Personal Protective Equipment Survey Neutralisation
General General General General General General
Metal Prodders Flail systems Head Marking Low order/thermal
GPR Excavators Tiller systems Upper-body Spatial
information
Low order/chemical
Infrared Other Sifter systems Extremities Positioning High order/explosive
foam
Trace Explosives   Combined systems      
Multi-sensor   Multi-tool systems   Integrated methods
Vehicle based specifics   Mine-protected vehicles   Mechanical Demining Equipment/Mine Detection Dogs
Other   Other   Other
Figure 1: Structure of the DIETEC database.

The structure of the database is illustrated in Figure 1 above. Category and sub-category fields have been selected in order to provide the user with a structured overview of the main stages and factors that should be considered during the T&E of humanitarian demining equipment. The structure may be used as a guide for users when drawing up their own test plan. The utilized categories expand on the structure given in the International Mine Action Standards (IMAS) 03.40 on Test and Evaluation of Mine Action Equipment (first edition, 01.01.2003) and include the main technical categories utilized by ITEP and the GICHD Mechanical Demining Equipment Catalogue.

Implementation

The database consists of two main categories: Generic Considerations and Equipment-Specific Considerations. Each main category has subcategories to explore the experiences in detail and to guide the user to add relevant experiences. Each subcategory can be browsed by clicking on the corresponding individual cell or by executing a detailed search. The user can add an experience to a subcategory by clicking on the corresponding individual cell or on "Add Experience." Table 1 includes two examples extracted from the database in order to illustrate the type of information provided by DIETEC.

Category Generic Considerations to T&E > Planning > Supporting information
Subject Test site layout and facilities
Experience/Advice Detailed prior knowledge of the test site layout and facilities will allow for more efficient data collection and archiving schemes to be designed. For instance, if a large amount of data are planned to be collected in order to evaluate the performance of detection equipment under development, knowledge of the test site layout could allow for automating part of the data collection process and/or use of purpose built data collection platforms. Also, knowledge of internet connection characteristics might influence the data archiving provisions.
Reference I.M. Dibsdall, S.M. Bowen, D.J. Allsopp, Portable Humanitarian Mine Detector 2003 US Trials, 2003
Reference Link http://www.itep.ws/pdf/PHMD_2003_US_trialsreport.pdf
Posted by ITEP Secretariat (ITEP) on 2/14/2004
 
Category Equipment Specific Considerations to T&E > Detection > Metal
Subject Test design, environmental data records
Experience/Advice The following environmental characteristics of the test area should be recorded, in order to more efficiently compare test results: magnetic ground properties, scrap metal content density and distribution, soil composition and texture, distribution of stones/rocks, soil moisture content, vegetation type, density and development, meteorological conditions. Furthermore, details on how the environmental characteristics were measured also need to be registered.
Reference D. Guelle, A. Smith, A. Lewis, T. Bloodworth, Metal Detector Handbook for Humanitarian Demining, EC, 2003
Reference Link http://www.itep.ws/pdf/metal_detector_handbook.pdf
Posted by ITEP Secretariat (ITEP) on 2/14/2004
Table 1: Two examples of testing and evaluation experiences extracted from the DIETEC.

As mentioned in the introduction, during the analysis of the reports, valuable information was also encountered that is related to technical and operational deployment of the equipment. For instance, the report The Severe Duty Vegetation Shredder Technical Testing of Capability by the U.S. Night Vision and Electronic Sensors Directorate (NVESD, online at http://www.humanitarian-demining.org/demining/pubs/clearance/svc_test_report.asp) mentions the fact that fence wire tangled up in the rotating machinery and inflicted considerable damage/disturbances during the testing. This finding could also have implications for the operational use of this type of equipment, not only in fenced minefields but also in dense vegetation where liana-like plants can have the same effect. Hours may be needed for the machinery to become operational again. Important feedback from the reader could be an indication that similar information would be worthy of being included in DIETEC.

Final Remarks

Care should be taken when using the term "lessons learned." In general, the process followed for compiling a "lesson learned" is quite complex and consists of an information-gathering and -processing chain spread over a considerable time period. For instance, the Swedish EOD and Demining Centre (SWEDEC) lessons learned project includes several phases such as the collection of an "experience report" using a standard form, which is then analysed and commented on in an "extended experience report." This stage is followed by validation of the information (i.e., Did it happen several times? Is it useful for another organisation? etc.), which leads to the implementation phase and the compilation of the "lessons learned report." A similar process is being applied by the Department of Energy (DOE) Corporate Lessons Learned Collection database, for example. However, this is not the approach followed at present by the JMU MAIC lessons learned database and is, in our opinion, at this moment in time not necessary, mainly due to the given structure of the database.

Both the JMU MAIC database and DIETEC are currently being evaluated by an international team of "experts." No final results are available yet, but preliminary results of the assessment indicate that an important percentage of the DIETEC experiences have been classified as relevant to the T&E process.

The ITEP would like to get your feedback and opinions when you are visiting the database via http://maic.jmu.edu/iteplessons/ or http://www.itep.ws/.

Contact Information

ITEP Secretariat
JRC/EC T.P. 723
IT-21020 Ispra (VA)
Italy

Franciska Borry
E-mail: Franciska.borry@jrc.it
Tel: +39 0332 789138

Dieter Guelle
E-mail: Dieter.Guelle@jrc.it
Tel: +39 0332 785576