Periodic Reporting for period 3 - FOLDOUT (Through-foliage detection, including in the outermost regions of the EU)
Reporting period: 2021-07-01 to 2022-08-31
The FOLDOUT platform is meant to assist border guards by providing prompt detection of illegal activity at borders and trace the movement and routes prior to arrival in border areas. FOLDOUT builds a system that combines various sensors and technologies and intelligently fuses these into an effective and robust intelligent detection platform. FOLDOUT makes the tasks of Border Guards simpler and faster by combining events from various sensors to give a complete situation threat assessment combined with suggested reaction scenarios. With multiple pilots in Bulgaria over the course of 2 years’ and demonstrations in Greece, Finland and French Guiana,FOLDOUT provided fundamental enhancements in the domain of border surveillance and improved search & rescue scenarios.
The technical concept of FOLDOUT is based on the combination of several sensor technologies on the ground and on special, high rising platforms with data fusion algorithms into a single, seamlessly integrated system. This concept allows for the real-time detection of critical events (e.g. illegal cross border activities, lost persons) even under dense foliage. The alarms are presented to border guard operators in a common operational picture in a unified data presentation for all sensors. To penetrate the foliage to a certain extent under day and night conditions, long range, multi-spectral, LIDAR and RADAR sensors are used. For the surveillance of large border areas stratospheric platforms are employed to yield unobstructed field-of-view and unprecedented detection range for the sensors they carry. Ground EM (radio transmitter detection), acoustic and seismic (movement) detectors deliver complementary data where the vegetation is too dense to penetrate. The activities at the fringe of the foliage, such as suspicious car traffic, are monitored by conventional ground-based cameras and EOS sensors completed by satellite SAR data. Unmanned vehicles could be employed on demand to scan areas where sensor coverage is too low or data received are ambiguous. All the different sensors are integrated in a common system and data model to provide information on different aspects and in a different light spectrum or modality or even on a different time resolution or scale.
Data processing algorithms based on machine learning are used to fuse and reliably interpret all data to derive alarms on the presence of persons or critical situations in the surveilled area. Reasoning methods are used to filter unusual from usual behavior in the surveillance area. To reduce operator workload and improve situational awareness sensor information and alarms are uniformly presented to the operator on a command and control software tool. The tool provides a map-based graphic user interface with a standardised symbology to observe, track and react with maximum efficiency.
The technical concept of FOLDOUT is based on the combination of several sensor technologies on the ground and on special, high rising platforms with data fusion algorithms into a single, seamlessly integrated system. This concept allows for the real-time detection of critical events (e.g. illegal cross border activities, lost persons) even under dense foliage. The alarms are presented to border guard operators in a common operational picture in a unified data presentation for all sensors. To penetrate the foliage to a certain extent under day and night conditions, long range, multi-spectral, LIDAR and RADAR sensors are used. For the surveillance of large border areas stratospheric platforms are employed to yield unobstructed field-of-view and unprecedented detection range for the sensors they carry. Ground EM (radio transmitter detection), acoustic and seismic (movement) detectors deliver complementary data where the vegetation is too dense to penetrate. The activities at the fringe of the foliage, such as suspicious car traffic, are monitored by conventional ground-based cameras and EOS sensors completed by satellite SAR data. Unmanned vehicles could be employed on demand to scan areas where sensor coverage is too low or data received are ambiguous. All the different sensors are integrated in a common system and data model to provide information on different aspects and in a different light spectrum or modality or even on a different time resolution or scale.
Data processing algorithms based on machine learning are used to fuse and reliably interpret all data to derive alarms on the presence of persons or critical situations in the surveilled area. Reasoning methods are used to filter unusual from usual behavior in the surveillance area. To reduce operator workload and improve situational awareness sensor information and alarms are uniformly presented to the operator on a command and control software tool. The tool provides a map-based graphic user interface with a standardised symbology to observe, track and react with maximum efficiency.
Main project's achievements can be summarized as follows:
- All milestones planned for this period have been successfully achieved: MS8, MS9, MS10.
- The FOLDOUT system has been tested and evaluated in all demonstration sites (Bulgaria, Greece, Finland, French Guiana) under the most diverse environmental conditions. Results of the KPI evaluation are reported in each of the test reports D2.3 to D2.6.
- The SAR demonstrator airborne campaign was successfully accomplished and took place in November 2021 at the Bulgarian test site.
- Following sensors and detection technologies are part of the FOLDOUT system: AIT Thermal camera, BHE EO camera, AIT smart camera platform, ITTI acoustic-seismic sensors, ONERA terrestrial LiDAR, ONERA Multispectral IR camera, ONERA LiDAR on UAV, BHE RF detector, VTT active hyperspectral sensor, VTT radar fence. Data collected by the airborne sensors are processed offline and integrated into the FOLDOUT framework.
- Novel algorithms based on machine learning techniques were explored to provide automatic event detection.
- Test studies and preparation of satellite and stratosphere missions are finalised (see also D8.1-D8.4 and D8,8-D8.10).
- Five software components are fully developed and tested to improve situational awareness through fusion: Fusion and Reasoning Platform, Registration Module, Fusion Module, Local Dynamic Maps and Situational Awareness.
- All data information is standardised into a Common Alerting Protocol (CAP)-based data model and documented in the FOLDOUT CAP document (Annex 1 of D9.1).
- A Mobile Command Centre has been developed as a base station for deploying the FOLDOUT System in remote areas.
- A GIS-based Command and Control platform has been developed that provides a map-based and layered approach, to ensure that all different pieces of information are all placed in one single interface, for advancing the operators’ experience and efficiency when dealing with a complex situation.
- A public annotated dataset for visual tracking and surveillance has been made available to the scientific community and the analytics challenged launched at the PETS workshop 2021, organized by FOLDOUT within the AVSS 2021 conference.
- The development process and demonstrations have been supported by the L-E-P partners providing legal and ethical advice and monitoring, including the implementation of the informed consent protocol. L-E-P tasks have also included developing the methodology for the study and assessment on as reported in D3.4.
In summary, FOLDOUT has achieved innovation and promising new concepts that are detailed in the relevant evaluation reports.
- All milestones planned for this period have been successfully achieved: MS8, MS9, MS10.
- The FOLDOUT system has been tested and evaluated in all demonstration sites (Bulgaria, Greece, Finland, French Guiana) under the most diverse environmental conditions. Results of the KPI evaluation are reported in each of the test reports D2.3 to D2.6.
- The SAR demonstrator airborne campaign was successfully accomplished and took place in November 2021 at the Bulgarian test site.
- Following sensors and detection technologies are part of the FOLDOUT system: AIT Thermal camera, BHE EO camera, AIT smart camera platform, ITTI acoustic-seismic sensors, ONERA terrestrial LiDAR, ONERA Multispectral IR camera, ONERA LiDAR on UAV, BHE RF detector, VTT active hyperspectral sensor, VTT radar fence. Data collected by the airborne sensors are processed offline and integrated into the FOLDOUT framework.
- Novel algorithms based on machine learning techniques were explored to provide automatic event detection.
- Test studies and preparation of satellite and stratosphere missions are finalised (see also D8.1-D8.4 and D8,8-D8.10).
- Five software components are fully developed and tested to improve situational awareness through fusion: Fusion and Reasoning Platform, Registration Module, Fusion Module, Local Dynamic Maps and Situational Awareness.
- All data information is standardised into a Common Alerting Protocol (CAP)-based data model and documented in the FOLDOUT CAP document (Annex 1 of D9.1).
- A Mobile Command Centre has been developed as a base station for deploying the FOLDOUT System in remote areas.
- A GIS-based Command and Control platform has been developed that provides a map-based and layered approach, to ensure that all different pieces of information are all placed in one single interface, for advancing the operators’ experience and efficiency when dealing with a complex situation.
- A public annotated dataset for visual tracking and surveillance has been made available to the scientific community and the analytics challenged launched at the PETS workshop 2021, organized by FOLDOUT within the AVSS 2021 conference.
- The development process and demonstrations have been supported by the L-E-P partners providing legal and ethical advice and monitoring, including the implementation of the informed consent protocol. L-E-P tasks have also included developing the methodology for the study and assessment on as reported in D3.4.
In summary, FOLDOUT has achieved innovation and promising new concepts that are detailed in the relevant evaluation reports.
The main result of FOLDOUT is to develop, test and demonstrate a system and solution to detect and locate people and vehicles operating under the coverage of trees and other foliage over large areas.
FOLDOUT:
• Developed requirements closely together with the end users, e.g. border guards;
• Improved sensor technologies with innovative approaches specifically adapted to through foliage detection scenarios,functioning at day as well as night time and in harsh environment;
• Improved situational awareness through fusion of advanced aerial and space-based sensor platforms with ground based sensors into one surveillance solution;
• Demonstrated the effectiveness of the FOLDOUT concept in realistic operational scenarios;
• Provided a planning tool for decision makers to configure a surveillance system for the specific requirements of a target deployment area;
• Created a scientific/industrial development community and putting at their disposal a set of reference data, which will be used to tune and assess the analytics performance.
FOLDOUT:
• Developed requirements closely together with the end users, e.g. border guards;
• Improved sensor technologies with innovative approaches specifically adapted to through foliage detection scenarios,functioning at day as well as night time and in harsh environment;
• Improved situational awareness through fusion of advanced aerial and space-based sensor platforms with ground based sensors into one surveillance solution;
• Demonstrated the effectiveness of the FOLDOUT concept in realistic operational scenarios;
• Provided a planning tool for decision makers to configure a surveillance system for the specific requirements of a target deployment area;
• Created a scientific/industrial development community and putting at their disposal a set of reference data, which will be used to tune and assess the analytics performance.