Periodic Reporting for period 2 - FOLDOUT (Through-foliage detection, including in the outermost regions of the EU)
Reporting period: 2020-01-01 to 2021-06-30
The FOLDOUT platform will assist border guards by providing prompt detection of illegal activity at borders and trace the movement and routes prior to arrival in border areas. FOLDOUT will build a system that combines various sensors and technologies and intelligently fuses these into an effective and robust intelligent detection platform. FOLDOUT will make the tasks of Border Guards simpler and faster by combining events from various sensors to give a complete situation threat assessment combined with suggested reaction scenarios. With multiple pilots in Bulgaria over the course of 2 years’ and demonstrations in Greece, Finland and French Guiana,FOLDOUT will provide fundamental enhancements in the domain of border surveillance and improved search & rescue scenarios.
The technical concept of FOLDOUT is based on the combination of several sensor technologies on the ground and on special, high rising platforms with data fusion algorithms into a single, seamlessly integrated system. This concept allows for the real-time detection of critical events (e.g. illegal cross border activities, lost persons) even under dense foliage. The alarms are presented to border guard operators in a common operational picture in a unified data presentation for all sensors. To penetrate the foliage to a certain extent under day and night conditions, long range, multi-spectral, LIDAR and RADAR sensors are used. For the surveillance of large border areas stratospheric platforms are employed to yield unobstructed field-of-view and unprecedented detection range for the sensors they carry. Ground EM (radio transmitter detection), acoustic and seismic (movement) detectors will deliver complementary data where the vegetation is too dense to penetrate. The activities at the fringe of the foliage, such as suspicious car traffic, are monitored by conventional ground-based cameras and EOS sensors completed by satellite SAR data. Unmanned vehicles will be employed on demand to scan areas where sensor coverage is too low or data received are ambiguous. All the different sensors are integrated in a common system and data model to provide information on different aspects and in a different light spectrum or modality or even on a different time resolution or scale.
Data processing algorithms based on machine learning will therefore be used to fuse and reliably interpret all data to derive alarms on the presence of persons or critical situations in the surveilled area. Reasoning methods will be used to filter unusual from usual behavior in the surveillance area. To reduce operator workload and improve situational awareness sensor information and alarms are uniformly presented to the operator on a command and control software tool. The tool will provide a map-based graphic user interface with a standardised symbology to observe, track and react with maximum efficiency.
The technical concept of FOLDOUT is based on the combination of several sensor technologies on the ground and on special, high rising platforms with data fusion algorithms into a single, seamlessly integrated system. This concept allows for the real-time detection of critical events (e.g. illegal cross border activities, lost persons) even under dense foliage. The alarms are presented to border guard operators in a common operational picture in a unified data presentation for all sensors. To penetrate the foliage to a certain extent under day and night conditions, long range, multi-spectral, LIDAR and RADAR sensors are used. For the surveillance of large border areas stratospheric platforms are employed to yield unobstructed field-of-view and unprecedented detection range for the sensors they carry. Ground EM (radio transmitter detection), acoustic and seismic (movement) detectors will deliver complementary data where the vegetation is too dense to penetrate. The activities at the fringe of the foliage, such as suspicious car traffic, are monitored by conventional ground-based cameras and EOS sensors completed by satellite SAR data. Unmanned vehicles will be employed on demand to scan areas where sensor coverage is too low or data received are ambiguous. All the different sensors are integrated in a common system and data model to provide information on different aspects and in a different light spectrum or modality or even on a different time resolution or scale.
Data processing algorithms based on machine learning will therefore be used to fuse and reliably interpret all data to derive alarms on the presence of persons or critical situations in the surveilled area. Reasoning methods will be used to filter unusual from usual behavior in the surveillance area. To reduce operator workload and improve situational awareness sensor information and alarms are uniformly presented to the operator on a command and control software tool. The tool will provide a map-based graphic user interface with a standardised symbology to observe, track and react with maximum efficiency.
Main project's achievements can be summarized as follows:
First period
- End-users requirements and scenarios definition: End user requirements have been established by relevant stakeholders (e.g. border guards) and specific scenarios defined for: Bulgaria, Finland, French Guiana, Greece and Lithuania.
- Early demonstrator: sensors to reproduce state-of-the-art technologies available at the borders have been integrated into the Early demonstrator and evaluated at a selected testing site in Bulgaria. An intra-communication framework between sensors, algorithms and visualization modules have also been tested in the Early demonstrator, as well as a multi sensor fusion algorithm as a proof of concept to show how fusion improves detection accuracy and decreases false alarms.
- Baseline dataset: a dataset has been acquired from the sensors integrated in the Early demonstrator.
- Testing sites: the cooperation and exchange between technical partners and end-users have been intense. Several test-sites have been visited already in the first period to better understand current challenges at the borders.
- Ethical principles and legal requirements were established to guide both the project research and its technological development activities.
Second period:
- All milestones planned for this period have been successfully achieved: MS4, MS5, MS6, MS7, MS11
- The FOLDOUT prototype has been deployed and tested at BHE’s premises
- The FOLDOUT final system has been deployed and first tested at BHE’s premises (HU) and then in Stara Zagora (BG)
- The first demonstration of the FOLDOUT system was performed at a selected location close to the Bulgarian-Turkish border.
- Following sensors and detection technologies are part of the FOLDOUT system: AIT Thermal camera, BHE EO camera, AIT smart camera platform, ITTI seismic sensors, ITTI acoustic sensors, ITTI PIR sensors, ONERA terrestrial LiDAR, ONERA LiDAR on UAV, BHE RF detector, VTT hyperspectral sensor.
- Novel algorithms based on machine learning techniques provides automatic event detection
- Preparation of satellite and stratosphere missions are finalised, the SAR demonstrator trial campaign shifted to the next period
- Five software components are fully developed and tested to improve situational awareness through fusion: Fusion and Reasoning Platform, Registration Module, Fusion Module, Local Dynamic Maps and Situational Awareness
- All data information is standardised into a Common Alerting Protocol (CAP)-based data model and documented in a FOLDOUT CAP document
- A Mobile Command Centre has been developed as a base station for deploying the FOLDOUT System in remote areas
- A GIS-based Command and Control platform has been developed that provides a map-based and layered approach, to ensure that all different pieces of information are all placed in one single interface, for advancing the operators’ experience and efficiency when dealing with a complex situation
- Dissemination & Communication: FOLDOUT has already been presented at 17 EU/International security events and dissemination material in the form of presentations (virtual or real) combined with website, posters, flyers, videos produced and distributed.
- The development process and demonstrations have been supported by the L-E-P partners providing legal and ethical advice and monitoring, including the implementation of the informed consent protocol.
In summary, FOLDOUT has achieved innovation and promising new concepts that are detailed in the relevant evaluation reports.
First period
- End-users requirements and scenarios definition: End user requirements have been established by relevant stakeholders (e.g. border guards) and specific scenarios defined for: Bulgaria, Finland, French Guiana, Greece and Lithuania.
- Early demonstrator: sensors to reproduce state-of-the-art technologies available at the borders have been integrated into the Early demonstrator and evaluated at a selected testing site in Bulgaria. An intra-communication framework between sensors, algorithms and visualization modules have also been tested in the Early demonstrator, as well as a multi sensor fusion algorithm as a proof of concept to show how fusion improves detection accuracy and decreases false alarms.
- Baseline dataset: a dataset has been acquired from the sensors integrated in the Early demonstrator.
- Testing sites: the cooperation and exchange between technical partners and end-users have been intense. Several test-sites have been visited already in the first period to better understand current challenges at the borders.
- Ethical principles and legal requirements were established to guide both the project research and its technological development activities.
Second period:
- All milestones planned for this period have been successfully achieved: MS4, MS5, MS6, MS7, MS11
- The FOLDOUT prototype has been deployed and tested at BHE’s premises
- The FOLDOUT final system has been deployed and first tested at BHE’s premises (HU) and then in Stara Zagora (BG)
- The first demonstration of the FOLDOUT system was performed at a selected location close to the Bulgarian-Turkish border.
- Following sensors and detection technologies are part of the FOLDOUT system: AIT Thermal camera, BHE EO camera, AIT smart camera platform, ITTI seismic sensors, ITTI acoustic sensors, ITTI PIR sensors, ONERA terrestrial LiDAR, ONERA LiDAR on UAV, BHE RF detector, VTT hyperspectral sensor.
- Novel algorithms based on machine learning techniques provides automatic event detection
- Preparation of satellite and stratosphere missions are finalised, the SAR demonstrator trial campaign shifted to the next period
- Five software components are fully developed and tested to improve situational awareness through fusion: Fusion and Reasoning Platform, Registration Module, Fusion Module, Local Dynamic Maps and Situational Awareness
- All data information is standardised into a Common Alerting Protocol (CAP)-based data model and documented in a FOLDOUT CAP document
- A Mobile Command Centre has been developed as a base station for deploying the FOLDOUT System in remote areas
- A GIS-based Command and Control platform has been developed that provides a map-based and layered approach, to ensure that all different pieces of information are all placed in one single interface, for advancing the operators’ experience and efficiency when dealing with a complex situation
- Dissemination & Communication: FOLDOUT has already been presented at 17 EU/International security events and dissemination material in the form of presentations (virtual or real) combined with website, posters, flyers, videos produced and distributed.
- The development process and demonstrations have been supported by the L-E-P partners providing legal and ethical advice and monitoring, including the implementation of the informed consent protocol.
In summary, FOLDOUT has achieved innovation and promising new concepts that are detailed in the relevant evaluation reports.
The main result of FOLDOUT is to develop, test and demonstrate a system and solution to detect and locate people and vehicles operating under the coverage of trees and other foliage over large areas.
FOLDOUT will:
• Develop requirements closely together with the end users, e.g. border guards;
• Improve sensor technologies with innovative approaches specifically adapted to through foliage detection scenarios,functioning at day as well as night time and in harsh environment;
• Improve situational awareness through fusion of advanced aerial and space-based sensor platforms with ground based sensors into one surveillance solution;
• Demonstrate the effectiveness of the FOLDOUT concept in realistic operational scenarios;
• Provide a planning tool for decision makers to configure a surveillance system for the specific requirements of a target deployment area;
• Create a scientific/industrial development community and putting at their disposal a set of reference data, which will be used to tune and assess the analytics performance.
FOLDOUT will:
• Develop requirements closely together with the end users, e.g. border guards;
• Improve sensor technologies with innovative approaches specifically adapted to through foliage detection scenarios,functioning at day as well as night time and in harsh environment;
• Improve situational awareness through fusion of advanced aerial and space-based sensor platforms with ground based sensors into one surveillance solution;
• Demonstrate the effectiveness of the FOLDOUT concept in realistic operational scenarios;
• Provide a planning tool for decision makers to configure a surveillance system for the specific requirements of a target deployment area;
• Create a scientific/industrial development community and putting at their disposal a set of reference data, which will be used to tune and assess the analytics performance.