Cognitive Airborne multisEnsor System for Aerial Reconnaissance Intelligence and Surveillance

Projektinformationen

CAESARIS

ID Finanzhilfevereinbarung: 605093

Abgeschlossenes Projekt

Startdatum 1 Oktober 2013

Enddatum 30 September 2015

Finanziert unter

• Specific Programme "Capacities": Research for the benefit of SMEs

Gesamtkosten € 1 499 672,00

EU-Beitrag € 1 088 000,00

Koordiniert durch

CENTUM SOLUTIONS SL
Spain

Final Report Summary - CAESARIS (Cognitive Airborne multisEnsor System for Aerial Reconnaissance Intelligence and Surveillance)

Executive Summary:
The purpose of CAESARIS project was to develop a new, integrated security and search-and-rescue airborne solution conceived to detect people, both above visible and hidden behind opaque layers (like foliage, trailer covers, boat covers) or in reduced visibility conditions or darkness.

Considering market opportunities that arise from the European needs in the field of security and search and rescue, CAESARIS team intended to develop a real-time configurable, plug-and-play and SWaP system that would have fulfilled the requirements of the EU Security Agencies. Furthermore, CAESARIS was designed to detect and track the targets in line of sight, as well as under opaque layers of materials (such as ground, foliage, inside trucks, etc.), covering wide land and sea areas, 24/7, including bad weather conditions.

The REA cancelled the project after the first reporting period because it considered that operational scenarios would not be fully addressed by the system. The REA evaluated the partial results and decided that the expected performance would not be enough for the intended use.

The consortium proposed a new operational scenario for the application of CAESARIS: Maritime humanitarian relief, which was more suitable for the current performance of the system. This new approach was based on already developed components that would be improved with additional features, such as IR processing, vessel detection and recognition and net-centric approach. However, this alternative scenario was not accepted.

Despite the premature termination of the project, as a result of the work performed within the project, the SMEs obtained partial results which may be useful for their business.

The SMEs obtained new algorithms for automatic image analysis in the visual (EO) domain and, partially, in the radar (SAR) and new algorithms for data fusion.

Project Context and Objectives:
European societies face serious security challenges, challenges that are growing in scale and sophistication. Most of these issues are cross-border and cross-sectorial in nature and no single EU State is able to respond to them on its own. In order to strengthen security through border management, the Commission Directorate General of Home Affairs has established the enhanced use of new technology for border checks and surveillance as a strategic matter. CAESARIS project is aligned with the Commission Security Strategy, since the development of high-accuracy technology for detection of illegal border crossing is one of the main goals of the project. Additionally, the project provides search and rescue applications, such as victim detection, natural disaster area inspection, maritime missions, etc.
CAESARIS solution is conceived to operate on board of an aerial vehicle, making exhaustive and intelligent use of state-of-the-art sensors: Synthetic Aperture Radar (SAR) and Electro-Optical (EO) imagery in the visible range, in an innovative way based on cognitive multispectral sensing.
CAESARIS team is developing a real-time configurable, plug-and-play and low Size, Weight and Power (SWaP) system adjusted to the requirements of the EU Security Agencies. Moreover, the system is designed to detect and track not only visible targets, but also those under poor lighting conditions (night, shade), bad weather, or under opaque layers of materials (foliage, truck cabins, containers, etc.). Land and sea areas might be covered, 24/7.
To achieve CAESARIS concept, five specific industrial objectives have been set:
• Reconfigurable SAR. Development of a Synthetic Aperture Radar (SAR) hardware/software system that can be reconfigured while in operation; so it will be able to fulfil all application requirements, from high-resolution low-penetration to high-penetration signals.
• Multisensor people detection and tracking algorithms. Development of algorithms and software for target detection and tracking, using SAR and Electro-Optical (EO) colour images. There is a special focus on human targets, and particularly on people hidden under non-transparent layers.
• Software-defined cognitive multispectral sensing platform. Development of a software system for the automatic control of the multispectral sensing hardware in a context-aware manner. The system will be able to analyse and fuse information from EO colour imaging and SAR imaging, in order to take automatic decisions on the control of the reconfigurable SAR system that optimize the hidden target detection process.
• Control and Visualization Module. Development of a software module for the control of the software-defined cognitive multispectral sensing platform and its communication with a ground based management system. It will receive and manage control signals, and send sensor images and target detection metadata back to the ground system.
• DEMO platform. Development of a pilot airborne platform that integrates all hardware and software modules, allowing the test of the platform in real conditions.
CAESARIS consortium comprises six entities: three SMEs and three RTD performers. Centum Solutions from Spain, MetaSensing from Netherlands, and SmallGIS from Poland, are SMEs devoted to a complementary range of disciplines needed to develop CAESARIS concept. Gradiant from Spain, Air Force Institute of Technology of Poland (ITWL), and the Italian National Consortium for Telecommunications (CNIT) are RTD performers that hold key capabilities to successfully meet the project objectives and results.

Project Results:
Although the project has not been completed, some of the partial S&T results obtained are relevant and have a commercial value:

> Result #1: Real time land cover segmentation method for EO aerial video
Project Result: 2 – EOSRM; Deliverable(s): D4.2 D4.4; Tasks: T1.2 T1.4 T2.1 T2.2 T2.5 T4.2.
This methodology allows for terrain segmentation and classification according to land cover type from EO aerial video alone, without the use of special EO or NIR multispectral images, an in real time, with a performance that improves the state of the art. Real time operation is a unique feature of this solution, which provides a useful tool for extraction of highly valuable contextual information to aid the process of online threat analysis in an aerial situational awareness system. The method does not require special processing hardware, such as FPGA, GPU or DSP.

> Result #2: Real time pedestrian detection method for EO aerial video
Project Result: 2 – EOSRM; Deliverable(s): D4.2 D4.4; Tasks: T1.2 T1.4 T2.1 T2.2 T2.5 T4.2.
This methodology allows for detection of people from EO aerial video alone, achieving a detection performance that is similar to other state-of-the-art approaches requiring the use of dual EO/thermal video analysis. Besides, the method operates in real time without the use of special processing hardware, such as FPGA, GPU or DSP.

> Result #3: Real time target tracking method for EO aerial video
Project Result: 1 – EOSRM; Deliverable(s): D4.2 D4.4; Tasks: T1.2 T1.4 T2.1 T2.2 T2.5 T4.2.
This methodology allows for tracking of a target locked by the user in an EO aerial video sequence, and is robust to camera motion, target motion, and low resolution of the target. It can be used as input for control of the pan, tilt and zoom configuration of the camera, so as to perform an active PTZ tracking of the targets.

> Result #4: Target data fusion algorithm
Project Result: 3 – CSM; Deliverable(s): D4.5; Tasks: T1.2 T1.4 T2.2 T2.5 T4.3.
This algorithm permits obtain more valuable target data from sensors data taking advantage of the different nature of these. More concretely, it permits aggregation of data that is related to the same target and fuse data of the same type present on several samples of the target.

> Result #5: Region of interest data fusion algorithm
Project Result: 3 – CSM; Deliverable(s): D4.5; Tasks: T1.2 T1.4 T2.2 T2.5 T4.3.
This algorithm permits obtain more valuable region of interest data from sensors data taking advantage of the different nature of these. More concretely, it permits aggregation of data that is related to the same region of interest and fuse data of the same type present on several samples of the target.

> Result #6: MTL template for the transformation from SysML to IDL
Project Result: 1 – CMSP; Deliverable(s): D4.6; Tasks: T1.2 T1.4 T2.2 T2.5 T4.3.
Model to Text Language (MTL) document that permits to transform from Eclipse Foundation's PolarSys system modules definition to IDL files. This IDL files define the different interfaces of the model ready to generate software that implements the boilerplate of software modules that communicate using DDS middleware.

> Result #7: Virtual machine with GIS system
Project Result: 6 – GSCVM; Deliverable(s): D6.2; Tasks: T1.3 T1.4 T2.4 T6.2.
OSGeo Live based virtual machine that contains a QGIS base GIS system that leverages a web interface for real time visualization of targets and ROIs.

> Result #8: Land cover segmentation algorithm from SAR images.
Project Result: 5 – SARSRM ; Deliverable(s): D4.3; Tasks: T1.1 T1.4 T2.3 T2.5 T4.1.1 T.4.1.2
An algorithm to segment different areas of ground clutter has been developed and implemented. A preliminary test has been conducted on real data provided by MS. Theory and methodology are reported in deliverable D.4.1.

> Result #9 : Detection algorithm of static and moving targets from SAR data.
Project Result: 4 – Airborne SAR/ISAR target detection module; Deliverable(s): D4.3; Tasks: T1.1 T1.4 T2.3 T2.5 T3.2 T.4.1.3
An algorithm has been proposed for target detection that can be applied before and after STAP processing to detect stationary target and moving targets, respectively. Theory and methodology is reported in Deliverable D.4.3.

> Result #10: SAR reconfiguration algorithm.
Project Result: 4 – SARSRM; Deliverable(s): D4.3; Tasks: T1.1 T1.4 T2.3 T2.5 T3.1 T.4.1.4
A reconfigurable SAR has been designed based on the project objectives. Specifically, the method to reconfigure online the SAR system has been designed and theoretically formulated. The method is based on a knowledge-based architecture. Main results have been published on scientific papers and reported in deliverables D.3.1 and D.4.3. A preliminary implemented version of the algorithm has been accomplished based on the available dataset.

> Result #11: People detection through microDoppler Analysis.
Project Result: 4 – Airborne SAR/ISAR target detection module; Deliverable(s): D4.3; Tasks: T1.1 T1.4 T2.3 T2.5 T3.2 T.4.1.3
Two algorithms to detect people from SAR images has been identified. Such algorithms are based on the exploitation of the microMotions of people with respect to rigid body targets, like terrestrial vehicles. Such algorithms have been implemented effectively and tested on real data from a ground based radar, since we do not have yet a suitable dataset to test such algorithms. Such algorithms have proven their effectiveness on such data.

> Result #12: STAP algorithm for the detection of moving targets from SAR images.
Project Result: 4 – Airborne SAR/ISAR target detection module; Deliverable(s): D4.3; Tasks: T1.1 T1.4 T2.3 T2.5 T3.2 T.4.1.3
An algorithm to mitigate ground clutter and then enhance the detection performance of ground moving targets in a SAR scene has been implemented based on CNIT background. Algorithm has been optimize to reach the required time operational requirements.

> Result #13: SAR/EO image registration.
Project Result: 3 – CSM; Deliverable(s): D4.4; Tasks: T1.2 T1.4 T2.1 T2.5 T4.3.
An algorithm for precise alignment of geo-referenced multisensory images has been delivered, which has been proven effective for the case of SAR and EO image registration.

> Result #14: Design of fusion and control software architecture.
Project Result: 3 – CSM; Deliverable(s): D4.5; Tasks: T1.2 T1.4 T2.1 T2.5 T4.3.
A software architecture for multisensor data fusion and online sensor reconfiguration control has been developed based on the JDL standard. The architecture is scalable so that it can seamlessly incorporate additional sensing modules and other information sources in the different levels of fusion process.

> Result #15: CMSP Middleware.
Project Result: 1 – CSMP; Deliverable(s): D4.6; Tasks: T1.2 T1.4 T2.1 T2.5 T4.3.
The middleware for communication between CMSP modules and libraries for message logging definition has been achieved. Furthermore, the "boilerplate" software for all the CMSP modules has been generated. This software implements: all the communication primitives for each interface, the message logging subsystem and the configuration file reading primitives that all the modules in the CMSP need. The Core Framework component of the CMSP definition has been achieved and an initial implementation with the interfaces has been achieved.

In summary, important progress has been made in all expected project results, which have not been 100% completed due to premature termination of project. Overall, it can be said that main image analysis, fusion, control and visualization functionalities have been designed and validated in laboratory conditions. Implementation of all modules was in good progress, while there where some delays in integration of all system, due to project contingencies that were being managed at the time of project suspension. The current status of expected project results is the following:
1. CMSP (80% completed): The system middleware has been has been specified and an initial implementation with interfaces has been achieved. Integration of the ECVM module was pending at the time of project suspension.
2. EOSRM (90% completed): All EO video analysis functionalities have been designed and validated in laboratory tests. C++ implementation is in good progress. Integration in the CMSP had a certain delay at the time of project suspension and validation in demonstration conditions was cancelled.
3. CSM (70% completed): The data fusion and sensor control architecture has been specified. Fusion algorithms for both targets and regions have been also designed. Implementation was in progress at the time of termination of the project.
4. Airborne SAR/ISAR target detection module (100% completed): SAR hardware has been designed and algorithms for sensor reconfiguration and for moving and stationary target from SAR/ISAR images have been designed, tested and implemented in Matlab. Validation in demonstration conditions was cancelled due to termination of the project.
5. SARSRM (100% completed): Algorithms for land cover segmentation and classification in SAR images, as well as human detection in SAR images, have been developed, tested and implemented in Matlab. Validation in demonstration conditions was cancelled due to termination of the project.
6. GSCVM (100% completed): The Ground Station Control and Visualization Module has been developed and tested. Validation in demonstration conditions was cancelled due to project termination.
7. ECVM: (100% completed): The Embedded Control and Visualization Module has been developed and tested. Validation in demonstration conditions was cancelled due to project termination.

Potential Impact:
Although the project has been cancelled so the final system CAESARIS will not be available, participant SMEs obtained useful partial results: new technology that empowers their current products and services and permits, in some cases, to create new ones.

For instance, CENTUM has received new algorithms for people detection and tracking from the air and a new software architecture for fusing data from diverse sources aiming at getting more accurate situational awareness.

These technologies will be the base of a new generation of products created in the framework of the Civil UAVS Initiative, which is a Pre-Commercial Procurement. Galician government has launched the Civil UAVs Initiative for the use of unmanned aerial vehicles for civil purposes and to help improve the provision of public services.

CENTUM is part of the winning proposal submitted by the alliance of INAER and INDRA that was awarded with the public contract. In the framework of this contract, CENTUM will develop new on-board mission systems to support public services in these situations:

• Maritime Search & Rescue operations:
• Fire prevention

In order to cover the operational needs, the new mission system will relay in algorithms developed in the framework of CAESARIS project for fusing data from diverse sensors (e.g. EO/IR, SIGINT). The EO/IR sensor will be able to detect and track people over multiple surfaces (e.g. sea, land) as a result of the previous work performed in CAESARIS.

List of Websites:
http://www.caesaris.eu/en/

Contact details:

Isaac Ballesteros
[e-mail]
T [phone]
M [phone]