Final Report Summary - NEREIDS (New Service Capabilities for Integrated and Advanced Maritime Surveillance)
Executive Summary:
The NEREIDS project has ended at 31-05-2014 after three years of intense and fruitful work. The implication of all the partners has permitted an optimum collaboration that produced excellent results. Some of them provide a clear break-through with respect to the current state-of-the-art, which applicability into real-life operational procedures has proven to be feasible. Examples are new algorithms capable to detect small targets (length < 5-10 m) and/or to categorize ships by processing Earth Observation (EO) images, new multi-source data fusion approaches, the definition of complex anomalies or the propagation of consolidated tracks into the future.
At the end of project lifetime and according to EC feedback, NEREIDS has achieved all the proposed objectives. In some specific tasks, performance is beyond expectations, especially in what regards:
- the amount of covered topics either technological (new orbit and/or missions concepts) or technical (new processing algorithms)
- the amount of demonstration campaigns with an active user enrolment and with design constraints close to the operational case
- the dissemination effort with a joint collaboration with other projects
System interoperability is another remarkable feature as the NEREIDS system can transparently and automatically work with other operational systems, such as the IMDATE managed by EMSA. In addition, NEREIDS has adopted a system architecture and templates quite close to the ones available in the operational world. Users are already familiar with them so that system interaction is straightforward and training demands, minimum. The implication of users has been intense and constant and this has allowed the execution of very interesting campaigns and the compilation of large datasets.
Project Context and Objectives:
NEREIDS was conceived to provide an integrated vision of maritime policy and surveillance so that the different elements of the service become useful to the different domains (immigration, fisheries control...). In this context, the project aimed to enhance automatic and unsupervised ship monitoring capabilities for Maritime Situational Awareness (MSA) and, to support advanced and efficient decision making tools. Key overall objectives were:
- Understand the current limitations of EO imagery and to identify the maritime surveillance domains where added value can benefit users.
- Contribute in reducing existing barriers for the users to adopt new technologies in the maritime security domain where user community is very fragmented with different cultures and priorities.
- Contribute to the definition of the future maritime surveillance services within Copernicus and the future space component for the Security dimension of Copernicus.
- Ensure that key EU and national policies areas are addressed.
- Define an open architecture (toolbox) that simplifies the process of defining, implementing, testing and running into live demonstration new EO based capabilities for maritime surveillance.
- Implement a simple and robust architecture that is based on standards that integrates the different elements in a fully automatic way (no human operator intervention is needed to complement and/or publish the final reports). Key elements are:
o Collection: The collection of information shall be independent from the sensor, the technology and the format.
o Fusion: Fusion shall be independent of the data source and shall be focused to fill information gaps for uncertainty reduction.
o Analysis: Security sensitive analysis should be carried out separately.
o Dissemination: The right information should be moved to the right decision maker at the right time. Access to information requires appropriate permissions.
- Better understand of operator activities by analysing maritime anomalies and define how EO can support the identification process.
- Foster collaboration with other initiatives for an optimal use of resources and dissemination of results.
- Define and execute well defined demonstrations with measurable results over different areas covering multi-sectorial interests.
The NEREIDS goals were pursued by delivering a test benchmark that permitted assessing new algorithms that can overcome classical drawbacks and/or can deliver new features in MSA. For such purpose, a system of systems concept was conceived, designed and implemented. It is based in the so-called Service Oriented Architecture (SOA) that allows geographically distributed processing modules to be connected to a centralized kernel. Besides management and security issues, the kernel keeps logistic functionalities, such as input data and product ingestion, data entries storage and reports publication. The processing modules provide the system with the algorithms that permit retrieving the parameters of interest from the incoming data. If one fails the overall chain is not at risk. Different module versions can be tested at the same time without affecting overall system runtime. The system was implemented under operational time delivery constraints so that the processes were optimized in view of fully automating.
The NEREIDS effort has been done in alignment with current European initiatives and policies. Examples are the Integrated Maritime Policy (IMP), The European External Action (EEA), the Common Information Sharing Environment (CISE) or the EUROSUR (European Border Surveillance) network. These policies attain end-user activities with the aim to limit the exploitation of the natural resources, to reinforce the security of human beings and to harmonize the activities conduced on the maritime domain. The awareness of European authorities to manage the European waters under an integrated approach that balance the economic growth with the citizens’ security and environment protection has been notably increased in the last years. Among others, this is confirmed by the recent initiatives promoted through R&D programs (for instance, FP7 and H2020) and the issue of the Maritime Spatial Planning directive. Such management should be done beyond national competences so that the potential conflicts among Member States are correctly solved.
NEREIDS has been executed with a total alignment to a list of requirements that have been defined in base of the restrictions imposed by the applicable policies and user needs. Thus, the proposed solutions are potential candidates to be used into the operational chains that authorities use for law verification, surveillance, control and management. It is worth noting that a 3-year project is enough time to see an evolution in the policies, trends and collaboration among agencies. An example is the Service Level Agreement that EMSA, SATCEN and Frontex have signed with the aim to achieve an efficient use of resources and avoid duplications of efforts / duties. This has affected how the operational systems were conceived and worked. NEREIDS adapted to this dynamic environment by reissuing key requirements and adapting the test benchmark to the new system concepts.
Project Results:
Achieved objectives that deserved further attention are:
- Target detection by automatically processing SAR imagery with improved capabilities on locating small targets (comparable with the available resolution) and on estimating macro-scale features (length, width, heading and speed)
- Target categorization by automatically processing SAR imagery as a function of pre-defined categories
- Target detection and categorization by processing optical imagery with fully automatic algorithms. Up to now, semi-automatic approaches are only available
- Advanced fusion approaches able to deal with multi-sensor data and to compensate the associated aberrations (time shift, location inaccuracies…)
- Constraint-based track reconstruction where tracks are consistently generated by adding constraints such as time grid, coastline, corridors… The results are smooth tracks with realistic appearance
- Complex anomaly detection by taking the maritime domain under analysis into account. An example yields on differentiating fishery related anomalies with respect to the traffic monitoring ones
- Propagation of vessel routes for traffic picture compilation and analysis in case of sensor measurements lack; to this aim probabilistic non-linear models and novel tracking techniques exploit context information, such as traffic patterns, coastline and ports, and vessel data (MMSI, type, destination)
- Flexible, modular and automatic system architecture compliant with the standards used by currently operational systems (such as IMDATE from EMSA) and able to ingest any data format coming from any sensor (either EO or cooperative)
- Advanced DB structure that permit a multi-user multi-purpose approach. This is the key success of the system architecture performance as different modules were available to access the system to modify available data at the same time
- Real time Graphical User Interface (GUI) that permitted an integrated vision of the maritime scene under processing with an accuracy and realism close to the real life. This has opened the door the NEREIDS system to be exploited in real life operational missions
Along project lifetime, nine campaigns have been carried out.
- March 2013 at the West Africa coast focused to monitor piracy, traffic and fisheries
- May 2013 at the Algarve area in Portugal mainland focused to test the capabilities for border surveillance monitoring
- June 2013 at the Central Mediterranean sector focused to monitor fisheries (Bluefin tuna) and irregular immigration
- August 2013 at the Alboran sea to monitor traffic and irregular immigration
- September 2013 at the Alboran sea to monitor custom dealing
- March 2014 at the Alesund fishing harbour in Norway focused to monitor traffic and fisheries.
- April 2014 at the NAFO fishing area close to the Eastern Canada shoreline focused to monitor fisheries.
- May 2014 at Setubal and Algarve areas of Portugal focused to test the capabilities for border surveillance monitoring
- May 2014 at the Gulf of Lybia and Lampedusa focused to monitor fisheries and irregular immigration
All the campaigns were designed and executed under near-real time delivery constraint, even though this was not possible in some cases due to limitations in data delivery. Optical data were not available during the campaigns, but images were ordered later for three campaigns separately as archive data. Note that real-time capability is beyond project objectives because NEREIDS was conceived to work in a test benchmark environment far away from the operational scenarios. Thus, the link that has been developed between the scientific and operational world can be considered a big success of the project. This permitted the establishment of a continuous cyclic engineering process for which the feedback gathered in one campaign served to fix the major developments that would be tested in incoming campaigns. This provided the user with a closer vision about the advantages / limitations of the tested technologies, mainly EO imagery. In addition, this increased the perception about the usefulness of EO data in ship monitoring and surveillance.
The exercises were executed with a notable logistic effort. First of all, the acquisition of cooperative data was done through different data providers in order to perform a kind of test benchmark among them. The associated data quality was evaluated against a set of metrics. The results show that fusing the cooperative data delivered by different providers is not straightforward. Among the detected phenomena, time shift would be the worst one. This prevents the tracks to be smooth and follow a straight and realistic line. On the contrary, a continuous forward and backward effect is observed emulating the shape of a saw. New algorithms have been successfully developed to compensate this effect, which has been almost mitigated for the evaluated data sets.
The acquisition of satellite imagery has been benefited by the GEST agreement between ESA and EC for which NEREIDS was assigned with a specific quota within the Data Ware House (DWH). The interaction with the GEST team has been constant, intense and very useful. Training with users permitted evolving into a more efficient and straightforward way how data acquisition plans and the associated orders were defined and issued. The amount of data assigned to NEREIDS has been sufficient to cover all project needs; almost all the sensors and imaging modes have been tested. The findings showed that no asset is performing better than the rest as advantages and limitations are detected for each of them. Actually, the combination of the available assets in a kind of multi-sensor constellation would be the predominant trend into the future as this would permit reducing revisiting time. In this sense, new platform and orbit concepts were studied in NEREIDS to overcome the inherent limitations of polar orbits, very popular in EO satellite imagery. Equatorial orbits with real time video links would be the next step forward as a second generation of satellites with integrated AIS receivers will be launch at short term.
In some cases, the campaigns were complemented with local surveillance means, which track was synchronized with the satellite overpass. This is the case of the GNR demos where blind exercises were conceived and executed. Controlled scenarios were selected by the user with different target configuration and with variant sea conditions according to the real and uncontrolled weather. Satellite imagery (spotlight SAR imagery with 1 m of resolution) were processed with no support of cooperative data and no information about the scenario configuration. The goal was to detect challenging targets (non-metallic, dynamic, small and with lengths between 4 and 14 m) in challenging conditions within operational time delivery constraints. Almost all the scenarios were successfully detected but at the price of increasing the number of false alarms. Although IT issues prevented delivering the data in near-real time (< 3 h from image acquisition), the feedback received from users was very positive specially in what regards the success of detecting 4 m long rubber boats with external engines.
The results of the campaigns have proven the increased performance and accuracy that the new tested algorithms provide in the field of ship detection and categorization, data fusion, track generation, track propagation and anomaly processing. All the results were validated against a pre-defined validation plan that took specific metrics into account. These metrics permitted quantifying the performance of the modules and the reliability of the proposed algorithms, i.e. the capability to improve current performance. Topics that were quantified are
- The capability to detect small, non-metallic and dynamic targets
- The capability to categorize ships with diverse acquisition modes
- The capability to generate advanced ship tracking by integrating diverse data sources and by applying different constraints (coastline, bathymetry…)
- The capability to manage advanced anomaly detection by defining complex anomalies (e.g. unauthorised fishing vessels actually fishing in a restricted area)
- The capability to provide route propagation by analysing the current maritime conditions and the available boundary/context information
- The capability to provide an integrated maritime picture with a visor having complex features. Actually, the NEREIDS viewer is a version of the visor adopted by EMSA for its IMDATE system.
In all the cases, the reliability is larger than 80%, which means that in 80% of the cases the proposed algorithms provide an improved performance with respect to the former versions. This was evaluated in different challenging scenarios that cause problems to the processing methods (in terms of false alarms and similar). Especially remarkable is the test benchmark exercise executed for different algorithms in the field of SAR-based ship detection, data fusion and anomaly analysis. The associated performance was cross-checked under certain challenging scenarios with the aim to identify the associated pros and contras. All the algorithms were run in a fully automatic way so that manual global adjustments (e.g. manually setting a certain value for a global threshold or choosing a certain buffer area from the coast) were acceptable, but manually editing of the final reports was not.
For the VDS domain, four SAR-based ship detection systems were tested (from JRC, GMV, UPC and eOsphere) under five scenarios (ambiguities, sidelobes, structures clutter, coastline and large targets), which normally drop the performance of classical approaches. The conclusions show that there is not a perfect fully automatic detector and that there is room for improvements. The combination of the advantages of all of them would be of particular interest in view of creating a new system with impacting performance from the user point of view. At the moment, the final supervision by a human operator is still advisable and the estimation of length and width is closer to the accuracy level demanded by users. Similar conclusions can be applied to the algorithm for the detection and classification from optical images.
For the data fusion domain, the benchmarking was a twofold objective: 1) to evaluate the performance of different data providers against a consistent link (MSSIS network) and 2) to evaluate the performance of a unique fusion algorithm with different configuration parameters. The results show that the combination of multi-source data permits filling information gaps at the price of increasing the fusion problems (time shift and similar). Regarding configuration, diverse gating thresholds were adopted. The results show that the lower the gating threshold, the better.
For the anomaly domain, the benchmarking was aimed to evaluate the capability to differentiate domain-specific anomalies. Special interest was focused on fisheries because the complex dynamics of the associated tracks makes the exercise challenging. The results show that complex anomalies can be described in a simpler manner if ancillary sources of information (like authorized fishing ships and similar) are ingested. This makes their processing and analysis more straightforward. In particular, the vessel databases of EMSA and EFCA are essential for an operational ship monitoring system.
The relationship with users have been intense and fruitful. This has permitted conceiving complex campaigns that implied the coordinated exploitation of different data streams acquired from diverse data providers. In addition, users provided the project with specific datasets not available in the pure commercial net, which notably complemented the overall maritime picture (for instance, VMS and LRIT cooperative streams, ship-borne radar snapshots or GPS tracks of surveillance elements). During the campaigns, the feedback received from users have been continuous and excellent. It was oriented to give advice about potential areas and/or targets of interest, and to explain complex behaviours observed in the platform. After the campaigns, the evaluation of the results permitted identifying specific topics of interest and detecting the main deficiencies of the proposed algorithms. Most of them were handled by adopting a cyclic engineering process, which could be perfectly implemented thanks to the modular architecture of the system. The overall result has been a very complete and competitive system that can be aligned, in what regards performance, with other operational systems currently managed by users and/or European agencies.
In general, the collaboration with users has been very satisfactory for both sides. On the one hand, users were benefitted from a real and operational test benchmark with which they assessed the performance of a set of technologies and evaluated whether they would be included in the operational chains. On the other hand, project partners had access to a large range of data for testing new concepts and algorithms within a controlled environment that would be difficult to reach in real life. The main users involved in NEREIDS are:
- The Italian Coast Guard (ITCG) from Italy with mandates in monitoring irregular immigration, fisheries and traffic security. An agreement that established the datasets that would be exchanged between ITCG and NEREIDS was signed before executing the campaigns. The implication of ITCG in the Mediterranean campaigns has been intense as the user provided NEREIDS with
o Italian VMS and LRIT
o A link to the Pelagus system, which manages all the Terrestrial AIS within the Italian EEZ
NEREIDS provided ITCG with access to the NEREIDS system
- The Guarda Nacional Republicana (GNR) from Portugal with mandates in monitoring irregular immigration, border surveillance, custom dealing and traffic security. An agreement that established the datasets that would be exchanged between GNR and NEREIDS was signed before executing the campaigns. The implication of GNR in the Portugal campaigns has been total as the user has deployed ships exclusively for the NEREIDS project. This implied notable logistics efforts as personnel and equipment were mobilized. Main datasets delivered are
o Snapshots of ship-borne radar
o GPS tracks of the monitored ships
Both were delivered to NEREIDS once GNR received the monitoring results from NEREIDS.
- The Spanish Fishery Authority (SFA) from Spain with mandates in fisheries. The implication of SFA in the Mediterranean campaigns has been intense as the user provided NEREIDS with
o Spanish VMS
- The European Fishery Control Agency (EFCA) from Europe with mandates in fisheries. The implication of EFCA in the Mediterranean, Alesund and NAFO campaigns has been intense as the user provided NEREIDS with
o Protected areas
o Blacklist of ships
o Description of normal and anomalous fishery behaviours
Potential Impact:
All these results were intensively disseminated. The dissemination strategy of NEREIDS was coordinated with the other two projects (DOLPHIN and SIMTYSIS) granted in the same call than NEREIDS “FP7-SPACE-2010-1.1-05” under the activity “Contributing to the ―S “ in GMES “Developing pre-operational service capabilities for Maritime Surveillance”. This approach was proposed by EC in an effort to efficiently use dissemination resources, to present the three projects under the same maritime surveillance umbrella and to avoid annoying users by receiving similar messages from three different interlocutors and perspectives. On the contrary, the joint dissemination approach tried to harmonize the contents by assigning specific domains (piracy, immigration, fishery…) to each of the projects. It is worth noting that the concept is novel in EC and its success was not clearly foreseen at the beginning. However, the implication of EGEOS, SpaceTech, Thales Alenia Italy and GMV permitted to present the results in a consistent and coordinated way. The potential audience increased notably and end-users were benefited with clearer messages and a more complete set of dissemination elements. It is acknowledged by the three projects that a coordinated approach has benefitted all of them by not imposing too many additional constraints in terms of management and costs, and by increasing the chances of possible cross-fertilization.
The project NEREIDS participated in a common Projects Coordination Board (PCB) that was set up to define the scope of the joint dissemination and communication activities to be undertaken. The PCB was composed of the individual project coordinators (or their delegates), and REA and EC representatives as observers. Depending on the agenda, the relevant WP and Task leaders of each project was invited to participate in the PCB session. The chair of the PCB rotated among the project coordinators and the PCB meetings took place in Brussels, within the premises of the REA and/or possibly collocated with other meetings, workshops, or conferences to limit the possible additional costs resulting from the PCB meetings. The PCB was formally gathered every six months.
The main dissemination activities in NEREIDS has been
- NEREIDS has collaborated in the different dissemination activities that resulted from the joint effort among NEREIDS, DOLPHIN and SIMTISYS. To be highlighted:
o Joint brochure with information about the maritime domains that each project will work with (http://maritimesurveillance.security-copernicus.eu/);
o Multimedia presentation with details about the main results achieved by each project (http://maritimesurveillance.security-copernicus.eu/);
o Co-organization and attendance to the Maritime workshop organized in the European Maritime Day 2013 and 2014;
- The NEREIDS team has collaborated in some EU publications:
o Windows on Copernicus
o Copernicus Observer
o Parliament Magazine
o GMES Security User Forum
- The NEREIDS team has generated some scientific publications in peer per review journals. Relevant examples are:
o Margarit, G.; Tabasco, A, "Ship Classification in Single-Pol SAR Images Based on Fuzzy Logic," Geoscience and Remote Sensing, IEEE Transactions on , vol.49 no.8 pp.3129,3138 Aug. 2011 doi: 10.1109/TGRS.2011.2112371
o Margarit, G., “NEREIDS: New concepts in maritime surveillance for consolidating operational developments”, ESA SEASAR2012 workshop.
o Margarit, G., "Integrated maritime picture for surveillance and monitoring applications," Geoscience and Remote Sensing Symposium (IGARSS), 2013 IEEE International , vol., no., pp.1517,1520 21-26 July 2013 doi: 10.1109/IGARSS.2013.6723075
o Margarit, G., “Operational Activations of Maritime Surveillance Services within the Framework of MARISS, NEREIDS and SAGRES Projects”, ESA Living Planet Symposium 2013.
o Dekker, R., et al, “Maritime Situation Awareness Capabilities from Satellite and Terrestrial Sensor Systems”, MAST 2013 conference.
o Pallotta, G. et al, “Context-Enhanced Vessel Prediction Based On Ornstein-Uhlenbeck Processes Using Historical AIS Traffic Patterns: Real-World Experimental Results”, FUSION 2014 conference.
o Santamaria, C. et al, “Improvement of maritime target signatures in satellite SAR images”, EUSAR 2014 conference.
o Yam, L.E. Mallorqui, J.J. , Rius, J.M. Validation of a sea surface model for simulations of dynamic maritime SAR images, IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2012, pp. 2813-2816, July 2012. http://dx.doi.org/10.1109/IGARSS.2012.6350847
o Luis Yam, Jordi J. Mallorquí, Juan M. Rius, ”Constraints in azimuth resolution by refocusing slowly moving targets from single-look complex SAR vignettes”, 10th European Conference on Synthetic Aperture Radar, Berlin, Germany, June 2014.
o Rius, J.M. Carbo, A.; Bjerkemo, J.; Ubeda, E.; Heldring, A.; Mallorqui, J.J.; Broquetas, A., New Graphical Processing Technique for Fast Shadowing Computation in PO Surface Integral, IEEE Transactions on Antennas and Propagation, Volume: 62 , Issue: 5, Page(s): 2587 – 2595, May 2014, http://dx.doi.org/10.1109/TAP.2014.2307321
o Kanjir, U., Marsetič, A., Pehani, P. and Oštir, K., An automatic procedure for small vessel detection from very-high resolution optical imagery, 5th Geographic Object-Based Image Analysis Conference, Thessaloniki, Greece, May 21-24, 2014, pp. 559-562.
In view of the results obtained by NEREIDS, the following technical recommendations arise:
- Further evaluate the results of the VDS test benchmark to evolve current ship detectors based on SAR and optic imagery. The available performance is robust, but it degrades in front very specific challenging phenomena (ambiguities, sidelobes, coastline, large targets and coastline). New works shall focus on compensating the effects of all them in almost all the scenarios
- Fine tune categorization methods to adapt to very particular categories
- Integrate into the operational chains algorithms that compensate the aberrations and defocusing that would affect the signatures of targets in EO imagery (especially applies for SAR images). This degrades the capability to estimate target dimensions
- Devote efforts to permit the estimation of the target speed by others methods rather than the comparison of the target location with other static reference, such as the wake or the AIS track. New ideas in SAR images propose the exploitation of the Doppler information in the raw data
- A better integration between data fusion and route propagation is certainly a future field of research. The data collected in NEREIDS helped to understand a first approach and investigate ideas, but more work is required to automate the switching between data fusion and route propagation in order to correlate VDS contacts to tracks when there is a significant gap in time between the VDS contact and the tracks due to low data rates. This becomes even more crucial in case of sparse data.
- Development of new methods to understand how to deal with uncertainties in timestamps in order to use VDS operationally. The NEREIDS campaigns helped to better understand the problem of misaligned timestamps, but new tracking algorithms able to cope with timestamp uncertainty are needed. CMRE identified some benchmarks metrics that can be used for assessing track fusion, but more metrics are needed to assess the performance more thoroughly. The focus should be on unsupervised metrics.
- Integration of new type of context information in the Route Propagation Module, such as bathymetric data, meteorological information and traffic density maps
- Diversification of route propagation strategies according to the type of vessel. The definition of corridors or areas of transit for specific vessel types could broaden the prediction module capabilities
- Enhancement of the route propagation strategy for scenarios with high traffic density by including ship to ship interaction models
- Anomaly detection will be improved by adding more information such as vessel type and other information from vessel databases (e.g. EMSA and EFCA databases)
- Validation of the accuracy EO SAR correlations with ship tracks remains a challenge, as is to differentiate between false alarms in the EO SAR detections and detections for which there are no other data sources for correlation.
- Propose new visor features, such as the integration of real-time video or any other ground based sensor. The static reference shall as complete as possible, and shall include the references available at user premises
- System ontology should be carefully reviewed and adapted to the domains and user needs
From a technological point of view, the platform has considered mature enough to base the design of a potential Copernicus maritime service. Recommendations are:
- The service shall deal with the largest range of sensors
o Cooperative reporting systems
o Ground-based and/or coastal sensors (radar, IR, thermal, video…) with limited coverage but continuous surveillance.
o Airborne-based sensors with extended coverage but limited surveillance capability
o Satellite-based sensors with very large coverage but reduced surveillance capability
- All the assets communicate to an integrated Coordination Centre (CC) that receives all the data streams and builds an integrated maritime picture, which would be useful for operational, tactical and/or strategic missions
o The communication links would be exploited through communication satellites with adequate encryption levels
o All the assets can be interconnected among themselves
o Standards should apply in what regards data format
- More than one CC, which can cover specific regions, would be interconnected to build extended maritime pictures and/or to increase the strategic information
o A kind of EUROSUR network with different user profiles and different levels for accessing information would be defined
o Each CC shall assure the enough material and human resources so that CC performance is within the requirements by a Service Level Agreement (SLA)
o Each CC would have the own validation plan and the own resources to validate system performance
- The service will operate by means of a centralized maritime system with a data and service catalogue. It shall be interoperable with other systems
- Licensing is important and data shall be filtered in terms of user rights
- A service portfolio shall be defined. As minimum, the following services should be available:
o Ship detection with estimation of macro-scale features
o Ship tracking by combining Ship detection and cooperative entries
o Anomaly analysis
o Route propagation
o Pre-frontier Image analysis
- A unique entry point for users shall be available. The associated GUI shall permit sending service activations and visualizing the reports in the most complete and efficient way
- Even though automating is essential and would be achieved in almost all the procedures, the presence of a human operator is needed to at least confirm all the decisions made by the system. Otherwise, resources would not be efficiently exploited and/or they would not be properly tuned to user needs
From a policy point of view, the following recommendations apply:
(1) There is a clear need for integrating all the available information in a common surveillance system which the different agencies can access. Integration of National Maritime Surveillance Systems is a long running on-going process at MS level (e.g. IT, FR) for Maritime Situational Awareness purposes and at EU level, in support of the implementation of EU sectorial policies (e.g. EDA, EFCA, EMSA, FRONTEX, EUSC, etc.). NEREIDS can contribute with its modular architecture to support CISE and EUROSUR as one of the available solutions in the effective integration of all data sources in future MSA systems.
(2) NEREIDS was not perceived to propose new standards. Instead in the framework of the project, the consortium reviewed the use of standards on current and past initiatives and based on user requirements adopted the most relevant, matching with the objectives of the project.
Further innovation on the technical standards and enterprise services area is required. This, however, can only be achieved through a higher-level implementation project or initiatives. The contribution of the upcoming EU CISE2020 project is expected to be of major importance since the definition of standards is a key task for establishing CISE.
(3) Since 2008, the European Commission has programmed, under the 7th Framework Programme for Research and Development (FP7 - COPERNICUS), a dozen projects (amounting to 100 million Euros) in order to develop, integrate and demonstrate technical solutions for maritime border surveillance and in particular the urgent need for detection and tracking of small vessels.
Frontex would then gradually implement these solutions as a permanent service in the framework of EUROSUR. A wide dissemination of the outputs of NEREIDS can contribute to this implementation and support other project as SAGRES/LOBOS.
(4) Support to the decision making and operators is available in NEREIDS and a big effort was done to support the verification of specific policies. In this sense, the different users agree on the fact that the maritime surveillance awareness is improved by adding the processing of satellite imagery. However, some improvements are needed, as the delivery time shall be compatible with the operational missions. This is a general issue using satellite imagery and future developments of the systems will improve their efficiency.
(5) Inter-application: a surveillance system shall not only take into account those products related to the main task. For instance, security applications (ship tracking) shall consider products derived from the environmental maritime area (water quality, winds, currents…) or from external disciplines such weather forecast or socio-economic factors. Being modular, the NEREIDS platform is easily extendable to cover various needs for other derived applications. Again, the proposed platform can support initiatives already mentioned as CISE, EUROSUR, etc. due its characteristics.
(6) R&D: as mentioned before, during the project much relevant solutions have been addressed. We would like to highlight the Performance Measures and Effectiveness, which provide the possibility to evaluate quantitatively data fusion algorithms. Also, this can be used to do not generate some false expectations on user that sometimes are quite high.
(7) Avoid the overlapping of different initiatives in the field of maritime surveillance that may have an impact national and European level – and transform them instead on complementary actions
As a final summary, NEREIDS has successfully fulfilled all the overall and technical goals specified at the beginning of the project. The outcomes are very promising to be integrated into an operational ship monitoring system that combines cooperative and non-cooperative data. Beyond that, the concept of ‘system of systems’ has performed so efficiently that it is recommended to be used as a potential design for the Copernicus core maritime service. Note that NEREIDS differs from other monitoring and surveillance systems by the fact that the system not only integrates the reports received by external data and service providers, but permits the direct interaction of the external processing modules with the system resources (mainly DB). This is a new concept and is founded on new system strategies that open the door to the multi-purpose multi-data approach. Standardization and integration have been key design drivers in order to allow the system to interact with any other service.
NEREIDS is leaving as a heritage a consistent base for new research work where the lessons learned can be applied in the operational world. It has also built the experience to combine pure R&D studies with operational campaigns with a very active user enrolment. A ‘close-to-real’ platform was conceived, designed and implemented to provide users and service providers with a complex, but efficient test benchmark. Performance was close to the one reached by current operational services, which the system can transparently communicate with (IMDATE from EMSA). From a technical point of view, topics that deserve special attention would be:
- the capability to detect dynamic rubber boats with a length < 5 m,
- the capability to categorize ships in SAR images,
- the provision of a fully automatic ship detection chain working with optic data
- the capability to fuse multi-source data.
Oceans deserve to be safe and protected. The advanced knowledge managed in NEREIDS is a step forward towards this vital goal.
Project Coordinator: Gerard Margarit Martin (gmargarit@gmv.com) +932922484
The NEREIDS project has ended at 31-05-2014 after three years of intense and fruitful work. The implication of all the partners has permitted an optimum collaboration that produced excellent results. Some of them provide a clear break-through with respect to the current state-of-the-art, which applicability into real-life operational procedures has proven to be feasible. Examples are new algorithms capable to detect small targets (length < 5-10 m) and/or to categorize ships by processing Earth Observation (EO) images, new multi-source data fusion approaches, the definition of complex anomalies or the propagation of consolidated tracks into the future.
At the end of project lifetime and according to EC feedback, NEREIDS has achieved all the proposed objectives. In some specific tasks, performance is beyond expectations, especially in what regards:
- the amount of covered topics either technological (new orbit and/or missions concepts) or technical (new processing algorithms)
- the amount of demonstration campaigns with an active user enrolment and with design constraints close to the operational case
- the dissemination effort with a joint collaboration with other projects
System interoperability is another remarkable feature as the NEREIDS system can transparently and automatically work with other operational systems, such as the IMDATE managed by EMSA. In addition, NEREIDS has adopted a system architecture and templates quite close to the ones available in the operational world. Users are already familiar with them so that system interaction is straightforward and training demands, minimum. The implication of users has been intense and constant and this has allowed the execution of very interesting campaigns and the compilation of large datasets.
Project Context and Objectives:
NEREIDS was conceived to provide an integrated vision of maritime policy and surveillance so that the different elements of the service become useful to the different domains (immigration, fisheries control...). In this context, the project aimed to enhance automatic and unsupervised ship monitoring capabilities for Maritime Situational Awareness (MSA) and, to support advanced and efficient decision making tools. Key overall objectives were:
- Understand the current limitations of EO imagery and to identify the maritime surveillance domains where added value can benefit users.
- Contribute in reducing existing barriers for the users to adopt new technologies in the maritime security domain where user community is very fragmented with different cultures and priorities.
- Contribute to the definition of the future maritime surveillance services within Copernicus and the future space component for the Security dimension of Copernicus.
- Ensure that key EU and national policies areas are addressed.
- Define an open architecture (toolbox) that simplifies the process of defining, implementing, testing and running into live demonstration new EO based capabilities for maritime surveillance.
- Implement a simple and robust architecture that is based on standards that integrates the different elements in a fully automatic way (no human operator intervention is needed to complement and/or publish the final reports). Key elements are:
o Collection: The collection of information shall be independent from the sensor, the technology and the format.
o Fusion: Fusion shall be independent of the data source and shall be focused to fill information gaps for uncertainty reduction.
o Analysis: Security sensitive analysis should be carried out separately.
o Dissemination: The right information should be moved to the right decision maker at the right time. Access to information requires appropriate permissions.
- Better understand of operator activities by analysing maritime anomalies and define how EO can support the identification process.
- Foster collaboration with other initiatives for an optimal use of resources and dissemination of results.
- Define and execute well defined demonstrations with measurable results over different areas covering multi-sectorial interests.
The NEREIDS goals were pursued by delivering a test benchmark that permitted assessing new algorithms that can overcome classical drawbacks and/or can deliver new features in MSA. For such purpose, a system of systems concept was conceived, designed and implemented. It is based in the so-called Service Oriented Architecture (SOA) that allows geographically distributed processing modules to be connected to a centralized kernel. Besides management and security issues, the kernel keeps logistic functionalities, such as input data and product ingestion, data entries storage and reports publication. The processing modules provide the system with the algorithms that permit retrieving the parameters of interest from the incoming data. If one fails the overall chain is not at risk. Different module versions can be tested at the same time without affecting overall system runtime. The system was implemented under operational time delivery constraints so that the processes were optimized in view of fully automating.
The NEREIDS effort has been done in alignment with current European initiatives and policies. Examples are the Integrated Maritime Policy (IMP), The European External Action (EEA), the Common Information Sharing Environment (CISE) or the EUROSUR (European Border Surveillance) network. These policies attain end-user activities with the aim to limit the exploitation of the natural resources, to reinforce the security of human beings and to harmonize the activities conduced on the maritime domain. The awareness of European authorities to manage the European waters under an integrated approach that balance the economic growth with the citizens’ security and environment protection has been notably increased in the last years. Among others, this is confirmed by the recent initiatives promoted through R&D programs (for instance, FP7 and H2020) and the issue of the Maritime Spatial Planning directive. Such management should be done beyond national competences so that the potential conflicts among Member States are correctly solved.
NEREIDS has been executed with a total alignment to a list of requirements that have been defined in base of the restrictions imposed by the applicable policies and user needs. Thus, the proposed solutions are potential candidates to be used into the operational chains that authorities use for law verification, surveillance, control and management. It is worth noting that a 3-year project is enough time to see an evolution in the policies, trends and collaboration among agencies. An example is the Service Level Agreement that EMSA, SATCEN and Frontex have signed with the aim to achieve an efficient use of resources and avoid duplications of efforts / duties. This has affected how the operational systems were conceived and worked. NEREIDS adapted to this dynamic environment by reissuing key requirements and adapting the test benchmark to the new system concepts.
Project Results:
Achieved objectives that deserved further attention are:
- Target detection by automatically processing SAR imagery with improved capabilities on locating small targets (comparable with the available resolution) and on estimating macro-scale features (length, width, heading and speed)
- Target categorization by automatically processing SAR imagery as a function of pre-defined categories
- Target detection and categorization by processing optical imagery with fully automatic algorithms. Up to now, semi-automatic approaches are only available
- Advanced fusion approaches able to deal with multi-sensor data and to compensate the associated aberrations (time shift, location inaccuracies…)
- Constraint-based track reconstruction where tracks are consistently generated by adding constraints such as time grid, coastline, corridors… The results are smooth tracks with realistic appearance
- Complex anomaly detection by taking the maritime domain under analysis into account. An example yields on differentiating fishery related anomalies with respect to the traffic monitoring ones
- Propagation of vessel routes for traffic picture compilation and analysis in case of sensor measurements lack; to this aim probabilistic non-linear models and novel tracking techniques exploit context information, such as traffic patterns, coastline and ports, and vessel data (MMSI, type, destination)
- Flexible, modular and automatic system architecture compliant with the standards used by currently operational systems (such as IMDATE from EMSA) and able to ingest any data format coming from any sensor (either EO or cooperative)
- Advanced DB structure that permit a multi-user multi-purpose approach. This is the key success of the system architecture performance as different modules were available to access the system to modify available data at the same time
- Real time Graphical User Interface (GUI) that permitted an integrated vision of the maritime scene under processing with an accuracy and realism close to the real life. This has opened the door the NEREIDS system to be exploited in real life operational missions
Along project lifetime, nine campaigns have been carried out.
- March 2013 at the West Africa coast focused to monitor piracy, traffic and fisheries
- May 2013 at the Algarve area in Portugal mainland focused to test the capabilities for border surveillance monitoring
- June 2013 at the Central Mediterranean sector focused to monitor fisheries (Bluefin tuna) and irregular immigration
- August 2013 at the Alboran sea to monitor traffic and irregular immigration
- September 2013 at the Alboran sea to monitor custom dealing
- March 2014 at the Alesund fishing harbour in Norway focused to monitor traffic and fisheries.
- April 2014 at the NAFO fishing area close to the Eastern Canada shoreline focused to monitor fisheries.
- May 2014 at Setubal and Algarve areas of Portugal focused to test the capabilities for border surveillance monitoring
- May 2014 at the Gulf of Lybia and Lampedusa focused to monitor fisheries and irregular immigration
All the campaigns were designed and executed under near-real time delivery constraint, even though this was not possible in some cases due to limitations in data delivery. Optical data were not available during the campaigns, but images were ordered later for three campaigns separately as archive data. Note that real-time capability is beyond project objectives because NEREIDS was conceived to work in a test benchmark environment far away from the operational scenarios. Thus, the link that has been developed between the scientific and operational world can be considered a big success of the project. This permitted the establishment of a continuous cyclic engineering process for which the feedback gathered in one campaign served to fix the major developments that would be tested in incoming campaigns. This provided the user with a closer vision about the advantages / limitations of the tested technologies, mainly EO imagery. In addition, this increased the perception about the usefulness of EO data in ship monitoring and surveillance.
The exercises were executed with a notable logistic effort. First of all, the acquisition of cooperative data was done through different data providers in order to perform a kind of test benchmark among them. The associated data quality was evaluated against a set of metrics. The results show that fusing the cooperative data delivered by different providers is not straightforward. Among the detected phenomena, time shift would be the worst one. This prevents the tracks to be smooth and follow a straight and realistic line. On the contrary, a continuous forward and backward effect is observed emulating the shape of a saw. New algorithms have been successfully developed to compensate this effect, which has been almost mitigated for the evaluated data sets.
The acquisition of satellite imagery has been benefited by the GEST agreement between ESA and EC for which NEREIDS was assigned with a specific quota within the Data Ware House (DWH). The interaction with the GEST team has been constant, intense and very useful. Training with users permitted evolving into a more efficient and straightforward way how data acquisition plans and the associated orders were defined and issued. The amount of data assigned to NEREIDS has been sufficient to cover all project needs; almost all the sensors and imaging modes have been tested. The findings showed that no asset is performing better than the rest as advantages and limitations are detected for each of them. Actually, the combination of the available assets in a kind of multi-sensor constellation would be the predominant trend into the future as this would permit reducing revisiting time. In this sense, new platform and orbit concepts were studied in NEREIDS to overcome the inherent limitations of polar orbits, very popular in EO satellite imagery. Equatorial orbits with real time video links would be the next step forward as a second generation of satellites with integrated AIS receivers will be launch at short term.
In some cases, the campaigns were complemented with local surveillance means, which track was synchronized with the satellite overpass. This is the case of the GNR demos where blind exercises were conceived and executed. Controlled scenarios were selected by the user with different target configuration and with variant sea conditions according to the real and uncontrolled weather. Satellite imagery (spotlight SAR imagery with 1 m of resolution) were processed with no support of cooperative data and no information about the scenario configuration. The goal was to detect challenging targets (non-metallic, dynamic, small and with lengths between 4 and 14 m) in challenging conditions within operational time delivery constraints. Almost all the scenarios were successfully detected but at the price of increasing the number of false alarms. Although IT issues prevented delivering the data in near-real time (< 3 h from image acquisition), the feedback received from users was very positive specially in what regards the success of detecting 4 m long rubber boats with external engines.
The results of the campaigns have proven the increased performance and accuracy that the new tested algorithms provide in the field of ship detection and categorization, data fusion, track generation, track propagation and anomaly processing. All the results were validated against a pre-defined validation plan that took specific metrics into account. These metrics permitted quantifying the performance of the modules and the reliability of the proposed algorithms, i.e. the capability to improve current performance. Topics that were quantified are
- The capability to detect small, non-metallic and dynamic targets
- The capability to categorize ships with diverse acquisition modes
- The capability to generate advanced ship tracking by integrating diverse data sources and by applying different constraints (coastline, bathymetry…)
- The capability to manage advanced anomaly detection by defining complex anomalies (e.g. unauthorised fishing vessels actually fishing in a restricted area)
- The capability to provide route propagation by analysing the current maritime conditions and the available boundary/context information
- The capability to provide an integrated maritime picture with a visor having complex features. Actually, the NEREIDS viewer is a version of the visor adopted by EMSA for its IMDATE system.
In all the cases, the reliability is larger than 80%, which means that in 80% of the cases the proposed algorithms provide an improved performance with respect to the former versions. This was evaluated in different challenging scenarios that cause problems to the processing methods (in terms of false alarms and similar). Especially remarkable is the test benchmark exercise executed for different algorithms in the field of SAR-based ship detection, data fusion and anomaly analysis. The associated performance was cross-checked under certain challenging scenarios with the aim to identify the associated pros and contras. All the algorithms were run in a fully automatic way so that manual global adjustments (e.g. manually setting a certain value for a global threshold or choosing a certain buffer area from the coast) were acceptable, but manually editing of the final reports was not.
For the VDS domain, four SAR-based ship detection systems were tested (from JRC, GMV, UPC and eOsphere) under five scenarios (ambiguities, sidelobes, structures clutter, coastline and large targets), which normally drop the performance of classical approaches. The conclusions show that there is not a perfect fully automatic detector and that there is room for improvements. The combination of the advantages of all of them would be of particular interest in view of creating a new system with impacting performance from the user point of view. At the moment, the final supervision by a human operator is still advisable and the estimation of length and width is closer to the accuracy level demanded by users. Similar conclusions can be applied to the algorithm for the detection and classification from optical images.
For the data fusion domain, the benchmarking was a twofold objective: 1) to evaluate the performance of different data providers against a consistent link (MSSIS network) and 2) to evaluate the performance of a unique fusion algorithm with different configuration parameters. The results show that the combination of multi-source data permits filling information gaps at the price of increasing the fusion problems (time shift and similar). Regarding configuration, diverse gating thresholds were adopted. The results show that the lower the gating threshold, the better.
For the anomaly domain, the benchmarking was aimed to evaluate the capability to differentiate domain-specific anomalies. Special interest was focused on fisheries because the complex dynamics of the associated tracks makes the exercise challenging. The results show that complex anomalies can be described in a simpler manner if ancillary sources of information (like authorized fishing ships and similar) are ingested. This makes their processing and analysis more straightforward. In particular, the vessel databases of EMSA and EFCA are essential for an operational ship monitoring system.
The relationship with users have been intense and fruitful. This has permitted conceiving complex campaigns that implied the coordinated exploitation of different data streams acquired from diverse data providers. In addition, users provided the project with specific datasets not available in the pure commercial net, which notably complemented the overall maritime picture (for instance, VMS and LRIT cooperative streams, ship-borne radar snapshots or GPS tracks of surveillance elements). During the campaigns, the feedback received from users have been continuous and excellent. It was oriented to give advice about potential areas and/or targets of interest, and to explain complex behaviours observed in the platform. After the campaigns, the evaluation of the results permitted identifying specific topics of interest and detecting the main deficiencies of the proposed algorithms. Most of them were handled by adopting a cyclic engineering process, which could be perfectly implemented thanks to the modular architecture of the system. The overall result has been a very complete and competitive system that can be aligned, in what regards performance, with other operational systems currently managed by users and/or European agencies.
In general, the collaboration with users has been very satisfactory for both sides. On the one hand, users were benefitted from a real and operational test benchmark with which they assessed the performance of a set of technologies and evaluated whether they would be included in the operational chains. On the other hand, project partners had access to a large range of data for testing new concepts and algorithms within a controlled environment that would be difficult to reach in real life. The main users involved in NEREIDS are:
- The Italian Coast Guard (ITCG) from Italy with mandates in monitoring irregular immigration, fisheries and traffic security. An agreement that established the datasets that would be exchanged between ITCG and NEREIDS was signed before executing the campaigns. The implication of ITCG in the Mediterranean campaigns has been intense as the user provided NEREIDS with
o Italian VMS and LRIT
o A link to the Pelagus system, which manages all the Terrestrial AIS within the Italian EEZ
NEREIDS provided ITCG with access to the NEREIDS system
- The Guarda Nacional Republicana (GNR) from Portugal with mandates in monitoring irregular immigration, border surveillance, custom dealing and traffic security. An agreement that established the datasets that would be exchanged between GNR and NEREIDS was signed before executing the campaigns. The implication of GNR in the Portugal campaigns has been total as the user has deployed ships exclusively for the NEREIDS project. This implied notable logistics efforts as personnel and equipment were mobilized. Main datasets delivered are
o Snapshots of ship-borne radar
o GPS tracks of the monitored ships
Both were delivered to NEREIDS once GNR received the monitoring results from NEREIDS.
- The Spanish Fishery Authority (SFA) from Spain with mandates in fisheries. The implication of SFA in the Mediterranean campaigns has been intense as the user provided NEREIDS with
o Spanish VMS
- The European Fishery Control Agency (EFCA) from Europe with mandates in fisheries. The implication of EFCA in the Mediterranean, Alesund and NAFO campaigns has been intense as the user provided NEREIDS with
o Protected areas
o Blacklist of ships
o Description of normal and anomalous fishery behaviours
Potential Impact:
All these results were intensively disseminated. The dissemination strategy of NEREIDS was coordinated with the other two projects (DOLPHIN and SIMTYSIS) granted in the same call than NEREIDS “FP7-SPACE-2010-1.1-05” under the activity “Contributing to the ―S “ in GMES “Developing pre-operational service capabilities for Maritime Surveillance”. This approach was proposed by EC in an effort to efficiently use dissemination resources, to present the three projects under the same maritime surveillance umbrella and to avoid annoying users by receiving similar messages from three different interlocutors and perspectives. On the contrary, the joint dissemination approach tried to harmonize the contents by assigning specific domains (piracy, immigration, fishery…) to each of the projects. It is worth noting that the concept is novel in EC and its success was not clearly foreseen at the beginning. However, the implication of EGEOS, SpaceTech, Thales Alenia Italy and GMV permitted to present the results in a consistent and coordinated way. The potential audience increased notably and end-users were benefited with clearer messages and a more complete set of dissemination elements. It is acknowledged by the three projects that a coordinated approach has benefitted all of them by not imposing too many additional constraints in terms of management and costs, and by increasing the chances of possible cross-fertilization.
The project NEREIDS participated in a common Projects Coordination Board (PCB) that was set up to define the scope of the joint dissemination and communication activities to be undertaken. The PCB was composed of the individual project coordinators (or their delegates), and REA and EC representatives as observers. Depending on the agenda, the relevant WP and Task leaders of each project was invited to participate in the PCB session. The chair of the PCB rotated among the project coordinators and the PCB meetings took place in Brussels, within the premises of the REA and/or possibly collocated with other meetings, workshops, or conferences to limit the possible additional costs resulting from the PCB meetings. The PCB was formally gathered every six months.
The main dissemination activities in NEREIDS has been
- NEREIDS has collaborated in the different dissemination activities that resulted from the joint effort among NEREIDS, DOLPHIN and SIMTISYS. To be highlighted:
o Joint brochure with information about the maritime domains that each project will work with (http://maritimesurveillance.security-copernicus.eu/);
o Multimedia presentation with details about the main results achieved by each project (http://maritimesurveillance.security-copernicus.eu/);
o Co-organization and attendance to the Maritime workshop organized in the European Maritime Day 2013 and 2014;
- The NEREIDS team has collaborated in some EU publications:
o Windows on Copernicus
o Copernicus Observer
o Parliament Magazine
o GMES Security User Forum
- The NEREIDS team has generated some scientific publications in peer per review journals. Relevant examples are:
o Margarit, G.; Tabasco, A, "Ship Classification in Single-Pol SAR Images Based on Fuzzy Logic," Geoscience and Remote Sensing, IEEE Transactions on , vol.49 no.8 pp.3129,3138 Aug. 2011 doi: 10.1109/TGRS.2011.2112371
o Margarit, G., “NEREIDS: New concepts in maritime surveillance for consolidating operational developments”, ESA SEASAR2012 workshop.
o Margarit, G., "Integrated maritime picture for surveillance and monitoring applications," Geoscience and Remote Sensing Symposium (IGARSS), 2013 IEEE International , vol., no., pp.1517,1520 21-26 July 2013 doi: 10.1109/IGARSS.2013.6723075
o Margarit, G., “Operational Activations of Maritime Surveillance Services within the Framework of MARISS, NEREIDS and SAGRES Projects”, ESA Living Planet Symposium 2013.
o Dekker, R., et al, “Maritime Situation Awareness Capabilities from Satellite and Terrestrial Sensor Systems”, MAST 2013 conference.
o Pallotta, G. et al, “Context-Enhanced Vessel Prediction Based On Ornstein-Uhlenbeck Processes Using Historical AIS Traffic Patterns: Real-World Experimental Results”, FUSION 2014 conference.
o Santamaria, C. et al, “Improvement of maritime target signatures in satellite SAR images”, EUSAR 2014 conference.
o Yam, L.E. Mallorqui, J.J. , Rius, J.M. Validation of a sea surface model for simulations of dynamic maritime SAR images, IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2012, pp. 2813-2816, July 2012. http://dx.doi.org/10.1109/IGARSS.2012.6350847
o Luis Yam, Jordi J. Mallorquí, Juan M. Rius, ”Constraints in azimuth resolution by refocusing slowly moving targets from single-look complex SAR vignettes”, 10th European Conference on Synthetic Aperture Radar, Berlin, Germany, June 2014.
o Rius, J.M. Carbo, A.; Bjerkemo, J.; Ubeda, E.; Heldring, A.; Mallorqui, J.J.; Broquetas, A., New Graphical Processing Technique for Fast Shadowing Computation in PO Surface Integral, IEEE Transactions on Antennas and Propagation, Volume: 62 , Issue: 5, Page(s): 2587 – 2595, May 2014, http://dx.doi.org/10.1109/TAP.2014.2307321
o Kanjir, U., Marsetič, A., Pehani, P. and Oštir, K., An automatic procedure for small vessel detection from very-high resolution optical imagery, 5th Geographic Object-Based Image Analysis Conference, Thessaloniki, Greece, May 21-24, 2014, pp. 559-562.
In view of the results obtained by NEREIDS, the following technical recommendations arise:
- Further evaluate the results of the VDS test benchmark to evolve current ship detectors based on SAR and optic imagery. The available performance is robust, but it degrades in front very specific challenging phenomena (ambiguities, sidelobes, coastline, large targets and coastline). New works shall focus on compensating the effects of all them in almost all the scenarios
- Fine tune categorization methods to adapt to very particular categories
- Integrate into the operational chains algorithms that compensate the aberrations and defocusing that would affect the signatures of targets in EO imagery (especially applies for SAR images). This degrades the capability to estimate target dimensions
- Devote efforts to permit the estimation of the target speed by others methods rather than the comparison of the target location with other static reference, such as the wake or the AIS track. New ideas in SAR images propose the exploitation of the Doppler information in the raw data
- A better integration between data fusion and route propagation is certainly a future field of research. The data collected in NEREIDS helped to understand a first approach and investigate ideas, but more work is required to automate the switching between data fusion and route propagation in order to correlate VDS contacts to tracks when there is a significant gap in time between the VDS contact and the tracks due to low data rates. This becomes even more crucial in case of sparse data.
- Development of new methods to understand how to deal with uncertainties in timestamps in order to use VDS operationally. The NEREIDS campaigns helped to better understand the problem of misaligned timestamps, but new tracking algorithms able to cope with timestamp uncertainty are needed. CMRE identified some benchmarks metrics that can be used for assessing track fusion, but more metrics are needed to assess the performance more thoroughly. The focus should be on unsupervised metrics.
- Integration of new type of context information in the Route Propagation Module, such as bathymetric data, meteorological information and traffic density maps
- Diversification of route propagation strategies according to the type of vessel. The definition of corridors or areas of transit for specific vessel types could broaden the prediction module capabilities
- Enhancement of the route propagation strategy for scenarios with high traffic density by including ship to ship interaction models
- Anomaly detection will be improved by adding more information such as vessel type and other information from vessel databases (e.g. EMSA and EFCA databases)
- Validation of the accuracy EO SAR correlations with ship tracks remains a challenge, as is to differentiate between false alarms in the EO SAR detections and detections for which there are no other data sources for correlation.
- Propose new visor features, such as the integration of real-time video or any other ground based sensor. The static reference shall as complete as possible, and shall include the references available at user premises
- System ontology should be carefully reviewed and adapted to the domains and user needs
From a technological point of view, the platform has considered mature enough to base the design of a potential Copernicus maritime service. Recommendations are:
- The service shall deal with the largest range of sensors
o Cooperative reporting systems
o Ground-based and/or coastal sensors (radar, IR, thermal, video…) with limited coverage but continuous surveillance.
o Airborne-based sensors with extended coverage but limited surveillance capability
o Satellite-based sensors with very large coverage but reduced surveillance capability
- All the assets communicate to an integrated Coordination Centre (CC) that receives all the data streams and builds an integrated maritime picture, which would be useful for operational, tactical and/or strategic missions
o The communication links would be exploited through communication satellites with adequate encryption levels
o All the assets can be interconnected among themselves
o Standards should apply in what regards data format
- More than one CC, which can cover specific regions, would be interconnected to build extended maritime pictures and/or to increase the strategic information
o A kind of EUROSUR network with different user profiles and different levels for accessing information would be defined
o Each CC shall assure the enough material and human resources so that CC performance is within the requirements by a Service Level Agreement (SLA)
o Each CC would have the own validation plan and the own resources to validate system performance
- The service will operate by means of a centralized maritime system with a data and service catalogue. It shall be interoperable with other systems
- Licensing is important and data shall be filtered in terms of user rights
- A service portfolio shall be defined. As minimum, the following services should be available:
o Ship detection with estimation of macro-scale features
o Ship tracking by combining Ship detection and cooperative entries
o Anomaly analysis
o Route propagation
o Pre-frontier Image analysis
- A unique entry point for users shall be available. The associated GUI shall permit sending service activations and visualizing the reports in the most complete and efficient way
- Even though automating is essential and would be achieved in almost all the procedures, the presence of a human operator is needed to at least confirm all the decisions made by the system. Otherwise, resources would not be efficiently exploited and/or they would not be properly tuned to user needs
From a policy point of view, the following recommendations apply:
(1) There is a clear need for integrating all the available information in a common surveillance system which the different agencies can access. Integration of National Maritime Surveillance Systems is a long running on-going process at MS level (e.g. IT, FR) for Maritime Situational Awareness purposes and at EU level, in support of the implementation of EU sectorial policies (e.g. EDA, EFCA, EMSA, FRONTEX, EUSC, etc.). NEREIDS can contribute with its modular architecture to support CISE and EUROSUR as one of the available solutions in the effective integration of all data sources in future MSA systems.
(2) NEREIDS was not perceived to propose new standards. Instead in the framework of the project, the consortium reviewed the use of standards on current and past initiatives and based on user requirements adopted the most relevant, matching with the objectives of the project.
Further innovation on the technical standards and enterprise services area is required. This, however, can only be achieved through a higher-level implementation project or initiatives. The contribution of the upcoming EU CISE2020 project is expected to be of major importance since the definition of standards is a key task for establishing CISE.
(3) Since 2008, the European Commission has programmed, under the 7th Framework Programme for Research and Development (FP7 - COPERNICUS), a dozen projects (amounting to 100 million Euros) in order to develop, integrate and demonstrate technical solutions for maritime border surveillance and in particular the urgent need for detection and tracking of small vessels.
Frontex would then gradually implement these solutions as a permanent service in the framework of EUROSUR. A wide dissemination of the outputs of NEREIDS can contribute to this implementation and support other project as SAGRES/LOBOS.
(4) Support to the decision making and operators is available in NEREIDS and a big effort was done to support the verification of specific policies. In this sense, the different users agree on the fact that the maritime surveillance awareness is improved by adding the processing of satellite imagery. However, some improvements are needed, as the delivery time shall be compatible with the operational missions. This is a general issue using satellite imagery and future developments of the systems will improve their efficiency.
(5) Inter-application: a surveillance system shall not only take into account those products related to the main task. For instance, security applications (ship tracking) shall consider products derived from the environmental maritime area (water quality, winds, currents…) or from external disciplines such weather forecast or socio-economic factors. Being modular, the NEREIDS platform is easily extendable to cover various needs for other derived applications. Again, the proposed platform can support initiatives already mentioned as CISE, EUROSUR, etc. due its characteristics.
(6) R&D: as mentioned before, during the project much relevant solutions have been addressed. We would like to highlight the Performance Measures and Effectiveness, which provide the possibility to evaluate quantitatively data fusion algorithms. Also, this can be used to do not generate some false expectations on user that sometimes are quite high.
(7) Avoid the overlapping of different initiatives in the field of maritime surveillance that may have an impact national and European level – and transform them instead on complementary actions
As a final summary, NEREIDS has successfully fulfilled all the overall and technical goals specified at the beginning of the project. The outcomes are very promising to be integrated into an operational ship monitoring system that combines cooperative and non-cooperative data. Beyond that, the concept of ‘system of systems’ has performed so efficiently that it is recommended to be used as a potential design for the Copernicus core maritime service. Note that NEREIDS differs from other monitoring and surveillance systems by the fact that the system not only integrates the reports received by external data and service providers, but permits the direct interaction of the external processing modules with the system resources (mainly DB). This is a new concept and is founded on new system strategies that open the door to the multi-purpose multi-data approach. Standardization and integration have been key design drivers in order to allow the system to interact with any other service.
NEREIDS is leaving as a heritage a consistent base for new research work where the lessons learned can be applied in the operational world. It has also built the experience to combine pure R&D studies with operational campaigns with a very active user enrolment. A ‘close-to-real’ platform was conceived, designed and implemented to provide users and service providers with a complex, but efficient test benchmark. Performance was close to the one reached by current operational services, which the system can transparently communicate with (IMDATE from EMSA). From a technical point of view, topics that deserve special attention would be:
- the capability to detect dynamic rubber boats with a length < 5 m,
- the capability to categorize ships in SAR images,
- the provision of a fully automatic ship detection chain working with optic data
- the capability to fuse multi-source data.
Oceans deserve to be safe and protected. The advanced knowledge managed in NEREIDS is a step forward towards this vital goal.
Project Coordinator: Gerard Margarit Martin (gmargarit@gmv.com) +932922484
