Final Report Summary - AMASS (Autonomous maritime surveillance system)
Executive summary:
The concept of the AMASS project was to develop a surveillance system for the observation and security of wide critical maritime areas in order to reduce actual and potential illegal immigration and the trafficking of drugs, weapons and illicit substances.
The objectives of the project were to carry out the key research and technological development work which would lead to an engineered product following the completion of the project comprising of a network of unmanned platforms located a considerable distance from shore.
The project has successfully designed a complete system consisting of a number of integrated modules, including: flotation platform, optronics, hydrophones, communications, power management, image exploitation and command and control. Initial test and evaluation as carried out in the Atlantic Ocean off Gran Canaria and these together with further simulation have proved the concept of the system.
Project context and objectives:
The concept of the AMASS project was to develop a surveillance system for the observation and security of wide critical maritime areas in order to reduce actual and potential illegal immigration and the trafficking of drugs, weapons and illicit substances.
The objectives of the project were to carry out the key research and technological development work which would lead to an engineered product following the completion of the project comprising of a network of unmanned platforms located a considerable distance from shore. The platforms had to be designed to detect small boats both during the day and at night. Each platform was designed to be fitted with cutting-edge sensors and to operate self-sufficiently, i.e. without the need for manual intervention. Data captured by the sensors is transmitted to a central command centre, where an operator views it on screen. If a suspicious entity is detected, a crew can be dispatched to investigate.
Project results:
Description of main Science and technology (S&T) results / foregrounds
An overall system was developed which consisted of a number of key modules and their integration, including: flotation platform, optronics, hydrophones, communications, power management, image exploitation and command and control. The following section illustrates the buoy concept and realisation. The next sections present individual modules and the final section shows different buoys at sea during tests.
Autonomous maritime surveillance system (AMASS prototype)
Knowledge of end user requirements for autonomous maritime surveillance system
The project included actual end users and organisation with close contacts to end users. The initial phase of the project collected and documented the end user requirements for an autonomous maritime surveillance system.
Overall system engineering
The system was designed, developed, integrated and deployed.
Optronic subsystem
Carl Zeiss Optronics developed the optronic subsystem comprising of an uncooled thermal imaging camera module (UCM) with very low power consumption mounted on a stabilised platform. The camera module itself is sealed and uses the specially coated front lens as a window. Thus it can withstand the harsh environmental conditions at sea without an additional housing being necessary which would have put more weight on the top of the platform. Based on the statistics of ocean waves and buoy movement and a combination of passive and active means a stabilisation level of the camera was achieved in 3 axis (pitch, roll and yaw) that constitutes a good basis for the following image stabilisation.
Uncooled camera module UCM with automated athermalisation
The requirements of low-power consumption, low-weight and long-life time prohibit the use of a thermal imaging camera equipped with a cooled infra-red detector because the necessary cooling engine would impose narrow limits in regard to all three requirements. Instead microbolometer detectors are the best solution not only fulfilling the requirements stated above but also featuring high sensitivity in the Long-wave infrared (LWIR) spectral band comparable to what was state-of-the-art at cooled second generation detectors only a few years ago. The design of the optical concept was backed up by a range modelling using the software package TRM2 which is accepted for range calculation worldwide. Based on this concept a compact and robust camera module was designed and tested.
Virtually all LWIR objectives are using germanium as a substrate material for lens elements because of its favourable optical properties (extremely high-refractive index n equal to 4, high transmission and good machining properties). The only shortcoming is that the optical power of any germanium lens is strongly depending on temperature resulting in a significant defocusing of the camera. Therefore a procedure for compensating this deviation must be implemented which is commonly known as athermalisation. In general, temperature may vary from one lens element to the next and also may not be uniform over the mechanical structure. Therefore, an auto-collimation method was developed as an effective means for refocusing the camera instead of a simple shifting mechanism of the lens. The infrared radiation emitted by a reference target is collimated by the infrared objective and after being partially reflected back by a plane mirror travels through the same objective a second time in the reverse direction and is then imaged onto the detector surface. Thus a correct focusing state is indicated by a sharply focused image of the reference target which is assessed by a fast software algorithm.
2-stage stabilisation unit for UCM
Analysis of the buoy movement caused by the ocean waves shows that large angles appear in pitch and roll. In order to compensate this movement at least partially without employing powerful electric motors which would consume more electrical power than the buoy possibly could deliver a passive stabilisation means was developed. The basic element of the passive unit is a mass pendulum fixed in a gimbal frame mounting beneath the optronic platform. To reduce the oscillations of the camera's line-of-sight to an acceptable amount in all four directions effective spring-damper combinations are installed. By careful theoretic modelling and intensive testing on a 6-axis motion platform all parameters relevant for damping the oscillations (length of pendulum, mass of balance weight, spring and damper parameters and position) were optimised using the motion data for pitch, roll and heave obtained in the first phase of the project.
Furthermore, active stabilisation is added by integrating a stabilised pan-tilt unit PTU with integrated Inertial measurement unit (IMU) between sensor platform and the pendulum in order to:
- allow panning of the sensor platform;
- achieve stabilisation of the line of sight also in regard to yaw motions;
- as a fine stabilisation means along the elevation axis of the camera.
All together, the introduced 2-stage stabilisation unit is capable to attenuate the ocean wave induced oscillations of the camera's line-of-sight by a factor of almost 5. As most of the stabilisation work is carried out passively the power consumption is significant lower than that of purely active systems.
Array of hydrophone (AoH) module
Two units of AoH were customised within the AMASS project. The AoH encompasses up-to-date hardware as well as state of the art software to efficiently detect and acquire acoustic signatures of various underwater acoustically noisy maritime object.
Hardware
Set of single AoH consist of following units:
- Underwater antenna - suspended under the buoy at depth of 20 m:
i. 4 × NAXYS Ethernet hydrophones 02345 (EtH) regularly assembled on the frame.
ii. Honeywell HMR 3000 compass and inclinometer assembled on the frame.
iii. 5 × Underwater cables with connectors joining 5 sensors of the antenna with receiver.
iv. Mechanical parts: octagonal frame, ropes, shackles, coupler to fix underwater antenna to mooring rope of the buoy, roxtec sealing system to waterproof underwater cables from 5 sensors of the antenna to receiver.
- Receiver - assembled inside the buoy:
i. power supply converter transforming input voltage of 24 Voltage direct current (VDC) into 15 and 5 VDCs;
ii. 2 Field-programmable gate array (FPGA) boards performing I / Q detection and synchronisation of the EtHs signals;
iii. PC / 104 computing direction of arrival (DoA) Multiple signal classification (MUSIC) detection algorithms and Fast Fourier transform (FFT) spectrum, controlling AoH operation and communicating with C3;
iv. waterproof and pressure proof housing with 7 underwater connectors to 5 sensors of the antenna, power supply system of the bouy and C3.
Operational modes
The AoH operates at two modes:
- Detection - primary mode: narrowband signal processing at 375 Hz. The detection data is sent every 5 s from AoH to C3. It enables estimation of DoA and tracking of echo signal.
- Classification - auxiliary mode: broadband signal processing in frequency range from 3 to 3 000 Hz. The classification is performed on operator request.
Signal processing
The signal processing of the passive AoH is a multistage process. It encompasses operations performed by the individual sensors of the underwater antenna and state-of-the-art digital signal processing carried out at the receiver.
1. Hydrophone:
i. acoustic pressure to electric signal conversion: performed by the piezoelectric transduces;
ii. analogue signal processing: low pass anti-aliasing filtering, amplification;
iii. analogue to Ethernet frame conversion: Analog-to-digital converter (ADC), First in, first out (FIFO) memory, real time Ethernet controller.
2. Receiver
i. FPGA: generation of common reference digital signal, time stamps, decimation Hogenaur, filter, transmission of I / Q demodulation data to CPU (PC / 104 standard) via PCI.
ii. CPU (PC / 104 standard):
- reception of the hydroacoustic synchronised data from FPGA boards;
- reception of the antenna position data (heading, roll and pitch) from Compass and inclinometer (C&I);
- reception of the steering commands from C3: remote turn on / off, switching between detection and classification modes;
- calculation of DoA MUSIC based on narrowband hydroacoustic data;
- tracking of the detected target based on DoA results and C&I data;
- computing of acoustic signatures applying FFT for received wideband signal;
- auto-diagnostic;
- transmission of detection, tracking, classification and diagnostic data to C3.
3. Guidelines for AoH data processing at C3
i. Target location:
- single AoH: target heading and status of echo signal;
- multiple AoHs: calculation of target geographical position based on Global positioning system (GPS) data and Target motion analyses (TMA) providing to its distance from buoy, speed and course.
ii. Ship's acoustic classification:
- initial requirements for acquisition of acoustic signatures;
- creation and modification of data base including acoustic signatures of recognised maritime objects;
- real-time recognising of detected targets.
iii. Monitoring of AoH operational status.
Results of the sea trials - in brief
The sea trials of both AoHs have been executed in the Baltic Sea and in the Atlantic Ocean.
- The Baltic Sea: 6 sea trials were carried out between March 2010 and May 2011
i. Detection: AoHs correctly detected and determined the heading of a small motorboat.
ii. Classification: acquisition of the wideband hydroacoustic data work correctly.
2048 Real samples received from North hydrophone of the antenna
Calculated on the basis of real samples acoustic signature consisting of 1 024 spectral lines.
- The Atlantic Ocean
i. First AoH Spring 2011 - lack of the data, damage of two shackles after 6-week deployment.
ii. Second AoH 1-2 of September 2011: correct data.
Level of the Atlantic Ocean noise acquired between 1-2 September 2011. The stochastic process is not stationary.
Power MUSIC criterion at chosen 10-second time interval corresponding to data acquired at 4 o'clock on 2011-09-02. The strongest target is visible at heading of 220 degrees, whereas the weaker appeared at 80 degrees.
The acquired data are very interesting unless very reduced sea trials program of the AMASS project. Both AoHs detect correctly targets heading in the very shallow water of the Baltic Sea. Whereas the results obtained in the Atlantic Ocean are also efficient but associated with unknown targets. Additionally developed and implemented software tools enable detail analysis of AoH operation and data storage. Generally, the AoH efficiently detects various maritime directional targets even if the arriving to the underwater antenna signal is lower than surrounding sea noise level.
Communication module
Modelling, investigating and evaluating of the maritime radio channel
Based on the statistics of ocean wave / buoy movement, the parameters of the radio channel in terms of delay spread, fading and dependency on time were derived. Based on collected RX level date during a 2-week measuring campaign an analysis was performed regarding:
- Rx level vs. time performance and Rx level variation per time unit;
- fading depth and statistical distribution of Rx levels;
- maximum Rx level variation per time unit;
- relationship between sea wave heights and Rx level variation.
Hardware design
A state-of-the-art radio communication system was developed, manufactured and installed to provide a high capacity and long range uplink communication for transmitting video and other user data and a lower capacity downlink communication for user and control data. This point-to-multipoint radio communication system is able to provide asymmetrical data streams from up to 64 terminals to a base station using a high capacity uplink connection. Both transceivers (base and buoy) consist of two mainboards with the main functional units:
- radio frequency circuit (Rx / Tx switch for TDD-mode, DQPSK modulator / demodulator and power amplifier);
- signal processing unit (baseband-processing (FPGA), FEC);
- main controller;
- Ethernet controller (10 / 100 BaseT Ethernet interface);
- power supply.
Power management module
Energy system design
Fugro Oceanor designed a power system capable of handling a relatively large load combined with autonomy and long service intervals. The production and storage of energy have been central issues in the process. In light of the nature of the AMASS system, it was decided that the requirements will be met best by a hybrid system containing generator systems based on both renewable and stored energy. Photovoltaic systems and fuel cell generators stood out as the most promising options. A wind generator was also included in the system, in order to improve power production. For storage elements, using Sealed lead acid (SLA) batteries was deemed the most suitable solution. A 24 V system is preferred over a 12 V system, in order to keep the currents and losses at an acceptable level. The power system offers monitoring, control and protection of the energy flow between generators, storage elements and consuming elements.
The energy system was designed to have a redundant configuration with two separate battery banks, where the primary battery bank was continually replenished by the renewable energy sources of sun and wind. The secondary battery bank was replenished by direct methanol fuel cells installed in the buoy. Such cells need a battery bank as an intermediate storage, and the operational mode is to start producing energy from stored methanol at a predefined battery voltage, in order to keep the batteries charged.
Power control unit (PCU)
An electronics module was designed to control and supervise the energy system. The so-called PCU interfaces with the control system through a Transmission control protocol / Internet protocol (TCP / IP) network connection, and provides a software interface and the hardware architecture to control and monitor the system. Energy sources and consumers may be switched on and off at will and the current received from generators or consumed by users is monitored - the data being logged internally as well as reported to the control system.
The PCU also offers some autonomous features. It will automatically switch between the two battery banks depending on their states, a useful feature if the environmental conditions do not allow the primary bank to be replenished sufficiently fast. A safeguard is also in place to avoid discharging the battery banks below the level which would cause them damage - the system is reduced to a state with the minimal infrastructure running.
Energy simulator
ULPGC developed an energy simulator. The aim of this software is to evaluate the buoy circuit's autonomous capacity under different conditions. We have called it Autonomous energy system simulator (AESS).
This simulator gives us an idea of the energy balance and therefore the circuit viability and battery availability. The use of a Graphic user interface (GUI) and different wizards have simplified the user interaction. The simulator onion-like structure where the GUI surrounds the main simulator core and its algorithms has created a good platform for circuit test giving the facilities to create, to access and to load climate scenarios, circuit and power consumption scenario data files.
This simulator was developed under Matlab R2007b Linux version but it should work with minor graphical differences under the same Microsoft Windows version and newer versions. The main features of this simulator are:
- the possibility to create or load climatic scenarios that affect a defined circuit, simulating consumptions under determinate events;
- the ability to represent graphically any parameter versus battery state of charge (SOC) along the simulation time period;
- a wizard-guided data sourcing with a standardised Extensible markup language (XML) data file structure following the W3C's Document object model (DOM) standards;
- a flexible file format and internal design that will let users characterise new user-defined elements and scenarios;
- the possibility to induce elements faults;
- the ability to save and load results for a later visual comparison;
- a settable time step with a minimum resolution of 1 minute.
Optronics and image exploitation module
Fraunhofer-IOSB has developed the image exploitation subsystem for maritime surveillance and it was evaluated during the deployment of the buoy in Melenara Bay, Gran Canaria. Task of the image exploitation subsystem is to detect and track small distant vessels with a thermal imager (uncooled) with long focal length on a moderately stabilised autonomous platform.
Therefore, IOSB developed a robust multi-algorithm solution, exploiting complementary image cues and integrated them in flexible multi-layer software architecture. The image exploitation subsystem consists of the following hardware and software components.
Hardware:
- fanless, low power, image exploitation PC;
- low-cost IMU.
Command and control processing (CCP):
- receiving commands from land control station;
- sending of alarms to land control station;
- steering and managing of the optronics system;
- managing of the video exploitation algorithms.
Video exploitation system:
- low-cost IMU-based estimation of camera orientation;
-image-based horizon-line detection;
- fusion of IMU data, compass data and video exploitation;
- three different boat detection algorithms for IR sensor data;
- fusion of the boat detection results;
- scene / world-model generation;
- tracking algorithm;
- 2-stage Support vector machine (SVM)-based classification algorithms;
- alarm generation.
These components were developed to work under the conditions of the moderately stabilised optronics platform of the buoy.
Evaluation took place during deployment of the buoy in Melenara Bay. It was carried out up to 5 km distance with a rubber boot (6 m long with 4 persons). 50 GB of data were collected and processed.
It turned out that the actual movements of the AMASS optronics platform were up to a factor 10 larger than specified and expected. This resulted in challenging working conditions for the image exploitation subsystem. Nevertheless, also under these rough conditions, the image exploitation provided good results.
For detection and tracking-based alarm generation already without classification encouraging results could be obtained:
- Evaluation of 119 video clips with a length of 60 seconds each resulted in a 90.6 % true positive detection rate with a 14.3 % rate of false positive alarms. This includes the results of a detection range of 5 km, where also high-detection rates could be shown.
- These results were obtained with a mean observation time of the vessels of less than 3 seconds. The median observation time is approximately 1 second. In the field of main application of such a system much longer observation times than 3 seconds can be expected. This has the benefit that detection rates can be tuned towards even higher detection rates and lower false alarm rates in exchange for acceptable longer observation times (time until an alarm is generated).
The developed feature extraction and SVM-based classification showed further improvement of results, by sorting out false alarms. The number of false alarms (false positives) could be reduced by the classification step by a factor of 5.
Figures demonstrating detection results of the developed image exploitation software for the test campaign at Melenara Bay can be found in the file AM_RPT_D15_V1.0.PDF (also attached to this report).
Due to the stop of data acquisition caused by damage to the optronics hardware, no long time evaluation and tuning of the system could be carried out. Evaluation and tuning had to be undertaken with a smaller data set than planned. For higher generality and more appropriate training and test, more data should be processed (different scenes with different environmental conditions and different objects).
Still encouraging results for the automatic image exploitation for a volatile moving maritime autonomous platform were achieved. All components are developed and integrated. The developed algorithms are of general worth for moving maritime platforms (thin masts, vessels, maritime patrol aircrafts, and unmanned aerial vehicles) and should be further developed and used for many other applications.
Command and control centre software module
The C3 software application has been fully implemented in the original planned scope and additionally, the classification add-on was implemented, as described in the amendment no. 2.
Implementation of open interface protocol
The open interface protocols, which had been defined with the other partners have been fully implemented as planned. During the project there were new protocols proposed, which bring additional features and value to the AMASS solution.
The protocols which were defined from the early phase of the project have been fully implemented and intensively tested via Internet, in the laboratory tests and during the deployment phase. The communication runs smoothly without any real problems and during the test deployment phase a lot of useful data was gathered for further analysis.
Communication and control for image exploitation
The control for image exploitation is one of the OIP which was defined in the project early phase. The remote control of the image exploitation has been also tested via Internet and very intensively during laboratory testing in May 2010 by ICCM at Gran Canaria. All aspects of the communication and remote control of the image exploitation worked smoothly, the C3 was able to run the camera movement, to gather the images.
Platform simulator
The C3 simulator was implemented in 2 phases:
- version 1 was developed for the testing of the communication protocols;
- version 2 was developed to support the deployment test phase, where it started to be necessary to simulate some components of the AMASS solution to be able to run extended tests with the overall AMASS solution.
Both of these goals were achieved with success, a video of the full AMASS solution simulation in action is available.
Map visualisation
After an initial research about available technologies, the development group decided to use available open source toolkit Java Geo Toolkit and as a data source for map data the open standard GeoTIFF. The solution is capable of working with different types of defined coordination system, it is capable of transforming the data into another coordination system, which is useful for communication with other third party systems, which requires different coordinates format. The visualisation components are capable of working with map data in an amount of gigabytes, without any visible influence on usability.
The component responsible for visualisation in the C3 client is capable not only to present the map data but also the array of buoys, target, its paths, status of the buoys, and an operator can draw areas into the map. The component capabilities are again available to see in the simulation video.
Classification add-on
The classification add-on was developed in the cooperation with CTM. The additional classification communication protocol was designed, implemented and tested during the deployment test phase. Storing of the gathered patterns into the pattern library was implemented. This data are provided for the rest of the AMASS system afterwards. One of the direct usages of this pattern library is the C3 client, which represents the patterns visually to the AMASS operator.
Secure data access
The mechanism for preventing access to the data based on defined rules was fully implemented. This requirement is mostly based on a need not to show data older than a defined time. This mechanism was implemented in the C3 server and also in the C3 client, where the operator receives the information that the data on the old case is not available any more.
Classification add-on
The classification add-on was developed in the cooperation with CTM. The additional classification communication protocol was designed, implemented and tested during the deployment test phase. Storing of the gathered patterns into the pattern library was implemented. This data are provided for the rest of the AMASS system afterwards. One of the direct usages of this pattern library is the C3 client, which represents the patterns visually to the AMASS operator.
- implementation of open interface protocol;
- communication and control for image exploitation;
- platform simulator;
- map visualisation;
- pattern library;
- sound recognition;
- neural networks;
- secure data access.
Test campaigns
Images gathered during the test campaign provide an impression of the deployment and testing of the system.
Potential impact:
The AMASS project results are expected to lead towards a 24 / 7 security surveillance solution to be used by maritime border agencies providing the early, accurate warnings that they need about small boats entering or already in their waters. In addition, the sensors are expected to a 360-degree view of the area above water - significantly improving situational awareness for coast patrols. The platforms should also remain fully functional in all weather conditions.
AMASS should be also significantly more economical to operate than patrol vessels and will free up human resources for other tasks - providing an all-round more cost efficient solution. AMASS should, most importantly, help border agencies protect their own personnel and save the lives of people on unsafe small boats. This should lead to safer and more secure European borders.
List of websites: http://www.amass-project.eu
The concept of the AMASS project was to develop a surveillance system for the observation and security of wide critical maritime areas in order to reduce actual and potential illegal immigration and the trafficking of drugs, weapons and illicit substances.
The objectives of the project were to carry out the key research and technological development work which would lead to an engineered product following the completion of the project comprising of a network of unmanned platforms located a considerable distance from shore.
The project has successfully designed a complete system consisting of a number of integrated modules, including: flotation platform, optronics, hydrophones, communications, power management, image exploitation and command and control. Initial test and evaluation as carried out in the Atlantic Ocean off Gran Canaria and these together with further simulation have proved the concept of the system.
Project context and objectives:
The concept of the AMASS project was to develop a surveillance system for the observation and security of wide critical maritime areas in order to reduce actual and potential illegal immigration and the trafficking of drugs, weapons and illicit substances.
The objectives of the project were to carry out the key research and technological development work which would lead to an engineered product following the completion of the project comprising of a network of unmanned platforms located a considerable distance from shore. The platforms had to be designed to detect small boats both during the day and at night. Each platform was designed to be fitted with cutting-edge sensors and to operate self-sufficiently, i.e. without the need for manual intervention. Data captured by the sensors is transmitted to a central command centre, where an operator views it on screen. If a suspicious entity is detected, a crew can be dispatched to investigate.
Project results:
Description of main Science and technology (S&T) results / foregrounds
An overall system was developed which consisted of a number of key modules and their integration, including: flotation platform, optronics, hydrophones, communications, power management, image exploitation and command and control. The following section illustrates the buoy concept and realisation. The next sections present individual modules and the final section shows different buoys at sea during tests.
Autonomous maritime surveillance system (AMASS prototype)
Knowledge of end user requirements for autonomous maritime surveillance system
The project included actual end users and organisation with close contacts to end users. The initial phase of the project collected and documented the end user requirements for an autonomous maritime surveillance system.
Overall system engineering
The system was designed, developed, integrated and deployed.
Optronic subsystem
Carl Zeiss Optronics developed the optronic subsystem comprising of an uncooled thermal imaging camera module (UCM) with very low power consumption mounted on a stabilised platform. The camera module itself is sealed and uses the specially coated front lens as a window. Thus it can withstand the harsh environmental conditions at sea without an additional housing being necessary which would have put more weight on the top of the platform. Based on the statistics of ocean waves and buoy movement and a combination of passive and active means a stabilisation level of the camera was achieved in 3 axis (pitch, roll and yaw) that constitutes a good basis for the following image stabilisation.
Uncooled camera module UCM with automated athermalisation
The requirements of low-power consumption, low-weight and long-life time prohibit the use of a thermal imaging camera equipped with a cooled infra-red detector because the necessary cooling engine would impose narrow limits in regard to all three requirements. Instead microbolometer detectors are the best solution not only fulfilling the requirements stated above but also featuring high sensitivity in the Long-wave infrared (LWIR) spectral band comparable to what was state-of-the-art at cooled second generation detectors only a few years ago. The design of the optical concept was backed up by a range modelling using the software package TRM2 which is accepted for range calculation worldwide. Based on this concept a compact and robust camera module was designed and tested.
Virtually all LWIR objectives are using germanium as a substrate material for lens elements because of its favourable optical properties (extremely high-refractive index n equal to 4, high transmission and good machining properties). The only shortcoming is that the optical power of any germanium lens is strongly depending on temperature resulting in a significant defocusing of the camera. Therefore a procedure for compensating this deviation must be implemented which is commonly known as athermalisation. In general, temperature may vary from one lens element to the next and also may not be uniform over the mechanical structure. Therefore, an auto-collimation method was developed as an effective means for refocusing the camera instead of a simple shifting mechanism of the lens. The infrared radiation emitted by a reference target is collimated by the infrared objective and after being partially reflected back by a plane mirror travels through the same objective a second time in the reverse direction and is then imaged onto the detector surface. Thus a correct focusing state is indicated by a sharply focused image of the reference target which is assessed by a fast software algorithm.
2-stage stabilisation unit for UCM
Analysis of the buoy movement caused by the ocean waves shows that large angles appear in pitch and roll. In order to compensate this movement at least partially without employing powerful electric motors which would consume more electrical power than the buoy possibly could deliver a passive stabilisation means was developed. The basic element of the passive unit is a mass pendulum fixed in a gimbal frame mounting beneath the optronic platform. To reduce the oscillations of the camera's line-of-sight to an acceptable amount in all four directions effective spring-damper combinations are installed. By careful theoretic modelling and intensive testing on a 6-axis motion platform all parameters relevant for damping the oscillations (length of pendulum, mass of balance weight, spring and damper parameters and position) were optimised using the motion data for pitch, roll and heave obtained in the first phase of the project.
Furthermore, active stabilisation is added by integrating a stabilised pan-tilt unit PTU with integrated Inertial measurement unit (IMU) between sensor platform and the pendulum in order to:
- allow panning of the sensor platform;
- achieve stabilisation of the line of sight also in regard to yaw motions;
- as a fine stabilisation means along the elevation axis of the camera.
All together, the introduced 2-stage stabilisation unit is capable to attenuate the ocean wave induced oscillations of the camera's line-of-sight by a factor of almost 5. As most of the stabilisation work is carried out passively the power consumption is significant lower than that of purely active systems.
Array of hydrophone (AoH) module
Two units of AoH were customised within the AMASS project. The AoH encompasses up-to-date hardware as well as state of the art software to efficiently detect and acquire acoustic signatures of various underwater acoustically noisy maritime object.
Hardware
Set of single AoH consist of following units:
- Underwater antenna - suspended under the buoy at depth of 20 m:
i. 4 × NAXYS Ethernet hydrophones 02345 (EtH) regularly assembled on the frame.
ii. Honeywell HMR 3000 compass and inclinometer assembled on the frame.
iii. 5 × Underwater cables with connectors joining 5 sensors of the antenna with receiver.
iv. Mechanical parts: octagonal frame, ropes, shackles, coupler to fix underwater antenna to mooring rope of the buoy, roxtec sealing system to waterproof underwater cables from 5 sensors of the antenna to receiver.
- Receiver - assembled inside the buoy:
i. power supply converter transforming input voltage of 24 Voltage direct current (VDC) into 15 and 5 VDCs;
ii. 2 Field-programmable gate array (FPGA) boards performing I / Q detection and synchronisation of the EtHs signals;
iii. PC / 104 computing direction of arrival (DoA) Multiple signal classification (MUSIC) detection algorithms and Fast Fourier transform (FFT) spectrum, controlling AoH operation and communicating with C3;
iv. waterproof and pressure proof housing with 7 underwater connectors to 5 sensors of the antenna, power supply system of the bouy and C3.
Operational modes
The AoH operates at two modes:
- Detection - primary mode: narrowband signal processing at 375 Hz. The detection data is sent every 5 s from AoH to C3. It enables estimation of DoA and tracking of echo signal.
- Classification - auxiliary mode: broadband signal processing in frequency range from 3 to 3 000 Hz. The classification is performed on operator request.
Signal processing
The signal processing of the passive AoH is a multistage process. It encompasses operations performed by the individual sensors of the underwater antenna and state-of-the-art digital signal processing carried out at the receiver.
1. Hydrophone:
i. acoustic pressure to electric signal conversion: performed by the piezoelectric transduces;
ii. analogue signal processing: low pass anti-aliasing filtering, amplification;
iii. analogue to Ethernet frame conversion: Analog-to-digital converter (ADC), First in, first out (FIFO) memory, real time Ethernet controller.
2. Receiver
i. FPGA: generation of common reference digital signal, time stamps, decimation Hogenaur, filter, transmission of I / Q demodulation data to CPU (PC / 104 standard) via PCI.
ii. CPU (PC / 104 standard):
- reception of the hydroacoustic synchronised data from FPGA boards;
- reception of the antenna position data (heading, roll and pitch) from Compass and inclinometer (C&I);
- reception of the steering commands from C3: remote turn on / off, switching between detection and classification modes;
- calculation of DoA MUSIC based on narrowband hydroacoustic data;
- tracking of the detected target based on DoA results and C&I data;
- computing of acoustic signatures applying FFT for received wideband signal;
- auto-diagnostic;
- transmission of detection, tracking, classification and diagnostic data to C3.
3. Guidelines for AoH data processing at C3
i. Target location:
- single AoH: target heading and status of echo signal;
- multiple AoHs: calculation of target geographical position based on Global positioning system (GPS) data and Target motion analyses (TMA) providing to its distance from buoy, speed and course.
ii. Ship's acoustic classification:
- initial requirements for acquisition of acoustic signatures;
- creation and modification of data base including acoustic signatures of recognised maritime objects;
- real-time recognising of detected targets.
iii. Monitoring of AoH operational status.
Results of the sea trials - in brief
The sea trials of both AoHs have been executed in the Baltic Sea and in the Atlantic Ocean.
- The Baltic Sea: 6 sea trials were carried out between March 2010 and May 2011
i. Detection: AoHs correctly detected and determined the heading of a small motorboat.
ii. Classification: acquisition of the wideband hydroacoustic data work correctly.
2048 Real samples received from North hydrophone of the antenna
Calculated on the basis of real samples acoustic signature consisting of 1 024 spectral lines.
- The Atlantic Ocean
i. First AoH Spring 2011 - lack of the data, damage of two shackles after 6-week deployment.
ii. Second AoH 1-2 of September 2011: correct data.
Level of the Atlantic Ocean noise acquired between 1-2 September 2011. The stochastic process is not stationary.
Power MUSIC criterion at chosen 10-second time interval corresponding to data acquired at 4 o'clock on 2011-09-02. The strongest target is visible at heading of 220 degrees, whereas the weaker appeared at 80 degrees.
The acquired data are very interesting unless very reduced sea trials program of the AMASS project. Both AoHs detect correctly targets heading in the very shallow water of the Baltic Sea. Whereas the results obtained in the Atlantic Ocean are also efficient but associated with unknown targets. Additionally developed and implemented software tools enable detail analysis of AoH operation and data storage. Generally, the AoH efficiently detects various maritime directional targets even if the arriving to the underwater antenna signal is lower than surrounding sea noise level.
Communication module
Modelling, investigating and evaluating of the maritime radio channel
Based on the statistics of ocean wave / buoy movement, the parameters of the radio channel in terms of delay spread, fading and dependency on time were derived. Based on collected RX level date during a 2-week measuring campaign an analysis was performed regarding:
- Rx level vs. time performance and Rx level variation per time unit;
- fading depth and statistical distribution of Rx levels;
- maximum Rx level variation per time unit;
- relationship between sea wave heights and Rx level variation.
Hardware design
A state-of-the-art radio communication system was developed, manufactured and installed to provide a high capacity and long range uplink communication for transmitting video and other user data and a lower capacity downlink communication for user and control data. This point-to-multipoint radio communication system is able to provide asymmetrical data streams from up to 64 terminals to a base station using a high capacity uplink connection. Both transceivers (base and buoy) consist of two mainboards with the main functional units:
- radio frequency circuit (Rx / Tx switch for TDD-mode, DQPSK modulator / demodulator and power amplifier);
- signal processing unit (baseband-processing (FPGA), FEC);
- main controller;
- Ethernet controller (10 / 100 BaseT Ethernet interface);
- power supply.
Power management module
Energy system design
Fugro Oceanor designed a power system capable of handling a relatively large load combined with autonomy and long service intervals. The production and storage of energy have been central issues in the process. In light of the nature of the AMASS system, it was decided that the requirements will be met best by a hybrid system containing generator systems based on both renewable and stored energy. Photovoltaic systems and fuel cell generators stood out as the most promising options. A wind generator was also included in the system, in order to improve power production. For storage elements, using Sealed lead acid (SLA) batteries was deemed the most suitable solution. A 24 V system is preferred over a 12 V system, in order to keep the currents and losses at an acceptable level. The power system offers monitoring, control and protection of the energy flow between generators, storage elements and consuming elements.
The energy system was designed to have a redundant configuration with two separate battery banks, where the primary battery bank was continually replenished by the renewable energy sources of sun and wind. The secondary battery bank was replenished by direct methanol fuel cells installed in the buoy. Such cells need a battery bank as an intermediate storage, and the operational mode is to start producing energy from stored methanol at a predefined battery voltage, in order to keep the batteries charged.
Power control unit (PCU)
An electronics module was designed to control and supervise the energy system. The so-called PCU interfaces with the control system through a Transmission control protocol / Internet protocol (TCP / IP) network connection, and provides a software interface and the hardware architecture to control and monitor the system. Energy sources and consumers may be switched on and off at will and the current received from generators or consumed by users is monitored - the data being logged internally as well as reported to the control system.
The PCU also offers some autonomous features. It will automatically switch between the two battery banks depending on their states, a useful feature if the environmental conditions do not allow the primary bank to be replenished sufficiently fast. A safeguard is also in place to avoid discharging the battery banks below the level which would cause them damage - the system is reduced to a state with the minimal infrastructure running.
Energy simulator
ULPGC developed an energy simulator. The aim of this software is to evaluate the buoy circuit's autonomous capacity under different conditions. We have called it Autonomous energy system simulator (AESS).
This simulator gives us an idea of the energy balance and therefore the circuit viability and battery availability. The use of a Graphic user interface (GUI) and different wizards have simplified the user interaction. The simulator onion-like structure where the GUI surrounds the main simulator core and its algorithms has created a good platform for circuit test giving the facilities to create, to access and to load climate scenarios, circuit and power consumption scenario data files.
This simulator was developed under Matlab R2007b Linux version but it should work with minor graphical differences under the same Microsoft Windows version and newer versions. The main features of this simulator are:
- the possibility to create or load climatic scenarios that affect a defined circuit, simulating consumptions under determinate events;
- the ability to represent graphically any parameter versus battery state of charge (SOC) along the simulation time period;
- a wizard-guided data sourcing with a standardised Extensible markup language (XML) data file structure following the W3C's Document object model (DOM) standards;
- a flexible file format and internal design that will let users characterise new user-defined elements and scenarios;
- the possibility to induce elements faults;
- the ability to save and load results for a later visual comparison;
- a settable time step with a minimum resolution of 1 minute.
Optronics and image exploitation module
Fraunhofer-IOSB has developed the image exploitation subsystem for maritime surveillance and it was evaluated during the deployment of the buoy in Melenara Bay, Gran Canaria. Task of the image exploitation subsystem is to detect and track small distant vessels with a thermal imager (uncooled) with long focal length on a moderately stabilised autonomous platform.
Therefore, IOSB developed a robust multi-algorithm solution, exploiting complementary image cues and integrated them in flexible multi-layer software architecture. The image exploitation subsystem consists of the following hardware and software components.
Hardware:
- fanless, low power, image exploitation PC;
- low-cost IMU.
Command and control processing (CCP):
- receiving commands from land control station;
- sending of alarms to land control station;
- steering and managing of the optronics system;
- managing of the video exploitation algorithms.
Video exploitation system:
- low-cost IMU-based estimation of camera orientation;
-image-based horizon-line detection;
- fusion of IMU data, compass data and video exploitation;
- three different boat detection algorithms for IR sensor data;
- fusion of the boat detection results;
- scene / world-model generation;
- tracking algorithm;
- 2-stage Support vector machine (SVM)-based classification algorithms;
- alarm generation.
These components were developed to work under the conditions of the moderately stabilised optronics platform of the buoy.
Evaluation took place during deployment of the buoy in Melenara Bay. It was carried out up to 5 km distance with a rubber boot (6 m long with 4 persons). 50 GB of data were collected and processed.
It turned out that the actual movements of the AMASS optronics platform were up to a factor 10 larger than specified and expected. This resulted in challenging working conditions for the image exploitation subsystem. Nevertheless, also under these rough conditions, the image exploitation provided good results.
For detection and tracking-based alarm generation already without classification encouraging results could be obtained:
- Evaluation of 119 video clips with a length of 60 seconds each resulted in a 90.6 % true positive detection rate with a 14.3 % rate of false positive alarms. This includes the results of a detection range of 5 km, where also high-detection rates could be shown.
- These results were obtained with a mean observation time of the vessels of less than 3 seconds. The median observation time is approximately 1 second. In the field of main application of such a system much longer observation times than 3 seconds can be expected. This has the benefit that detection rates can be tuned towards even higher detection rates and lower false alarm rates in exchange for acceptable longer observation times (time until an alarm is generated).
The developed feature extraction and SVM-based classification showed further improvement of results, by sorting out false alarms. The number of false alarms (false positives) could be reduced by the classification step by a factor of 5.
Figures demonstrating detection results of the developed image exploitation software for the test campaign at Melenara Bay can be found in the file AM_RPT_D15_V1.0.PDF (also attached to this report).
Due to the stop of data acquisition caused by damage to the optronics hardware, no long time evaluation and tuning of the system could be carried out. Evaluation and tuning had to be undertaken with a smaller data set than planned. For higher generality and more appropriate training and test, more data should be processed (different scenes with different environmental conditions and different objects).
Still encouraging results for the automatic image exploitation for a volatile moving maritime autonomous platform were achieved. All components are developed and integrated. The developed algorithms are of general worth for moving maritime platforms (thin masts, vessels, maritime patrol aircrafts, and unmanned aerial vehicles) and should be further developed and used for many other applications.
Command and control centre software module
The C3 software application has been fully implemented in the original planned scope and additionally, the classification add-on was implemented, as described in the amendment no. 2.
Implementation of open interface protocol
The open interface protocols, which had been defined with the other partners have been fully implemented as planned. During the project there were new protocols proposed, which bring additional features and value to the AMASS solution.
The protocols which were defined from the early phase of the project have been fully implemented and intensively tested via Internet, in the laboratory tests and during the deployment phase. The communication runs smoothly without any real problems and during the test deployment phase a lot of useful data was gathered for further analysis.
Communication and control for image exploitation
The control for image exploitation is one of the OIP which was defined in the project early phase. The remote control of the image exploitation has been also tested via Internet and very intensively during laboratory testing in May 2010 by ICCM at Gran Canaria. All aspects of the communication and remote control of the image exploitation worked smoothly, the C3 was able to run the camera movement, to gather the images.
Platform simulator
The C3 simulator was implemented in 2 phases:
- version 1 was developed for the testing of the communication protocols;
- version 2 was developed to support the deployment test phase, where it started to be necessary to simulate some components of the AMASS solution to be able to run extended tests with the overall AMASS solution.
Both of these goals were achieved with success, a video of the full AMASS solution simulation in action is available.
Map visualisation
After an initial research about available technologies, the development group decided to use available open source toolkit Java Geo Toolkit and as a data source for map data the open standard GeoTIFF. The solution is capable of working with different types of defined coordination system, it is capable of transforming the data into another coordination system, which is useful for communication with other third party systems, which requires different coordinates format. The visualisation components are capable of working with map data in an amount of gigabytes, without any visible influence on usability.
The component responsible for visualisation in the C3 client is capable not only to present the map data but also the array of buoys, target, its paths, status of the buoys, and an operator can draw areas into the map. The component capabilities are again available to see in the simulation video.
Classification add-on
The classification add-on was developed in the cooperation with CTM. The additional classification communication protocol was designed, implemented and tested during the deployment test phase. Storing of the gathered patterns into the pattern library was implemented. This data are provided for the rest of the AMASS system afterwards. One of the direct usages of this pattern library is the C3 client, which represents the patterns visually to the AMASS operator.
Secure data access
The mechanism for preventing access to the data based on defined rules was fully implemented. This requirement is mostly based on a need not to show data older than a defined time. This mechanism was implemented in the C3 server and also in the C3 client, where the operator receives the information that the data on the old case is not available any more.
Classification add-on
The classification add-on was developed in the cooperation with CTM. The additional classification communication protocol was designed, implemented and tested during the deployment test phase. Storing of the gathered patterns into the pattern library was implemented. This data are provided for the rest of the AMASS system afterwards. One of the direct usages of this pattern library is the C3 client, which represents the patterns visually to the AMASS operator.
- implementation of open interface protocol;
- communication and control for image exploitation;
- platform simulator;
- map visualisation;
- pattern library;
- sound recognition;
- neural networks;
- secure data access.
Test campaigns
Images gathered during the test campaign provide an impression of the deployment and testing of the system.
Potential impact:
The AMASS project results are expected to lead towards a 24 / 7 security surveillance solution to be used by maritime border agencies providing the early, accurate warnings that they need about small boats entering or already in their waters. In addition, the sensors are expected to a 360-degree view of the area above water - significantly improving situational awareness for coast patrols. The platforms should also remain fully functional in all weather conditions.
AMASS should be also significantly more economical to operate than patrol vessels and will free up human resources for other tasks - providing an all-round more cost efficient solution. AMASS should, most importantly, help border agencies protect their own personnel and save the lives of people on unsafe small boats. This should lead to safer and more secure European borders.
List of websites: http://www.amass-project.eu
