DeeperSense addressed key capabilities for cognitive robotic systems, in particular the perception and interpretation of a robot's environment. The main objective was to apply state-of-the-art artificial intelligence and machine learning to significantly improve these capabilities and thereby to both enhance the performance and reliability of robotic systems, enable new functionality, and open up new application areas for cognitive robotic systems.
In technical terms, DeeperSense applied Artificial Neural Networks (ANNs), data-driven Machine Learning (ML) / Deep Learning (DL) and Generative AI to fuse the specific capabilities of different sensor modalities for better enviroment perception. In this approach, the ANNs connect sensors that use completely different physical principles to probe the space around a robot. When fed with sufficient training data, the ANNs can "learn" how to match and combine the outputs from the different sensors. With methods based on Generative AI, synthetic outputs related to one sensor type can be created on the basis of another sensor. This way, a blurry sonar image can be transformed into a sharp camera image, the sighting of an obstacle that is barely visible in the distance can be confirmed by a sonar reading, or sediment textures and plant coverage on the bottom of the sea can be reliably classfied based on low resolution sonar scans calibrated with high-definition camera images.
In principle, the DeeperSense concept can be applied to robotic sensing in any environment or medium. The underwater use-cases were chosen because underwater perception is one of the most challenging perception tasks for robots.
Furthermore, the three use cases were selected due to their significant societal and environmental relevance and their impact on concrete end-user and market needs. For each use case, one specific algorithm was developed, trained, and verified.
- For UC1, the SONAVision algorithm enabled the monitoring and securing of professional divers under low-visibility conditions, for example when working on the inspection and maintenance of critical infrastructures;
- For UC2, the EagleEye algorithm enhanced the forward looking and obstacle detection capabilities of autonomous underwater vehicles, for example when operating in complex underwater structures such as coral reefs;
- For UC3, the SmartSeabottomScan algorithm supported the creation of high-resolution maps of the marine sea-floor, including the precise classification of sediments and life forms, based on data from Side Scan Sonars.
All three algorithms were tested and verified in real-world environments. This included tests in lakes in Germany, in both the eastern and the western Mediterranean, and in the Red Sea.
A final demonstration was organized in Lake Starnberg, Germany.
The DeeperSense project team comprised researchers, technology providers and end-users bundled in three clusters in Germany, Israel, and Spain. Each cluster tackled one of
the use-cases described above. However, to optimize the project outcome, the researcher groups in the clusters shared their know-how and, even more importantly, the training data they collected in numerous lab- and field campaigns. The cooperation was supported by a technical infrastructure
for data and knowledge sharing. In addition to the algorithms described in scientific papers and publications, selected parts of the training data collected in the project were made available to the scientific community by publishing them on the EU ZENODO platform.
