Periodic Reporting for period 2 - DeeperSense (Deep-Learning for Multimodal Sensor Fusion)
Periodo di rendicontazione: 2022-07-01 al 2023-12-31
DeeperSense addressed key capabilities for cognitive robotic systems, in particular the perception and interpretation of a robot's environment. The main objective was to apply state-of-the-art artificial intelligence and machine learning to significantly improve these capabilities and thereby to both enhance the performance and reliability of robotic systems, enable new functionality, and open up new application areas for cognitive robotic systems.
In technical terms, DeeperSense applied Artificial Neural Networks (ANNs), data-driven Machine Learning (ML) / Deep Learning (DL) and Generative AI to fuse the specific capabilities of different sensor modalities for better enviroment perception. In this approach, the ANNs connect sensors that use completely different physical principles to probe the space around a robot. When fed with sufficient training data, the ANNs can "learn" how to match and combine the outputs from the different sensors. With methods based on Generative AI, synthetic outputs related to one sensor type can be created on the basis of another sensor. This way, a blurry sonar image can be transformed into a sharp camera image, the sighting of an obstacle that is barely visible in the distance can be confirmed by a sonar reading, or sediment textures and plant coverage on the bottom of the sea can be reliably classfied based on low resolution sonar scans calibrated with high-definition camera images.
In principle, the DeeperSense concept can be applied to robotic sensing in any environment or medium. The underwater use-cases were chosen because underwater perception is one of the most challenging perception tasks for robots.
Furthermore, the three use cases were selected due to their significant societal and environmental relevance and their impact on concrete end-user and market needs. For each use case, one specific algorithm was developed, trained, and verified.
- For UC1, the SONAVision algorithm enabled the monitoring and securing of professional divers under low-visibility conditions, for example when working on the inspection and maintenance of critical infrastructures;
- For UC2, the EagleEye algorithm enhanced the forward looking and obstacle detection capabilities of autonomous underwater vehicles, for example when operating in complex underwater structures such as coral reefs;
- For UC3, the SmartSeabottomScan algorithm supported the creation of high-resolution maps of the marine sea-floor, including the precise classification of sediments and life forms, based on data from Side Scan Sonars.
All three algorithms were tested and verified in real-world environments. This included tests in lakes in Germany, in both the eastern and the western Mediterranean, and in the Red Sea.
A final demonstration was organized in Lake Starnberg, Germany.
The DeeperSense project team comprised researchers, technology providers and end-users bundled in three clusters in Germany, Israel, and Spain. Each cluster tackled one of
the use-cases described above. However, to optimize the project outcome, the researcher groups in the clusters shared their know-how and, even more importantly, the training data they collected in numerous lab- and field campaigns. The cooperation was supported by a technical infrastructure
for data and knowledge sharing. In addition to the algorithms described in scientific papers and publications, selected parts of the training data collected in the project were made available to the scientific community by publishing them on the EU ZENODO platform.
In technical terms, DeeperSense applied Artificial Neural Networks (ANNs), data-driven Machine Learning (ML) / Deep Learning (DL) and Generative AI to fuse the specific capabilities of different sensor modalities for better enviroment perception. In this approach, the ANNs connect sensors that use completely different physical principles to probe the space around a robot. When fed with sufficient training data, the ANNs can "learn" how to match and combine the outputs from the different sensors. With methods based on Generative AI, synthetic outputs related to one sensor type can be created on the basis of another sensor. This way, a blurry sonar image can be transformed into a sharp camera image, the sighting of an obstacle that is barely visible in the distance can be confirmed by a sonar reading, or sediment textures and plant coverage on the bottom of the sea can be reliably classfied based on low resolution sonar scans calibrated with high-definition camera images.
In principle, the DeeperSense concept can be applied to robotic sensing in any environment or medium. The underwater use-cases were chosen because underwater perception is one of the most challenging perception tasks for robots.
Furthermore, the three use cases were selected due to their significant societal and environmental relevance and their impact on concrete end-user and market needs. For each use case, one specific algorithm was developed, trained, and verified.
- For UC1, the SONAVision algorithm enabled the monitoring and securing of professional divers under low-visibility conditions, for example when working on the inspection and maintenance of critical infrastructures;
- For UC2, the EagleEye algorithm enhanced the forward looking and obstacle detection capabilities of autonomous underwater vehicles, for example when operating in complex underwater structures such as coral reefs;
- For UC3, the SmartSeabottomScan algorithm supported the creation of high-resolution maps of the marine sea-floor, including the precise classification of sediments and life forms, based on data from Side Scan Sonars.
All three algorithms were tested and verified in real-world environments. This included tests in lakes in Germany, in both the eastern and the western Mediterranean, and in the Red Sea.
A final demonstration was organized in Lake Starnberg, Germany.
The DeeperSense project team comprised researchers, technology providers and end-users bundled in three clusters in Germany, Israel, and Spain. Each cluster tackled one of
the use-cases described above. However, to optimize the project outcome, the researcher groups in the clusters shared their know-how and, even more importantly, the training data they collected in numerous lab- and field campaigns. The cooperation was supported by a technical infrastructure
for data and knowledge sharing. In addition to the algorithms described in scientific papers and publications, selected parts of the training data collected in the project were made available to the scientific community by publishing them on the EU ZENODO platform.
The technology-provider / end-user sub-teams in the 3 participating countries (Germany, Spain, Israel) gathered detailed user-requirements for the 3 application use-cases in DeeperSense, i.e. “Diver Monitoring”, “AUV Navigation in Coral Reefs”, and “Seabed Mapping & Classification”.
These requirements were then used to select the tools needed to implement the DeeperSense concept, i.e. meaningful pairings of sensor modalities and artificial neural network (ANN) topologies with the potential to enable the envisioned ML and DL solutions.
Three algorithms, “SONAVision”, “EagleEye” and “SmartSeabottomScan” were designed to cope with the requirements of the 3 use-cases. For the training of the algorithms, data from legacy data sources and simulation tools, but mainly from extended data collection campaigns in the lab and various field locations, were used.
For the data acquisition campaigns, several tools had to be modified and new tools had to be conceptualized and built.
Data collection was supported by the end-users in the three clusters. THW, for example, mobilized up to 10 professional divers for several multi-day campaigns in the DFKI maritime exploration facility and in natural lakes near Bremen.
In Israel, data were collected both in the Mediterranean near Haifa and in the Red Sea near Eilat. For the Spanish use case, extensive data collection campaigns off the coast of Catalunya were conducted with the support of TA.
By the end of the project, the three AI algorithms were tested and validated in the lab and in real outdoor environments. The underwater perception data collected in DeeperSense are of considerable value to the scientific community. To share this outcome of the DeeperSense project with other roboticists in Europe, we published selected datasets on the EU supported ZENODO data portal, ase well as the code of the AI algorithms (via git-hub under an open-source licensing agreement).
The DeeperSense approach and scientific results, including data sets, were published in scientific journals and presented on scientific conferences. A workshop and tutorial was organized at the 2023 Breaking-the-surface international symposium.
These requirements were then used to select the tools needed to implement the DeeperSense concept, i.e. meaningful pairings of sensor modalities and artificial neural network (ANN) topologies with the potential to enable the envisioned ML and DL solutions.
Three algorithms, “SONAVision”, “EagleEye” and “SmartSeabottomScan” were designed to cope with the requirements of the 3 use-cases. For the training of the algorithms, data from legacy data sources and simulation tools, but mainly from extended data collection campaigns in the lab and various field locations, were used.
For the data acquisition campaigns, several tools had to be modified and new tools had to be conceptualized and built.
Data collection was supported by the end-users in the three clusters. THW, for example, mobilized up to 10 professional divers for several multi-day campaigns in the DFKI maritime exploration facility and in natural lakes near Bremen.
In Israel, data were collected both in the Mediterranean near Haifa and in the Red Sea near Eilat. For the Spanish use case, extensive data collection campaigns off the coast of Catalunya were conducted with the support of TA.
By the end of the project, the three AI algorithms were tested and validated in the lab and in real outdoor environments. The underwater perception data collected in DeeperSense are of considerable value to the scientific community. To share this outcome of the DeeperSense project with other roboticists in Europe, we published selected datasets on the EU supported ZENODO data portal, ase well as the code of the AI algorithms (via git-hub under an open-source licensing agreement).
The DeeperSense approach and scientific results, including data sets, were published in scientific journals and presented on scientific conferences. A workshop and tutorial was organized at the 2023 Breaking-the-surface international symposium.
DeeperSense pushed he current SoA in robotic perception, using examples in sub-sea applications, by using AI and ML methods to enable “inter-sensoric learning” in a smart fusion of data from different (visual, acoustic) sensor modalities.
The outcome of the project is a set of AI algorithms that were trained and verified for three specific use-cases, namely diver monitoring, undewater navigation, and seabed classification. After the project, the DeeperSense application partners, i.e. THW in Germany, the Israel National Park Agency in Israel, and Tecnoambiente S.A. in Spain will be able to use these algorithms to improve their operations.
DeeperSense used the example of the maritime and underwater domain to develop AI-enhanced robotic perception solutions. However, the DeeperSense concept of AI-based multi-modal sensor fusion can be generalized to other application domains. The algorithms can be adapted and re-trained to other sensor modalities and application use cases. Thus the DeeperSense methodology is expected to become the basis for further R&D projects and industry-driven product development both in maritime and terrestrial or space applications. This includes the possiblitiy to lower the cost for robotic perception with DeeperSense, i.e. by enhancing the capabilities of low-cost sensors with AI-based software.
DeeperSense was committed to an open-data and open-code policy. Thus selections of both the algorithms and the training data collected in the project were made publicly available.
This was part of the communication and dissemination strategy of DeeperSense, which had the objective to support the European robotics and AI communities and thus strengthen European science and technology.
The outcome of the project is a set of AI algorithms that were trained and verified for three specific use-cases, namely diver monitoring, undewater navigation, and seabed classification. After the project, the DeeperSense application partners, i.e. THW in Germany, the Israel National Park Agency in Israel, and Tecnoambiente S.A. in Spain will be able to use these algorithms to improve their operations.
DeeperSense used the example of the maritime and underwater domain to develop AI-enhanced robotic perception solutions. However, the DeeperSense concept of AI-based multi-modal sensor fusion can be generalized to other application domains. The algorithms can be adapted and re-trained to other sensor modalities and application use cases. Thus the DeeperSense methodology is expected to become the basis for further R&D projects and industry-driven product development both in maritime and terrestrial or space applications. This includes the possiblitiy to lower the cost for robotic perception with DeeperSense, i.e. by enhancing the capabilities of low-cost sensors with AI-based software.
DeeperSense was committed to an open-data and open-code policy. Thus selections of both the algorithms and the training data collected in the project were made publicly available.
This was part of the communication and dissemination strategy of DeeperSense, which had the objective to support the European robotics and AI communities and thus strengthen European science and technology.