Intelligent Scene Sensing and Analysis in Underwater Environments

Project Information

iSEAu

Grant agreement ID: 101030367

DOI

10.3030/101030367

Project closed

EC signature date 4 May 2021

Start date 1 March 2022

End date 30 December 2024

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 165 085,44

EU contribution

€ 165 085,44

Coordinated by

ETHNICON METSOVION POLYTECHNION
Greece

Periodic Reporting for period 1 - iSEAu (Intelligent Scene Sensing and Analysis in Underwater Environments)

Reporting period: 2022-03-01 to 2024-02-29

Summary of the context and overall objectives of the project

Reliable and detailed underwater scene sensing, and analysis has a fundamental role in numerous underwater activities with scientific and economic interest. In fact, blue economy in 2020 amounted to €218 BN (GVA) and counts more than 5 million jobs in EU. At the same time, intensive research work focuses in marine and underwater environments including the study of climate change and its impacts, marine biology, underwater archaeology, marine geology, and marine renewable energy.

The main objective of iSEAu is to advance the scene sensing and perception capabilities in underwater environemnts by employing state-of-the-art machine learning methods, and using combinations of conventional (RGB) and non-conventional cameras, like multispectral and single photons cameras (SPCs).

iSEAu aims to bring together the fields of computer vision, machine learning and remote sensing for optimally addressing the underwater visual sensing challenges. The project objectives address this challenge in two
levels. The first concerns the development of methods for reducing the geometric and radiometric distortions introduced by the water through learning-based methods, and the development of methods based on transient imaging for perception in challenging visibility conditions. The second level concerns the adaptation and enhancement to the underwater domain of state-of-the-art methods for image-based extraction of structural and semantic information, and their field-testing considering representative application scenarios.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The iSEAu project introduced significant advancements in underwater imaging and analysis through the development of advanced AI-driven methodologies. It successfully integrated deep learning, physics-based modelling, and specialized imaging techniques to enhance underwater image quality and enable intelligent scene understanding. Key achievements include the creation of StreamUR for real-time image restoration, MD2IP for training-free multispectral demosaicing, and a deep learning-powered system for autonomous gas bubble detection. Field deployments, notably in the deep hydrothermal field of Kolumbo underwater volcano near Santorini, Greece, and at the coastal hydrothermal field of Palaiochori in Milos, Greece, validated these methods, demonstrating their robustness in real-world environments. Furthermore, the project advanced low-power AI models for embedded systems, enabling their use in platforms with limited computational budget, as ROVs, AUVs and benthic stations. The work of the project led also to contributions in affine domains like satellite imaging and radiation mapping.

Overall, the results of the iSEAu project confirm the feasibility and effectiveness of AI-driven underwater sensing technologies. The project has demonstrated significant improvements in underwater imaging quality and automatic underwater scene analysis, offering a robust framework for future advancements in oceanographic research, while opening the way for potential business exploitation of its results. Ultimately, iSEAu delivered advancements in underwater technology, towards improved marine research, industrial applications, and environmental monitoring.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The iSEAu project pushes the boundaries of underwater imaging by developing a comprehensive suite of advanced techniques. It pioneers novel geometric correction methods, and significantly enhances radiometric correction through physics-aware deep learning and inverse problem formulations, extending to multispectral imaging for quantitative remote sensing. In Single-Photon Counting (SPC)-based imaging, iSEAu foresees backscatter removal and depth estimation, enabling accurate albedo recovery in extreme visibility conditions.

Furthermore, leveraging these advancements, iSEAu focuses on underwater scene analysis. It develops deep learning feature learning and 3D vision adaptations for robust geometric reconstruction, and adapts semantic segmentation methods with a focus on few-shot, weakly, and self-supervised learning to extract detailed semantic information like seabed composition and marine life monitoring. Finally, the project explores neural rendering for immersive 3D scene representation, incorporating multispectral data and addressing data scarcity, thereby advancing computer vision applications in underwater environments.

Multispectral 3D reconstruction of a hydrothermal vent chimney in Kolumbo

Restoration of underwater video frames based on the StreamUR method