Final Report Summary - POLMOSAIC (Advanced underwater image mosaicing through imaging polarimetry)
The main goal of the POLMOSAIC project was the development of a robust image-enhancing tool, which would significantly improve underwater imagery for construction of high-resolution (~ 1 - 10 Gigapixel) ocean-floor photomosaics. Image degradation issues due to strong light scattering and absorption present a great challenge for underwater imaging and until recently image processing methods could not provide a real solution for these problems. To this end, imaging polarimetry has been proposed as a methodological framework to be integrated into the underwater image mosaicing process.
Underlying this principal objective it has been aimed to:
5L (1) develop and adapt the technique of polarimetric image acquisition for the underwater environment;
(2) incorporate this technique into the visual system of an autonomous underwater vehicle (AUV) built by the Computer Vision and Robotics Group (VICOROB) at the University of Girona; and
(3) integrate polarisation data assessment algorithms into the image enhancement and image mosaicing process.
The work towards these main goals have progressed through parallel hardware and software development schemes. During the assigned period of the project, the simultaneous development of the Girona 500 AUV has been carried out by VICOROB and it was being prepared to accommodate the desired novel imaging set-up. The underwater environment presented a great challenge to end up with a robust camera system, which can provide us the required data, while it is fully integrated into the mapping payload of the AUV. We have assessed performance, risks and circumstances of technical difficulty associated with using different polarimetric systems after initial tests carried out in the swimming pool facility of our Underwater Robotic Centre (CIRS). Based on these findings, we have finally designed a stereo camera system with a custom underwater housing that is composed of two high-resolution professional digital single-lens reflex (DSLR) cameras and is also capable of measuring polarisation data. It utilises a simple interface and was relatively easy to build it into the mapping payload of the underwater vehicle. At the same time significant efforts were dedicated to explore the possibilities of image enhancement (de-hazing) with and without exploiting polarisation data per se, as well as to survey auxiliary image processing methods that can be useful in improving this process or in other aspects of general underwater imaging.
The developed image enhancement tools were merged into the image mosaicing workflow normally used by VICOROB to generate high-resolution, seamlessly blended mosaics of underwater scenes, and the necessary modifications were done within the mosaicing algorithms themselves. During the experimentation phases of the project the camera system has gone through successful tests first in the CIRS swimming pool, then it was been validated in several undersea scenarios in coastal waters in Costa Brava and subsequently it was put on duty on board of the Girona 500 AUV to carry out the high fidelity optical survey of a shipwreck lying in the seabed of the Mediterranean Sea at 96 m depth, near to the coast of Toulon, France. This ongoing project is the result of an international cooperation of several institutions that aim to carry out an archaeological study of the shipwreck. The achieved quality of the generated image mosaics of the scene has significantly surpassed that of the results given by previous methodological approaches, which did not rely on proper dehazing of the underwater images. The extractable level of details with our novel method can undoubtedly reveal features in the imaged scene, which would have remained unnoticed or only superficially analysed even by the expert observer (archaelogists, geologists, biologists, etc.).
An unplanned and unexpected achievement during the development of the POLMOSAIC research project is that we were able to develop a versatile imaging framework that can handle the underwater image restoration not only by exploiting polarisation data, but single photographs as well, or even using other data, like pixel-wise scene depth estimation from stereo. Our conclusive findings are that although polarisation data can definitely provide additional data on scene depth compared to a single image, it involves also significantly more technical challenges in image acquisition, in particular within the underwater environment. Polarisation data can be very noisy, while sensor noise is one of the most pronounced features that one faces in an underwater imaging scenario. Therefore in real-world applications it may be preferable to rely on single images or stereo information, using which outstanding results can be achieved. However, it would be an important task of future research to explore how depth estimation (necessary to appropriately dehaze the images) from various sources, including polarisation data, could be best combined for the sake of optimal image enhancement with the lowest amount of artifacts, while taking into account the constraints given by the underwater medium and that of the carrier vehicle.
The prospective impacts of the achieved results in this project are diverse and high, both in scientific and socio-economic sense. The study of the La Lune shipwreck is an expedition long-awaited by the international community of archaeologists, who will directly utilise the results produced by our methods. At the same time this expedition is going to benefit from powerful media coverage, including a documentary movie that will be broadcasted for the general public.
In general, the scientific fields of biology, geology, oceanography and archaeology can all benefit from the higher fidelity of underwater images granted by our methods. More specifically, these will facilitate monitoring of human activities that impact the environment, rapid recovery and investigation after maritime disasters, surveying areas with deposits of solid mineral sources or carrying out offshore platform inspection and maintenance (oil and gas industry).
We also note that a specifically adapted version of the developed image enhancement methods are also utilised for polarimetric cloud detection in a device developed by a Hungarian enterprise, and these algorithms were patented in Hungary. Such an enhanced polarimetric cloud detection will allow more accurate weather reports and forecasts.
Underlying this principal objective it has been aimed to:
5L (1) develop and adapt the technique of polarimetric image acquisition for the underwater environment;
(2) incorporate this technique into the visual system of an autonomous underwater vehicle (AUV) built by the Computer Vision and Robotics Group (VICOROB) at the University of Girona; and
(3) integrate polarisation data assessment algorithms into the image enhancement and image mosaicing process.
The work towards these main goals have progressed through parallel hardware and software development schemes. During the assigned period of the project, the simultaneous development of the Girona 500 AUV has been carried out by VICOROB and it was being prepared to accommodate the desired novel imaging set-up. The underwater environment presented a great challenge to end up with a robust camera system, which can provide us the required data, while it is fully integrated into the mapping payload of the AUV. We have assessed performance, risks and circumstances of technical difficulty associated with using different polarimetric systems after initial tests carried out in the swimming pool facility of our Underwater Robotic Centre (CIRS). Based on these findings, we have finally designed a stereo camera system with a custom underwater housing that is composed of two high-resolution professional digital single-lens reflex (DSLR) cameras and is also capable of measuring polarisation data. It utilises a simple interface and was relatively easy to build it into the mapping payload of the underwater vehicle. At the same time significant efforts were dedicated to explore the possibilities of image enhancement (de-hazing) with and without exploiting polarisation data per se, as well as to survey auxiliary image processing methods that can be useful in improving this process or in other aspects of general underwater imaging.
The developed image enhancement tools were merged into the image mosaicing workflow normally used by VICOROB to generate high-resolution, seamlessly blended mosaics of underwater scenes, and the necessary modifications were done within the mosaicing algorithms themselves. During the experimentation phases of the project the camera system has gone through successful tests first in the CIRS swimming pool, then it was been validated in several undersea scenarios in coastal waters in Costa Brava and subsequently it was put on duty on board of the Girona 500 AUV to carry out the high fidelity optical survey of a shipwreck lying in the seabed of the Mediterranean Sea at 96 m depth, near to the coast of Toulon, France. This ongoing project is the result of an international cooperation of several institutions that aim to carry out an archaeological study of the shipwreck. The achieved quality of the generated image mosaics of the scene has significantly surpassed that of the results given by previous methodological approaches, which did not rely on proper dehazing of the underwater images. The extractable level of details with our novel method can undoubtedly reveal features in the imaged scene, which would have remained unnoticed or only superficially analysed even by the expert observer (archaelogists, geologists, biologists, etc.).
An unplanned and unexpected achievement during the development of the POLMOSAIC research project is that we were able to develop a versatile imaging framework that can handle the underwater image restoration not only by exploiting polarisation data, but single photographs as well, or even using other data, like pixel-wise scene depth estimation from stereo. Our conclusive findings are that although polarisation data can definitely provide additional data on scene depth compared to a single image, it involves also significantly more technical challenges in image acquisition, in particular within the underwater environment. Polarisation data can be very noisy, while sensor noise is one of the most pronounced features that one faces in an underwater imaging scenario. Therefore in real-world applications it may be preferable to rely on single images or stereo information, using which outstanding results can be achieved. However, it would be an important task of future research to explore how depth estimation (necessary to appropriately dehaze the images) from various sources, including polarisation data, could be best combined for the sake of optimal image enhancement with the lowest amount of artifacts, while taking into account the constraints given by the underwater medium and that of the carrier vehicle.
The prospective impacts of the achieved results in this project are diverse and high, both in scientific and socio-economic sense. The study of the La Lune shipwreck is an expedition long-awaited by the international community of archaeologists, who will directly utilise the results produced by our methods. At the same time this expedition is going to benefit from powerful media coverage, including a documentary movie that will be broadcasted for the general public.
In general, the scientific fields of biology, geology, oceanography and archaeology can all benefit from the higher fidelity of underwater images granted by our methods. More specifically, these will facilitate monitoring of human activities that impact the environment, rapid recovery and investigation after maritime disasters, surveying areas with deposits of solid mineral sources or carrying out offshore platform inspection and maintenance (oil and gas industry).
We also note that a specifically adapted version of the developed image enhancement methods are also utilised for polarimetric cloud detection in a device developed by a Hungarian enterprise, and these algorithms were patented in Hungary. Such an enhanced polarimetric cloud detection will allow more accurate weather reports and forecasts.