Final Report Summary - IP4EC (Image processing for enhanced cinematography)
The objective of this project has been to develop image processing algorithms based mainly on vision science models that address different challenges in moviemaking, from shooting to exhibition.
Given that in terms of sensing capabilities cameras are in most regards better than human photoreceptors, the superiority of human vision over camera systems lies in the better processing which is carried out in the retina, thalamus and visual cortex. Therefore, rather than working on the hardware, improving lenses and sensors, we have resorted instead to knowledge on visual neuroscience and models of visual perception to develop software methods mimicking processes in the human visual system, and applied these methods to images captured with a regular camera. We have taken the same approach when addressing projection/exhibition, developing vision-based image processing algorithms that try to overcome the limitations in contrast and color reproduction that current display systems have. For shooting and post-production we have also used image processing methods based on PDEs.
In terms of applications, the main outcomes of our project include the development of state-of-the-art methods for noise removal, creation and encoding of high dynamic range (HDR) images and video, transformation of HDR content into standard dynamic range (SDR) for display on regular devices, adaptation of the color gamut of image and video to the color capabilities of the display where this content is going to be exhibited, color matching among different shots of the same scene for cinema post-production, and color and contrast enhancement. These have resulted in three patent applications and two ERC Proof of Concept grants to bring some of these inventions to market.
In terms of basic research, our main results include: showing for the first time that the visual perception phenomenon of brightness assimilation may already be happening at the retina; proposing novel visual perception models for lightness, contrast, color and subjective image quality that have advanced the state-of-the-art by providing better predictions of some perceptual phenomena; relating retinal models with the efficient coding approach from vision science and showing how the validity of vision models based on natural image statistics is not put into question by the influence of the dynamic range, because early retinal processes already remove most of this influence.
Given that in terms of sensing capabilities cameras are in most regards better than human photoreceptors, the superiority of human vision over camera systems lies in the better processing which is carried out in the retina, thalamus and visual cortex. Therefore, rather than working on the hardware, improving lenses and sensors, we have resorted instead to knowledge on visual neuroscience and models of visual perception to develop software methods mimicking processes in the human visual system, and applied these methods to images captured with a regular camera. We have taken the same approach when addressing projection/exhibition, developing vision-based image processing algorithms that try to overcome the limitations in contrast and color reproduction that current display systems have. For shooting and post-production we have also used image processing methods based on PDEs.
In terms of applications, the main outcomes of our project include the development of state-of-the-art methods for noise removal, creation and encoding of high dynamic range (HDR) images and video, transformation of HDR content into standard dynamic range (SDR) for display on regular devices, adaptation of the color gamut of image and video to the color capabilities of the display where this content is going to be exhibited, color matching among different shots of the same scene for cinema post-production, and color and contrast enhancement. These have resulted in three patent applications and two ERC Proof of Concept grants to bring some of these inventions to market.
In terms of basic research, our main results include: showing for the first time that the visual perception phenomenon of brightness assimilation may already be happening at the retina; proposing novel visual perception models for lightness, contrast, color and subjective image quality that have advanced the state-of-the-art by providing better predictions of some perceptual phenomena; relating retinal models with the efficient coding approach from vision science and showing how the validity of vision models based on natural image statistics is not put into question by the influence of the dynamic range, because early retinal processes already remove most of this influence.