Periodic Reporting for period 1 - VIPERCON (Emulating visual perception of contrast for image capture, post-production and synthesis)
Berichtszeitraum: 2016-10-01 bis 2018-03-31
The contrast in a scene is measured by the ratio, called dynamic range (DR), of light intensity values between its brightest and darkest points. While common natural scenes may have a DR of 1 million or higher, regular screens have a DR of around 100. Therefore all images and videos have to be transformed before display so that their contrast is reduced, at the same time trying to maintain the natural appearance and visibility of details of the original. All cameras automatically perform this transformation when they record a photo or video in any standard format; in professional cinema the transform is manually performed by a skilled technician/artist later in post-production.
The objective of the ERC Starting Grant “Image Processing for Enhanced Cinematography” (IP4EC), to which this ERC Proof of Concept project is associated, is to develop image processing algorithms for cinema that allow people watching a movie on a screen to see the same details and colors as people at the shooting location can. Rather than working on the hardware, e.g. improving sensors and displays, we develop instead software methods mimicking processes in the human visual system, and apply these methods to images captured with a regular camera and shown on a regular screen. A significant outcome from this research has been to propose a vision model for contrast perception that can be realized as an image-dependent transformation that can accurately reproduce the detail and contrast visible in the original scene. The model is based on findings from vision science studies and is well suited to the statistics of natural images. The resulting transform is automatic (no need for user-selected parameters) and of extremely low computational complexity, making it a good candidate for real-time applications. It produces images and videos that look natural, without any halos, spurious colors or artifacts. The model has been validated through psychophysical tests that confirm that it outperforms other state of the art algorithms in terms of users’ preference. A European patent application has been filed.
The core idea of this ERC Proof of Concept project was to take this research outcome, our vision model, and make a series of products that highlight its capabilities on two distinct scenarios: image and video capture, and professional post-production of photography and cinema. As a result, we have produced:
- a web application where users (professional or amateur) can upload high dynamic range (HDR) pictures or RAW image files, process them with our algorithm, download the results and also compare them with the default image produced by the camera;
- a plug-in for a popular cinema post-production software suite.
Regarding the socioeconomic impact, the web application shows how our method, if implemented in-camera, would allow to capture better, more realistic and natural-looking pictures and videos that would yield images of quality comparable to those obtained with high-end equipment or corrected by a professional. Clearly, the wide scope of this particular impact can not be overstated. There is a 16% compound annual growth rate in the number of photos taken, mainly due to the increase in usage and ownership of mobile phones, and the forecast is that by the end of the year nearly 80% of all photos will be taken with mobile phones. And both the web application and the cinema post-production plug-in show that, for the media industries, our method has a direct application at several stages of the production chain, allowing for better and faster work during shoots and a lesser need for post-production time, with the associated economic impact in savings of resources given the very expensive nature of these enterprises, and can also have a significant artistic impact, helping good-visual-quality productions become more affordable and giving more artistic freedom to the director to dedicate more shooting time to lighting aiming for artistic expression.
Through the tools developed in this project we have been able to raise the interest of major media companies in Europe and Asia, having performed visits and individual demonstrations, with very positive feedback.
The objective of the ERC Starting Grant “Image Processing for Enhanced Cinematography” (IP4EC), to which this ERC Proof of Concept project is associated, is to develop image processing algorithms for cinema that allow people watching a movie on a screen to see the same details and colors as people at the shooting location can. Rather than working on the hardware, e.g. improving sensors and displays, we develop instead software methods mimicking processes in the human visual system, and apply these methods to images captured with a regular camera and shown on a regular screen. A significant outcome from this research has been to propose a vision model for contrast perception that can be realized as an image-dependent transformation that can accurately reproduce the detail and contrast visible in the original scene. The model is based on findings from vision science studies and is well suited to the statistics of natural images. The resulting transform is automatic (no need for user-selected parameters) and of extremely low computational complexity, making it a good candidate for real-time applications. It produces images and videos that look natural, without any halos, spurious colors or artifacts. The model has been validated through psychophysical tests that confirm that it outperforms other state of the art algorithms in terms of users’ preference. A European patent application has been filed.
The core idea of this ERC Proof of Concept project was to take this research outcome, our vision model, and make a series of products that highlight its capabilities on two distinct scenarios: image and video capture, and professional post-production of photography and cinema. As a result, we have produced:
- a web application where users (professional or amateur) can upload high dynamic range (HDR) pictures or RAW image files, process them with our algorithm, download the results and also compare them with the default image produced by the camera;
- a plug-in for a popular cinema post-production software suite.
Regarding the socioeconomic impact, the web application shows how our method, if implemented in-camera, would allow to capture better, more realistic and natural-looking pictures and videos that would yield images of quality comparable to those obtained with high-end equipment or corrected by a professional. Clearly, the wide scope of this particular impact can not be overstated. There is a 16% compound annual growth rate in the number of photos taken, mainly due to the increase in usage and ownership of mobile phones, and the forecast is that by the end of the year nearly 80% of all photos will be taken with mobile phones. And both the web application and the cinema post-production plug-in show that, for the media industries, our method has a direct application at several stages of the production chain, allowing for better and faster work during shoots and a lesser need for post-production time, with the associated economic impact in savings of resources given the very expensive nature of these enterprises, and can also have a significant artistic impact, helping good-visual-quality productions become more affordable and giving more artistic freedom to the director to dedicate more shooting time to lighting aiming for artistic expression.
Through the tools developed in this project we have been able to raise the interest of major media companies in Europe and Asia, having performed visits and individual demonstrations, with very positive feedback.