Periodic Reporting for period 1 - RealVision (Hyperrealistic Imaging Experience)
Reporting period: 2018-01-01 to 2019-12-31
The main goal of the network is to produce the next generation of ESRs, who are trained to work across disciplinary and sectorial boundaries and therefore bring about step-changes in both research and technology development relating to hyper-realistic digital imaging.
The RealVision network will train the ESRs to be capable of working in all stages of the visual processing chain. The network is organized in 5 scientific goals, each reflecting a stage in the signal chain of Acquisition, Processing, Display, Perception and Quality of Experience. Complemented by education in entrepreneurship and innovation these will contribute to move Europe into a leading position in scientific and technological innovation in the area of hyper-realistic imaging, encoding and display technology.
The main scientific objective is to understand, analyse and control the steps of the entire signal path to be able to deliver the highest level of realism for specific applications and requirements based on a dual framework of the physical representations and the perception of the visual content.
Activities and results within the steps of the full signal path: Acquisition, Processing, Display, Perception and Quality of Experience are briefly reported below.
1) Acquisition: A reconfigurable multi-camera, multi-view capture was established and an algorithm for creating high-quality visual experiences based on the captured data. The multi-camera data may also be processed and represented as light field images.
2) Coding and Processing: The quality of light field images has been studied and an analysis of the advantage of Epipolar Plane Image representations was presented. A new technique for compression of the resulting large image data sets has been devise. A novel semantic aware Tone Mapping Operator for HDR images has also been developed.
3) Display: For faithful reconstruction of the 5D plenoptic function by hyper-realistic displays (H-RD), from the point of view of visual perception of light-field imaging, significant progress has been achieved on sampling of the plenoptic function. Efficient rendering of VR applications on HMD is also being studied.
4) Perceptual Models: The work is a synergetic mix of psychophysical experiments, hardware and software development and modelling. A novel technique taking advantage of binocular fusion to boost perceived local contrast and visual quality of images has been devised. It utilizes integration of how perceptual cues contribute to the impression that perceived scenes look highly realistic.
Research is conducted to improve the colour experience for individual observers, by exploiting knowledge of human colour vision including individual differences and adaptation. Furthermore, contributions have been made to developing a multi-primary HDR display.
5) Image Quality: Study and analysis of public light field data sets for QoE have been completed. To develop a novel surface quality metric for controlling object appearances, a CNN based metric was devised for detecting visible artifacts in 2D images, which can also be extended to light fields. A literature review on immersion was conducted, and discussed in light of implications on immersive audio-visual (AV) experiences.
Selected results in relation to the full signal path from capture to display include.
- The projects has generated three high quality visual data sets for developing novel techniques towards novel hyper-realistic imaging
- Software has been made publicly available to practioners for (Dichoptic) Contrast Enhancement.
- Already the project, the ideas and initial result have been presented at international trade shows.
- A novel method for rich media data from multi-camera, e.g. for digital cinema has been developed.
The project is anticipated to continue to produce and communicate results for the visual industry and scientific community.