Hyperrealistic Imaging Experience

RealVision focusses on realistic digital imaging and video aiming to create accurate, high-quality imagery, which faithfully represents the physical environment. The ultimate goal is to create images, which are perceptually indistinguishable from a real scene.
The main goal of the network was achieved producing the next generation of ESRs, who are trained to work across disciplinary and sectorial boundaries and therefore contributing to step-changes in both research and technology development relating to hyper-realistic digital imaging.
The RealVision network trained the ESRs to be capable of working in all stages of the visual processing chain. The network is organized in 5 scientific goals, each reflecting a stage in the signal chain of Acquisition, Processing, Display, Perception and Quality of Experience. Complemented by education in entrepreneurship and innovation, these will contribute to move Europe into a leading position in scientific and technological innovation in the area of hyper-realistic imaging, encoding and display technology.
The main scientific objective is to understand, analyse and control the steps of the entire signal path to be able to deliver the highest level of realism for specific applications and requirements based on a dual framework of the physical representations and the perception of the visual content.
The major high-lights listed below and all the scientific publications, and the trained ESRs is anticipated to have a great impact for the visual industry and scientific community, in Europe and internationally. This impact will be strengthened by the high-quality data sets developed and made publicly available, as is also the case for selected software.

A major actitivity was to train the ESRs scientifically, in communication and dissemination, in exploitation and entrepreneurship. Scientific training has taken place in the form of summer schools, first secondments and local training. Four week-long training events have been completed, including scientific training, transferable skills, entrepreneurship and teambuilding.

Activities and selected results within the 5 steps of the full signal path are reported below:
1) Acquisition: A reconfigurable multi-camera, multi-view capture was established and an algorithm was developed for creating high-quality visual experiences based on the captured data. Based on these, a first high-resolution high dynamic range light field data set was captured and made available to the community.
2) Coding and Processing: The quality of light field images has been studied and an analysis of the advantage of Epipolar Plane Image representations was presented. A new technique for compression of the resulting large image data sets has been devised. A novel semantic aware Tone Mapping Operator for HDR images has also been developed.
3) Display: For faithful reconstruction of the 5D plenoptic function by hyper-realistic displays (H-RD), from the point of view of visual perception of light-field imaging, significant progress has been achieved on sampling of the plenoptic function. Combined with the work on perceptual models, a number of novel experimental (in-lab) displays have been developed (see below.)
4) Perceptual Models: A novel technique taking advantage of binocular fusion to boost perceived local contrast and visual quality of images has been devised. It is also used with the novel developed High-Dynamic-Range Multi-Focal Stereo (HDR-MF-S) display with an end-to-end imaging and rendering system. Research is conducted to improve the colour experience for individual observers, e.g. contributions have been made to develop a multi-primary HDR display with perceptual control.
5) Image Quality: To develop a novel surface quality metric for controlling object appearances, a CNN based metric was devised for detecting visible artifacts in 2D images, which can also be extended to light fields. Research on immersion was conducted, and discussed in light of implications on immersive audio-visual (AV) experiences. This work comprised defining immersion, followed up by subjective tests on audio-visual data, to establish test methodology and conducting subjective testing of immersion.

Besides the many significant scientific papers and conference presentations, important means of dissemination has been in form of high quality data sets and selected software. The RealVision project, the ideas and results have been presented at international trade shows.
The direct exploitation is primarily in form of collaboration with RealVision, industrial beneficiaries, but more indirectly also the strong industrial associated partners.

Mayor advances beyond state-of-the-art:
1) Capturing and composing a first high resolution and high dynamic range light field data sets. The data set and insights shared with the scientific community contributes a thorough base in hyperrealistic imaging.
2) Investigating angular domain specific distortions of EPI, feature extraction method and quality metrics have been used to analyse the advantages of EPI. These findings were recognized by a best student paper award (EUVIP-19). Furthermore, an efficient novel coding scheme (EPIC) based on EPI representations was developed and published.
3) A novel learning-based tone mapping operator (G-SemTMO) was developed using graph convolutions to model aesthetic styles created by expert photographers over a large dataset of images.
4) Developed a High-Dynamic-Range Multi-Focal Stereo (HDR-MF-S) display. The system can visually reproduce a real-world 3D object with accurate colour, contrast, disparity, and focal depth and as a first achieves a close perceptual match between a near-eye real-world 3D object and its displayed counterpart. A novel technique was devised that takes advantage of binocular fusion to boost perceived local contrast and visual quality of images. Software has been made publicly available to practitioners for (Dichoptic) Contrast Enhancement.
5) Developed and built the first implementation of a multi-primary high-dynamic range (MPHDR) display system for studying the photoreceptor signals in isolation with spatio-temporal control across a large luminance range. This included the control of the MPHDR display which utilises five independent parameter control of six effective primaries. Through measurements high contrast, expanded colour gamut and the ability to generate melanopsin isolating stimuli.
6) A novel surface quality metric for controlling object appearances, was devised as a CNN based metric for detecting visible artifacts in 2D images, which can also be extended to light fields. Towards the same goal also texture resolution was studied for task-specific visibility.
7) The concept of immersion widely associated with novel displays and hyper-realistic technologies was studied. The concept was scientifically defined based on a literature study. This was followed up by a methodology for subjective testing of immersion and finally an experiment evaluating influence on immersion of experience of spatial audio in a audio-visual set-up.
8) An extensive subjective quality study for perceptual evaluations of audio-visual scenes of 360 video with the higher order ambisonics spatial audio was conducted and has been made publicly available together with a high-quality dataset with 28 audio-visual scenes of 360 video with the higher order ambisonics.

Periodic Reporting for period 2 - RealVision (Hyperrealistic Imaging Experience)

Diese Seite teilen

Herunterladen