European Training Network on Full Parallax Imaging

Project Information

ETN-FPI

Grant agreement ID: 676401

DOI

10.3030/676401

Project closed

EC signature date 5 August 2015

Start date 1 October 2015

End date 30 September 2019

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 3 801 840,48

EU contribution

€ 3 801 840,48

Coordinated by

TAMPEREEN KORKEAKOULUSAATIO SR
Finland

Periodic Reporting for period 2 - ETN-FPI (European Training Network on Full Parallax Imaging)

Reporting period: 2017-10-01 to 2019-09-30

Summary of the context and overall objectives of the project

A full-parallax display is expected to recreate the complete lightfield (LF), i.e. light travelling in every direction through every point in space. The need of such ultimatively-realistic display has been addressed by the European Training Network on Full Parallax Imaging by developing the related concept of and integrating the required work toward capturing, visualizing and interacting with highly realistic 3D visual scenes. The Network’s ultimate goal has been to train 15 next-generation researchers through 15 FPI-related novel multidisciplinary projects, spanning problems in optics, visual neuroscience, visual computing, and advanced signal and data processing.

Four objectives have been formulated and addressed by the research and training programmes:
• Advance the theory and practice of 3D scene sensing and content creation and develop new imaging systems operating at various scales, employing modern imaging concepts such as integral imaging for microscopy, plenoptic lenslet cameras for macroscopic content, and large-scale full parallax multiview and multi-sensor camera systems.
• Advance the theory and practice of LF data analysis and interpretation for novel FPI applications in various fields. Develop novel effective representations for LF data and associated reconstruction and conversion algorithms. Develop computationally-efficient compression and processing methods utilizing modern parallel computing platforms.
• Characterize optimal visualization of 3D visual scenes onto various displays aimed at full parallax. Develop methods and systems for realistic interaction with such displays.
• Organize network-wide training program aimed at establishing competence for the Early Stage Researchers (ESRs) helping them to become excellent researchers and future research leaders; and to convey the necessity of interdisciplinary and international collaborations to reach ground-breaking results.

All objectives have been achieved as planned. The research resulted in new devices, methods, and algorithms that have direct impacts on today’s life and at the same time support the future development of the area of FPI. Fifteen young researchers have received cross-disciplinary training forming a solid fundament for their future careers.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The network combined five fundamental areas for training and research, namely visual neuroscience, visual computing, signal and data processing, optics and management of innovation (Fig.1). The research has been organized in three work packages (WPs).

WP1 dealt with LF capture and content creation resulting in the following outcomes
• LF model for plenoptic capturing in geometric optics, complementing ray-based and wave-based theories
• Method for depth assisted demosaicing of plenoptic images
• Fourier-domain integral microscope (FiMic) prototype with improved spatial and depth-of-field resolution
• Approach for enhanced capture and visualization of transparent and semi transparent microscopic samples
• Computationally-efficient methods for estimating high-quality per-pixel depth maps captured by large-scale multi-camera arrays and gantry systems
• Model for the point-spread function of a plenoptic 2.0 camera
• Algorithm for rectifying the radial distortions of the micro images in a plenoptic camera
• Methods for multi-sensor data fusion and depth-image based rendering combining large-scale multi-camera array and Time-of-Flight cameras

WP2 dealt with challenges in computational imaging and compression, resulting in the following outcomes:
• Shearlet-domain dense reconstruction of LFs containing non-Lambertian objects
• Improved method for creating a focal stack from images acquired by FiMic microscope
• Convolutional neural network based approach for LF reconstruction
• Rate allocation and prediction scheme for LF compression
• LF compression method using a Shearlet-transform based prediction scheme for interview prediction
• Low-level algorithmic optimizations of FPI algorithms
• Simulation environment for simulating complex plenoptic-type camera configurations
• Robust calibration method for plenoptic cameras having microlenses with different focal lengths
• Algorithm for merging sparsely sampled camera views
• Rendering pipeline for non-planar inside-out LF view rendering
• Optimization of LF interpolation algorithms for achieving the best display-driven quality

WP3 focused on vision and visualization delivering the following results:
• Low-complexity algorithm for handling distortions in fish-eye capture and virtual view camera rendering
• Unsupervised calibration of multisensor (RGB-NIR) pairs
• Two watermarking algorithms for LF data with improved imperceptibility and robustness
• Ray tracing and screen-space ambient occlusion algorithms for the HoloVizio technology
• Software framework for simulation of F displays
• Continuous-parallax distortion visibility measure based on the spectral content of full-parallax sequences predicting the subject’s response for a given parallax distortion level
• Derivation of the spatial contrast sensitivity function at high luminance
• Study on the role of Longitudinal Chromatic Aberration for driving accommodative response of the viewer
• Evaluation of the limits of the temporal-resolving ability of the human visual system

The network run an extensive training program comprising 6 training schools, 3 workshops and 3 online seminars (c.f. Fig 2). it culminated in the 2019 European Light Field Imaging workshop, organized by the network.

The project results have been published in more than 74 conference and journal publications, and two patent applications. The results have been disseminated throughout 3 tutorials, 2 workshops, one special session and 8 invited presentations. Six datasets covering different aspects of FPI have been prepared and made available to the community. More than 30 events and 6 trade shows (NAB and IBC among them) reached the general public. The project also contributed to the work of JPEG and MPG standardization bodies

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Fifteen ESRs have been exposed to the visual media value chain (Fig. 3) through multidisciplinary projects and courses, and inter-sectorial secondments. As a result, innovative techniques and products matching the relevant ecosystems have been studied and developed. They constitute a fundamental step for the adoption of the LF technologies and have the potential to support creative media and game industries, industrial design, education, ultra-realistic video-conferencing, and bioimaging and microscopy. They also help in opening up new creative professions related with FPI content creation and use where the trained ESRs will be the prime candidates.

FPI is especially promising in medicine and biology, with the goal for volumetric rendering of complex medical data on a LF display. This would improve the doctors’ interactions with the 3D data and enable new workflows. Telehealth and remote surgery could benefit greatly from high-quality livestreamed 3D LF content, since multiple doctors would be able to view, discuss, monitor the progress and make decisions based on the full visual information. Results from psychophysical studies will be highly relevant for the design of near-eye displays that aim to support focus cues and for the compression of full parallax imagery utilizing the parallax perceptual thresholds. Standardization efforts within MPEG and JPEG will be also supported.

Fig. 3: Network contribution and position of partners with respect to the visual media value chain

Fig.1: ETN-FPI research topics, corresponding work packages and underlying training areas

Fig. 2: ETN-FPI Training programme and ESR involvement