Community Research and Development Information Service - CORDIS

Final Report Summary - PROLIGTH-IAPP (Modern Signal Processing Methods for Ultra-realistic Light-Field Displays)

The PROLIGHT project aims at advancing the research and development in the area of visual media through developing modern signal processing methods for ultra-realistic light-field displays. The project has formed a strategic intersectorial partnership between two entities: the Department of Signal Processing at Tampere University of Technology and the SME Holografika, with the aim to elevate to the next level of visual media experience through the mechanisms of advanced research training, career development and knowledge exchange. The research programme addresses the problems of capture, analysis, modelling, compression and rendering of light fields of real-world scenes, so to support the further development of the light-field display technologies and to ensure an enhanced user experience. The knowledge-of-transfer programme relies on the complementary expertise of the two partners and effectively supports the ambitious research through six two-way secondments of mixture of researchers within the consortium and three external recruitments of experienced researchers. Training activities in the form of advanced courses, summer schools and networking activities in the form of workshops are being implemented for ensuring project-specific as well as broader training of the people involved.

For light-field capture, various approaches have been developed. For capturing a full parallax light field three solutions have been studied and developed: a motorized camera rig that allows very precise sub-pixel positioning of the capture camera; a robotic arm emulating a camera array; and a multi-camera array allowing for capturing dynamic LF content. As an alternative to pure image-based capture, a real-time end-to-end system for 3D content capture, processing and visualization on auto-stereoscopic display based on a combination of a stereoscopic camera rig, and a depth sensor has been built. The implementation of a capture setting combing stereo with a coded aperture camera based on a DSLR has been investigated and the advantages and disadvantages of different codes, as well as optical setups has been studied. Modelling of plenoptic camera systems has been implemented both in terms of ray optics and wave optics and software for generating synthetic plenoptic content has been developed.

In the area of light field modelling and compression, a rigorous mathematical analysis (modelling) of the ray space of ideal and real light-field displays has been performed that resulted in a methodology for estimating the display passband and deriving an optimal camera setup for capturing scenes to be visualized on a given light field display. For further minimizing the number of required cameras, a shearlet based interpolation method has been introduced that enables the reconstruction of a densely sampled light field out of small number of captured views. For reducing the size of light field data, two use cases, namely display-specific and display-independent LF compression have been identified and thoroughly analysed which resulted in corresponding specifications and recommendations described in MPEG working documents as well as in a real-time LF streaming technology.

The light field display technology has been advanced by developing a new intensity / color calibration method for LF displays. Hardware components for the next-generation time-multiplexed LF displays have been developed. For rendering light field data on the display, two complementary approaches have been studied. In the first approach, a display-specific API has been developed that is capable of massive 3D models. In the second approach, a novel image based rendering algorithm has been developed that can extract & fuse depth maps from unstructured light field and then use those depths for rendering rays required by the display.

A set of subjective test performed on full parallax display showed that the presence of head-parallax in a LF visualisation increases the sensitivity of human observers to depth changes by a factor of two. This emphasizes the importance of (continuous) parallax in a 3D display. A set of subjective experiments that evaluate the perceived quality of a LF displays has been performed. It has been shown that they match pretty well the theoretically determined display passband and as such can be used for evaluation of a light field display.

For evaluating the display bandwidth of an ‘unknown’ display, a method for measuring the spatial and angular resolution has been developed. It allows for an objective quantification of LF displays thus paving the way of their quality improvement.

The research results achieved during the project have been published in scientific journals with high impact factor, in proceedings of reputable conferences, and presented at various specialised events.

Altogether, the research program resulted in 47 publications and 22 presentations. Full list of publications is available on the project webpage: http://www.prolight-iapp.eu/activities/publications/, with most of them being available for download.

Planed secondments and recruits of researchers have been successfully accomplished. All recruited and seconded researchers got appropriate cross-sectorial training substantially strengthening their expertise and future carrier prospects.

All researchers involved in the project have been very active in dissemination and outreach activities, either as presenters or event organizers. The project organized three workshops which were used as venue for continuous dissemination of latest project results to professionals in the field. For a higher exposure, the PROLIGHT workshops have been co-located with other major events in Europe, namely, the NEM Summit (2013) and the 3DTV-CON series of conferences (2014, 2015). In order to support the planed training activities, the project co-organized, jointly with the 3D-ConTourNet COST Action, two training schools: The Training school on Plenoptics in Sundsvall, Sweden (2013) and the Training school on Rich 3D Content, in Budapest, Hungary (2014). The training program was complemented by a short course on Computational Imaging Algorithms for Light-Field Acquisition, Processing and Visualization organized at TUT (2013). TUT and Holografika jointly organized 3DTV-Con 2014 held in Budapest, Hungary whereas 3DTV-CON 2015 held in Lisbon, Portugal, was technically sponsored by PROLIGHT-IAPP. Project researchers gave tutorials on “Signal Processing Methods for Stereoscopic and Multi-view 3D Displays” at IEEE ICME 2013 and IEEE VCIP 2014 conferences. The project was very active in the MPEG standardization activates related with light field compression and free viewpoint video compression; and participated in the 3D-ConTourNet COST Action. The project was presented at Stanford University in the SCIEN-Colloquia series, had a demo booth at the NEM Summit in Nantes and ICIP 2014, featured demos at the GPU Technology Conference, where Holografika got also the NVIDIA NVIDIA’s „One to Watch” award.

The light-field related technology has been presented to general population through Girl’s days organized by Holografika and TUT Tech Days organized by TUT – both being annual events.

The project generated 18 exploitable foreground items most of them with strong commercialization potential. After successfully contributing to the project, a number of the involved researchers continued their careers in reputable R&D institutions such as Google, Huawei European Research Centre, Autodesk, AImotive, while the others continued toward finalization of their doctoral theses.

More information about the project can be found on the project website: http://www.prolight-iapp.eu/

Contact

Atanas Gotchev, (3D Media Team Leader)
Tel.: +358408490733
Fax: +358331153817
E-mail

Subjects

Life Sciences
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top