Light fields cameras capture light rays as they interact with the scene. The flow of rays yields a rich description of the scene enabling advanced image creation capabilities, 3D scene geometry estimation and scene reconstruction. Applications include photography, augmented reality, autonomous vehicles, surveillance, but also microscopy, medical imaging, and particle image velocimetry. However, the trajectory to a deployment of light fields remains cumbersome. Barriers are limitations of capturing devices in terms of spatial or angular resolution, or noise. Another barrier is the huge amount of high-dimensional data produced by light fields with implications on storage and processing time. The development of efficient methods for scene analysis, depth estimation or scene flow estimation, from light fields, for editing is another challenge for technology adoption.
The objective was to address these barriers by leveraging advances in image processing, computer vision and machine learning, and lay algorithmic foundations for the light field processing chain. A first challenge was the design of camera models to capture light fields with good spatio-angular resolution. This involves algorithmic developments in the framework of compressive sensing with deep learning reconstruction. A second challenge was related to the high data dimensionality. Data processing becomes tougher as dimensionality increases, hence the need for tools for data dimensionality reduction or low dimensional embedding. These models, together with scene analysis algorithms, have been shown to be key components of light field compression architectures. A third challenge was related to technological limitations of capturing devices with impact on the light field spatio-angular resolution and noise.
The project methodology has evolved from signal processing to machine learning, from hand-crafted to learned signal priors. The project has thus contributed to the leveraging of advances in deep learning in the light field processing chain, going from compressive acquisition with novel camera designs and deep reconstruction, low rank and neural radiance field representation and compression, to restoration, including computer vision problems such as view synthesis, scene flow estimation. The project has also contributed to novel methodologies in the underlying fields of scene modeling and inverse problems with machine learning.
