Periodic Reporting for period 1 - PLENOPTIMA (Plenoptic Imaging)
Reporting period: 2021-01-01 to 2022-12-31
The PLENOPTIMA project goal is to establish new cross-sectorial, cross-national, multi-university sustainable doctoral degree programmes in the area of plenoptic imaging and to train the first fifteen future researchers and creative professionals for the benefit of a variety of application sectors. PLENOPTIMA develops a trans-disciplinary approach to imaging, which includes the physics of light, new optical materials and sensing principles, signal processing methods, new computing architectures, and vision science modelling.
The project aims at achieving its goals by addressing the following three research objectives:
• Advance theory and practice for novel design of full computational plenoptic imaging acquisition systems
• Design models and methods for plenoptic data processing, with a focus on dimensionality reduction and compression, and on inverse problems using machine learning
• Advance theory and practice for efficient rendering and interactive visualization on displays capable of reproducing all physiological visual depth cues, and enabling realistic interaction
Advanced research conducted by cross-disciplinary experts will bring improvements in various areas such as medicine (live cell microscopy), creative industries (gaming, movies), industrial environment (tele-operations), consumer electronics (AR, VR), among others.
WP1 deals with computational plenoptic acquisition systems and has generated the following main outcomes:
• Novel computational camera with extended depth-of-field obtained by simultaneously co-designing optical and computational elements
• State of the art survey of implicit and explicit scene representations
• Method for generating a neural radiance field from 360 degree cameras
• Optimized segmentation model for detection of human body tissues and organs, such as varicose veins
• Analysis of dynamic speckle imaging techniques
WP2 deals with models and methods for plenoptic data processing and has generated the following main outcomes:
• Algorithms for reconstruction of two-dimensional images using deep equilibrium models
• Accelerated optimization of gated networks by incorporating block-based steered mixture of experts in the initialization stage
• Novel image denoising approaches based on steered mixture of experts
• Deep learning-based video compression approach adapted to compression of light fields
• EPINET structure with reduced complexity usable for light field applications
WP3 deals with perception and interactive visualization of plenoptic content and has generated the following main outcomes:
• An LF reconstruction scheme for near-eye display which incorporates perceptually-driven end-to-end optimization
• Dynamic Load Balancing approach for Real-Time Multiview Path Tracing on Multi-GPU Architectures
• Multimodal dataset of human emotion awareness under partial occlusion
• Method for reconstruction of light field from a single planar image
In addition to research, the network organized two Training Schools, three Workshops and two Webinar series. The covered topics ranged from modern signal processing, ray and wave optics, and computer vision, to plenoptic data and machine learning modelling. In addition, the development of complementary skills, such as scientific writing and ethics in research was supported by workshop presentations and group works.
The PLENOPTIMA project will address shortcoming in the current state of the art and strengthen the area of plenoptic imaging. It is expected to have impact on various industries as mentioned above. From the social perspective, the outcomes of PLENOPTIMA will open new possibilities for realistic collaborative working environments. Natural virtual shared environments will reduce travel time and travel cost and as such have impact on climate itself. Plenoptic Imaging is also strong enabler of health-related applications such as tomography and live-cell microscopy, safety-critical applications in machine and robot industries, and VR/AR/XR applications.
On an individual level, the involved ESRs will get a world-quality cross-disciplinary training making them versatile experts in the area of plenoptic imaging. During the project they will, in addition to specialized knowledge in their area of expertise, build a strong set of transferable skills. The gathered knowledge and skills will culminate in the double or joint doctoral degree diplomas, as illustrated in Figure 2. This will open various career opportunities either in academia or industry. By the end of their training, the Marie Sklodowska-Curie fellows will be prepared to lead the next generation of researchers and start a new cycle of breakthrough innovations.