Periodic Reporting for period 1 - PLENOPTIMA (Plenoptic Imaging)
Okres sprawozdawczy: 2021-01-01 do 2022-12-31
Plenoptic Imaging studies the phenomena of light field formation, propagation, sensing and perception along with the computational methods for extracting, processing and rendering the visual information.
The PLENOPTIMA project goal is to establish new cross-sectorial, cross-national, multi-university sustainable doctoral degree programmes in the area of plenoptic imaging and to train the first fifteen future researchers and creative professionals for the benefit of a variety of application sectors. PLENOPTIMA develops a trans-disciplinary approach to imaging, which includes the physics of light, new optical materials and sensing principles, signal processing methods, new computing architectures, and vision science modelling.
The project aims at achieving its goals by addressing the following three research objectives:
• Advance theory and practice for novel design of full computational plenoptic imaging acquisition systems
• Design models and methods for plenoptic data processing, with a focus on dimensionality reduction and compression, and on inverse problems using machine learning
• Advance theory and practice for efficient rendering and interactive visualization on displays capable of reproducing all physiological visual depth cues, and enabling realistic interaction
Advanced research conducted by cross-disciplinary experts will bring improvements in various areas such as medicine (live cell microscopy), creative industries (gaming, movies), industrial environment (tele-operations), consumer electronics (AR, VR), among others.
The PLENOPTIMA project goal is to establish new cross-sectorial, cross-national, multi-university sustainable doctoral degree programmes in the area of plenoptic imaging and to train the first fifteen future researchers and creative professionals for the benefit of a variety of application sectors. PLENOPTIMA develops a trans-disciplinary approach to imaging, which includes the physics of light, new optical materials and sensing principles, signal processing methods, new computing architectures, and vision science modelling.
The project aims at achieving its goals by addressing the following three research objectives:
• Advance theory and practice for novel design of full computational plenoptic imaging acquisition systems
• Design models and methods for plenoptic data processing, with a focus on dimensionality reduction and compression, and on inverse problems using machine learning
• Advance theory and practice for efficient rendering and interactive visualization on displays capable of reproducing all physiological visual depth cues, and enabling realistic interaction
Advanced research conducted by cross-disciplinary experts will bring improvements in various areas such as medicine (live cell microscopy), creative industries (gaming, movies), industrial environment (tele-operations), consumer electronics (AR, VR), among others.
The research work in PLENOPTIMA has been organized in three work packages as illustrated in Figure 1, each of them focusing on achieving one of the project’s objectives.
WP1 deals with computational plenoptic acquisition systems and has generated the following main outcomes:
• Novel computational camera with extended depth-of-field obtained by simultaneously co-designing optical and computational elements
• State of the art survey of implicit and explicit scene representations
• Method for generating a neural radiance field from 360 degree cameras
• Optimized segmentation model for detection of human body tissues and organs, such as varicose veins
• Analysis of dynamic speckle imaging techniques
WP2 deals with models and methods for plenoptic data processing and has generated the following main outcomes:
• Algorithms for reconstruction of two-dimensional images using deep equilibrium models
• Accelerated optimization of gated networks by incorporating block-based steered mixture of experts in the initialization stage
• Novel image denoising approaches based on steered mixture of experts
• Deep learning-based video compression approach adapted to compression of light fields
• EPINET structure with reduced complexity usable for light field applications
WP3 deals with perception and interactive visualization of plenoptic content and has generated the following main outcomes:
• An LF reconstruction scheme for near-eye display which incorporates perceptually-driven end-to-end optimization
• Dynamic Load Balancing approach for Real-Time Multiview Path Tracing on Multi-GPU Architectures
• Multimodal dataset of human emotion awareness under partial occlusion
• Method for reconstruction of light field from a single planar image
In addition to research, the network organized two Training Schools, three Workshops and two Webinar series. The covered topics ranged from modern signal processing, ray and wave optics, and computer vision, to plenoptic data and machine learning modelling. In addition, the development of complementary skills, such as scientific writing and ethics in research was supported by workshop presentations and group works.
WP1 deals with computational plenoptic acquisition systems and has generated the following main outcomes:
• Novel computational camera with extended depth-of-field obtained by simultaneously co-designing optical and computational elements
• State of the art survey of implicit and explicit scene representations
• Method for generating a neural radiance field from 360 degree cameras
• Optimized segmentation model for detection of human body tissues and organs, such as varicose veins
• Analysis of dynamic speckle imaging techniques
WP2 deals with models and methods for plenoptic data processing and has generated the following main outcomes:
• Algorithms for reconstruction of two-dimensional images using deep equilibrium models
• Accelerated optimization of gated networks by incorporating block-based steered mixture of experts in the initialization stage
• Novel image denoising approaches based on steered mixture of experts
• Deep learning-based video compression approach adapted to compression of light fields
• EPINET structure with reduced complexity usable for light field applications
WP3 deals with perception and interactive visualization of plenoptic content and has generated the following main outcomes:
• An LF reconstruction scheme for near-eye display which incorporates perceptually-driven end-to-end optimization
• Dynamic Load Balancing approach for Real-Time Multiview Path Tracing on Multi-GPU Architectures
• Multimodal dataset of human emotion awareness under partial occlusion
• Method for reconstruction of light field from a single planar image
In addition to research, the network organized two Training Schools, three Workshops and two Webinar series. The covered topics ranged from modern signal processing, ray and wave optics, and computer vision, to plenoptic data and machine learning modelling. In addition, the development of complementary skills, such as scientific writing and ethics in research was supported by workshop presentations and group works.
Research-wise, the project aims at advancing the area of plenoptic imaging by tackling problems related to acquisition, modeling, processing, and visualization of plenoptic data. On the acquisition side, this includes researching new optical meta materials for designing phase-code masks, using omnidirectional capture settings, modeling diverse surfaces by plenoptic point clouds, and investigating coherent light sensing approaches for 3D visual data acquisition. The acquisition stage is complemented with the modeling and processing stage where new methods for processing multi modal data are being developed, such as, learning algorithms for solving inverse problems with new imaging modalities, deep gating network based algorithms for noise reduction, compression, and super-resolution of light field data and optimization of distribute heterogenous computing for real time light field interpolation algorithms. It is expected that the developed acquisition and processing techniques will enable the capture and reconstruction of multi-dimensional content with high angular, spatial, temporal and wavelength resolution. On the visualization side, novel perceptually optimized light field displays will be developed, and a more realistic experience will be supported by removing the headset in XR applications and providing AR for interactive remote access and operation.
The PLENOPTIMA project will address shortcoming in the current state of the art and strengthen the area of plenoptic imaging. It is expected to have impact on various industries as mentioned above. From the social perspective, the outcomes of PLENOPTIMA will open new possibilities for realistic collaborative working environments. Natural virtual shared environments will reduce travel time and travel cost and as such have impact on climate itself. Plenoptic Imaging is also strong enabler of health-related applications such as tomography and live-cell microscopy, safety-critical applications in machine and robot industries, and VR/AR/XR applications.
On an individual level, the involved ESRs will get a world-quality cross-disciplinary training making them versatile experts in the area of plenoptic imaging. During the project they will, in addition to specialized knowledge in their area of expertise, build a strong set of transferable skills. The gathered knowledge and skills will culminate in the double or joint doctoral degree diplomas, as illustrated in Figure 2. This will open various career opportunities either in academia or industry. By the end of their training, the Marie Sklodowska-Curie fellows will be prepared to lead the next generation of researchers and start a new cycle of breakthrough innovations.
The PLENOPTIMA project will address shortcoming in the current state of the art and strengthen the area of plenoptic imaging. It is expected to have impact on various industries as mentioned above. From the social perspective, the outcomes of PLENOPTIMA will open new possibilities for realistic collaborative working environments. Natural virtual shared environments will reduce travel time and travel cost and as such have impact on climate itself. Plenoptic Imaging is also strong enabler of health-related applications such as tomography and live-cell microscopy, safety-critical applications in machine and robot industries, and VR/AR/XR applications.
On an individual level, the involved ESRs will get a world-quality cross-disciplinary training making them versatile experts in the area of plenoptic imaging. During the project they will, in addition to specialized knowledge in their area of expertise, build a strong set of transferable skills. The gathered knowledge and skills will culminate in the double or joint doctoral degree diplomas, as illustrated in Figure 2. This will open various career opportunities either in academia or industry. By the end of their training, the Marie Sklodowska-Curie fellows will be prepared to lead the next generation of researchers and start a new cycle of breakthrough innovations.