Periodic Reporting for period 1 - CONcISE (COmputatioNal Imaging as a training Network for Smart biomedical dEvices)
Berichtszeitraum: 2023-02-01 bis 2025-01-31
The aim of CONcISE is to develop novel, unconventional, multi-dimensional optical imaging techniques making use of structured-light, single-pixel imaging and adaptive, data-driven approaches for the detection and mapping of endogenous and exogenous contrast agents, such as absorption, scattering and fluorescence, in biological tissues with Visible and Near-Infrared Light (VIS-NIR). 11 Doctoral Candidates will develop three novel systems for relevant problems related to biomedical imaging:
1. SMART-DOT: a Diffuse-Optical Tomography for mapping absorption and scattering of thick biological tissues, using pulsed light together with structured illumination and compressive detection.
2. SMART-FLUO: a system for multispectral fluorescence imaging, using structured light illumination. The system aims to provide the physician with a fast-imaging tool to be used during guided surgery with exogenous contrast agents.
3. SMART-2PM: a system for high-resolution imaging of biological samples with wide-field 2-Photon Microscopy based on light-structured illumination and integrated detection. The system will combine new fs-light sources in the near-IR, sophisticated adaptive optics systems, temporal focusing, single-pixel imaging strategies, and computational imaging approaches based on machine learning for adaptive detection.
1. SMART-DOT: a Diffuse-Optical Tomography for mapping absorption and scattering of thick biological tissues, using pulsed light together with structured illumination and compressive detection.
2. SMART-FLUO: a system for multispectral fluorescence imaging, using structured light illumination. The system aims to provide the physician with a fast-imaging tool to be used during guided surgery with exogenous contrast agents.
3. SMART-2PM: a system for high-resolution imaging of biological samples with wide-field 2-Photon Microscopy based on light-structured illumination and integrated detection. The system will combine new fs-light sources in the near-IR, sophisticated adaptive optics systems, temporal focusing, single-pixel imaging strategies, and computational imaging approaches based on machine learning for adaptive detection.
The main results regarding the three systems are:
SMART-DOT for 3D mapping of absorption and scattering in thick diffuse biological samples.
- The hardware and software has been completely integrated for acquiring multiple-view time-resolved data with structured illumination and single-pixel camera detection.
- Model-based Bayesian reconstruction using Fourier transform in time-domain and Hadamard patterns based on a Finite-Element Method (FEM) solver.
- A deep study on the effect of the number of spatial and temporal frequencies has been performed.
- Measurements on a cylindrical diffusive solid phantom with different combinations of absorption and scattering perturbations have been acquired.
- A forward solver based on Monte Carlo code has been developed capable of simulating illumination and detection.
- A preliminary convolutional neural network has been developed on Monte Carlo simulated data for a sensor-to-image reconstruction capable of avoiding the model-based inversion.
SMART-FLUO.
- Development of a Single-Pixel Camera (SPC) system capable of acquiring multi-spectral, time-resolved, volumetric images of fluorescent specimens.
- A modular structure software, written in Python, to manage the instruments and measurement procedures. The system can be used to control both microscopic and mesoscopic systems.
- A Data Fusion algorithm to fuse the data set, obtained by combining structured light illumination for optical sectioning and single-pixel imaging for multispectral Fluorescence Lifetime Imaging (FLIM), in order to achieve the desired result of a high-resolution multidimensional (spatial, spectral and temporal) output.
- Development of a neural network-based approach for reconstructing images from compressed measurements for fluorescent microscopy.
- A deep study of the optimal combination of compression and data fusion algorithms towards real-time multidimensional data acquisition with SPC based system has been performed.
SMART-2PM.
- First prototype of the system for single-pixel microscopy with structured illumination that will be used as a platform for the whole nonlinear microscope (2PM). The system has been tested in linear microscopy modalities with both visible and infrared illumination and with biological specimens.
- Innovative single-pixel detection strategies for microscopy allowing to obtain quantitative phase images of transparent samples, multispectral images, and the application of data fusion techniques.
- Development of a first functional prototype of the all-fiber femtosecond light source in the near-IR (1500 nm) that will serve as the illumination system for the 2PM system.
- Adaptive optics module for 2PM based on multi-actuator adaptive lenses and wavefront sensor optimization algorithms. It has been integrated into the prototype of the microscope platform.
- Software framework, written in Python, to operate the microscope platform. It integrates modules to control the spatial light modulator, steering the adaptive optics module, controlling data acquisition, performing image reconstruction, and adapting the illumination pattern. It includes new methods for adaptive light sampling based on generative AI components.
SMART-DOT for 3D mapping of absorption and scattering in thick diffuse biological samples.
- The hardware and software has been completely integrated for acquiring multiple-view time-resolved data with structured illumination and single-pixel camera detection.
- Model-based Bayesian reconstruction using Fourier transform in time-domain and Hadamard patterns based on a Finite-Element Method (FEM) solver.
- A deep study on the effect of the number of spatial and temporal frequencies has been performed.
- Measurements on a cylindrical diffusive solid phantom with different combinations of absorption and scattering perturbations have been acquired.
- A forward solver based on Monte Carlo code has been developed capable of simulating illumination and detection.
- A preliminary convolutional neural network has been developed on Monte Carlo simulated data for a sensor-to-image reconstruction capable of avoiding the model-based inversion.
SMART-FLUO.
- Development of a Single-Pixel Camera (SPC) system capable of acquiring multi-spectral, time-resolved, volumetric images of fluorescent specimens.
- A modular structure software, written in Python, to manage the instruments and measurement procedures. The system can be used to control both microscopic and mesoscopic systems.
- A Data Fusion algorithm to fuse the data set, obtained by combining structured light illumination for optical sectioning and single-pixel imaging for multispectral Fluorescence Lifetime Imaging (FLIM), in order to achieve the desired result of a high-resolution multidimensional (spatial, spectral and temporal) output.
- Development of a neural network-based approach for reconstructing images from compressed measurements for fluorescent microscopy.
- A deep study of the optimal combination of compression and data fusion algorithms towards real-time multidimensional data acquisition with SPC based system has been performed.
SMART-2PM.
- First prototype of the system for single-pixel microscopy with structured illumination that will be used as a platform for the whole nonlinear microscope (2PM). The system has been tested in linear microscopy modalities with both visible and infrared illumination and with biological specimens.
- Innovative single-pixel detection strategies for microscopy allowing to obtain quantitative phase images of transparent samples, multispectral images, and the application of data fusion techniques.
- Development of a first functional prototype of the all-fiber femtosecond light source in the near-IR (1500 nm) that will serve as the illumination system for the 2PM system.
- Adaptive optics module for 2PM based on multi-actuator adaptive lenses and wavefront sensor optimization algorithms. It has been integrated into the prototype of the microscope platform.
- Software framework, written in Python, to operate the microscope platform. It integrates modules to control the spatial light modulator, steering the adaptive optics module, controlling data acquisition, performing image reconstruction, and adapting the illumination pattern. It includes new methods for adaptive light sampling based on generative AI components.
SMART-DOT
Progress beyond the state of the art:
- Analysis of the combined impact of spatial and temporal frequencies in DOT.
- Convolutional neural network for identifying and quantifying absorption and scattering perturbations in a diffusive medium.
Expected results:
- Devise an adaptive strategy for acquiring less data whilst maximizing the information during measurement stage.
- Refine the neural network including noise/variability of the experimental system, such as Instrumental Response Function, detector noise, etc.
SMART-FLUO
Progress beyond the state of the art:
- Development of a computational imaging strategy which allows fast 3D multispectral FLIM combining wide field structured illumination with time-resolved Single Pixel Camera detection together with data analysis algorithms based on compressed sensing and data fusion.
- Convolutional neural network for reconstructing multispectral time resolved images acquired by a fluorescent microscopy based on the Single Pixel Camera.
Expected results:
- Realization of real time multispectral FLIM systems working at a microscopic and mesoscopic scale exploiting different computational strategy such as data fusion and convolutional neural networks.
- Validation of the systems on phantoms and tissues to demonstrate its possible use for guided surgery.
SMART-2PM
Progress beyond the state of the art:
- Development of wide-field multiphoton microscopy techniques combining optimized structured illumination, an all-fiber near-IR femtosecond laser, flexible adaptive optics, multidimensional single-pixel detection, and adaptive compressive strategies.
- Introduction of intelligent automation into single-pixel microscopy through the development of innovative algorithms for adaptive pattern sampling, adaptive optics, and image reconstruction, powered by compressive sensing, data fusion, and neural networks.
Expected results:
- Novel nonlinear microscope system, featuring a cutting-edge near-IR femtosecond light source, dynamically programmable adaptive optics, and intelligent structured sampling, enabling high-speed, multidimensional imaging of biological samples with transformative spatial resolution and deep-tissue penetration.
- Quantitative validation of the system's performance through controlled experiments using phantoms and biological tissues labelled with specific fluorophores.
Potential impact of the CONcISE research project
Contribute to:
- Advanced multi-dimensional optical imaging techniques of biological tissues.
- Train a new generation of skilled multidisciplinary scientists with a strong vision of hardware/software integration, which is the key for innovative systems.
- Foster a network of scientific collaborations even beyond the duration of the project.
- Develop innovative and multidisciplinary approaches to doctoral programmes at the European level.
Progress beyond the state of the art:
- Analysis of the combined impact of spatial and temporal frequencies in DOT.
- Convolutional neural network for identifying and quantifying absorption and scattering perturbations in a diffusive medium.
Expected results:
- Devise an adaptive strategy for acquiring less data whilst maximizing the information during measurement stage.
- Refine the neural network including noise/variability of the experimental system, such as Instrumental Response Function, detector noise, etc.
SMART-FLUO
Progress beyond the state of the art:
- Development of a computational imaging strategy which allows fast 3D multispectral FLIM combining wide field structured illumination with time-resolved Single Pixel Camera detection together with data analysis algorithms based on compressed sensing and data fusion.
- Convolutional neural network for reconstructing multispectral time resolved images acquired by a fluorescent microscopy based on the Single Pixel Camera.
Expected results:
- Realization of real time multispectral FLIM systems working at a microscopic and mesoscopic scale exploiting different computational strategy such as data fusion and convolutional neural networks.
- Validation of the systems on phantoms and tissues to demonstrate its possible use for guided surgery.
SMART-2PM
Progress beyond the state of the art:
- Development of wide-field multiphoton microscopy techniques combining optimized structured illumination, an all-fiber near-IR femtosecond laser, flexible adaptive optics, multidimensional single-pixel detection, and adaptive compressive strategies.
- Introduction of intelligent automation into single-pixel microscopy through the development of innovative algorithms for adaptive pattern sampling, adaptive optics, and image reconstruction, powered by compressive sensing, data fusion, and neural networks.
Expected results:
- Novel nonlinear microscope system, featuring a cutting-edge near-IR femtosecond light source, dynamically programmable adaptive optics, and intelligent structured sampling, enabling high-speed, multidimensional imaging of biological samples with transformative spatial resolution and deep-tissue penetration.
- Quantitative validation of the system's performance through controlled experiments using phantoms and biological tissues labelled with specific fluorophores.
Potential impact of the CONcISE research project
Contribute to:
- Advanced multi-dimensional optical imaging techniques of biological tissues.
- Train a new generation of skilled multidisciplinary scientists with a strong vision of hardware/software integration, which is the key for innovative systems.
- Foster a network of scientific collaborations even beyond the duration of the project.
- Develop innovative and multidisciplinary approaches to doctoral programmes at the European level.