Periodic Reporting for period 1 - SpeckleCorr (Computational imaging through scattering materials using speckle correlation)
Período documentado: 2023-01-01 hasta 2025-06-30
When viewed under coherent imaging conditions (e.g. laser illumination), scattering materials such as biological tissues
create noise-like images known as speckle. Despite their seemingly random nature, speckle patterns have strong statistical
correlation properties that are highly informative of the material producing them. These can be used to enable remarkable
imaging capabilities, not possible with current state of the art, for example seeing through highly scattering layers.
Unfortunately, realizing these in practical settings (tissue imaging, fluorescence microscopy) remains a challenge. Research
efforts are hindered by a lack of modeling tools, resulting in an incomplete understanding of speckle properties. This project
aims to use computational techniques from computer vision and computer graphics to greatly enhance our understanding of
speckle statistics, and significantly expand the scope of their applications. To this end, the project will explore algorithmic
tools newly-developed by the PI that can accurately and efficiently simulate speckle patterns, to formulate better models of
speckle formation. We will exploit our new understanding to develop new types of computational imaging systems that can
directly measure speckle correlation, rather than the traditional pipeline where one captures speckle images and estimates
their correlations algorithmically in post-processing. Finally, we will exploit these tools in multiple computational imaging
applications, including: (i) Acquiring material parameters: estimating the type, size and density of particles composing a
material of interest. (ii) Imaging fluorescent sources deep inside scattering tissue. (iii) Adaptive optics imaging.
Potential impact is anticipated in numerous areas where speckle-based imaging techniques hold promise, including
medicine (increased depth penetration of tissue imaging techniques) and material fabrication and analysis (accurate characterization
of scattering materials).
create noise-like images known as speckle. Despite their seemingly random nature, speckle patterns have strong statistical
correlation properties that are highly informative of the material producing them. These can be used to enable remarkable
imaging capabilities, not possible with current state of the art, for example seeing through highly scattering layers.
Unfortunately, realizing these in practical settings (tissue imaging, fluorescence microscopy) remains a challenge. Research
efforts are hindered by a lack of modeling tools, resulting in an incomplete understanding of speckle properties. This project
aims to use computational techniques from computer vision and computer graphics to greatly enhance our understanding of
speckle statistics, and significantly expand the scope of their applications. To this end, the project will explore algorithmic
tools newly-developed by the PI that can accurately and efficiently simulate speckle patterns, to formulate better models of
speckle formation. We will exploit our new understanding to develop new types of computational imaging systems that can
directly measure speckle correlation, rather than the traditional pipeline where one captures speckle images and estimates
their correlations algorithmically in post-processing. Finally, we will exploit these tools in multiple computational imaging
applications, including: (i) Acquiring material parameters: estimating the type, size and density of particles composing a
material of interest. (ii) Imaging fluorescent sources deep inside scattering tissue. (iii) Adaptive optics imaging.
Potential impact is anticipated in numerous areas where speckle-based imaging techniques hold promise, including
medicine (increased depth penetration of tissue imaging techniques) and material fabrication and analysis (accurate characterization
of scattering materials).
The research plan for this project included two main aims:
Aim 1: Develop algorithms and theory for simulating, modelling and understanding the statistics of coherent speckle patterns resulting from the propagation of coherent light through scattering volumes.
Aim 2: Developing imaging systems taking advantage of speckle statistics. This aim can be divided into 2 parts:
Aim 2.1: Develop imaging systems for acquiring 3D shape as well as intrinsic material parameters.
Aim 2.2: Develop Adaptive optics (Wavefront shaping) systems for fluorescent imaging.
We have made progress in all fronts as described below:
Aim 1:
Paper with Chen Bar (Optica 23) on simulating and modelling the statistics of speckles in dynamic media in the presence of blood flow.
Paper with Marina Alterman (Arxiv, recently accepted to cvpr25) on approximating the speckle aberration through a tissue volume using a small number of planar layers. This has wide implications on the design of future adaptive optics systems.
Aim 2.1
A group of three papers with colleagues from CMU on exploiting speckle statistics for depth measurements.
Aim 2.2:
This turns out to be the most successful part of the project, and where we will focus most research efforts in subsequent years. We summarize here the main achievements and elaborate in later parts of this report.
Wavefront shaping systems attempt to use a spatial light modulator inside an optical imaging system and use it to correct aberration resulting from tissue scattering. Despite the large potential to revolutionize tissue imaging, before our work wavefront shaping was only applied to various synthetic targets and not on real biological signals, due to the difficulty of estimating the
desired correction efficiently from weak biological sources.
Our recent research addressed the hardest obstacles, and for the first time managed to apply wave front shaping to image weak neurons deep inside scattering brain tissue. This was published in Nature Communication 2024.
The paper has raised a lot of excitement and a conference presentation we have given on it at the Optica imaging congress has won the best student paper award. Serious neurobiological groups are interested in applying it for their in-vivo brain imaging applications, and we are submitting an NIH proposal about this together with a team at Berkeley university.
Another recent paper from our lab, with Sagi Monin (recently uploaded to ArXiv) develops fast efficient algorithms for estimating such a wavefront shaping modulation correction.
Aim 1: Develop algorithms and theory for simulating, modelling and understanding the statistics of coherent speckle patterns resulting from the propagation of coherent light through scattering volumes.
Aim 2: Developing imaging systems taking advantage of speckle statistics. This aim can be divided into 2 parts:
Aim 2.1: Develop imaging systems for acquiring 3D shape as well as intrinsic material parameters.
Aim 2.2: Develop Adaptive optics (Wavefront shaping) systems for fluorescent imaging.
We have made progress in all fronts as described below:
Aim 1:
Paper with Chen Bar (Optica 23) on simulating and modelling the statistics of speckles in dynamic media in the presence of blood flow.
Paper with Marina Alterman (Arxiv, recently accepted to cvpr25) on approximating the speckle aberration through a tissue volume using a small number of planar layers. This has wide implications on the design of future adaptive optics systems.
Aim 2.1
A group of three papers with colleagues from CMU on exploiting speckle statistics for depth measurements.
Aim 2.2:
This turns out to be the most successful part of the project, and where we will focus most research efforts in subsequent years. We summarize here the main achievements and elaborate in later parts of this report.
Wavefront shaping systems attempt to use a spatial light modulator inside an optical imaging system and use it to correct aberration resulting from tissue scattering. Despite the large potential to revolutionize tissue imaging, before our work wavefront shaping was only applied to various synthetic targets and not on real biological signals, due to the difficulty of estimating the
desired correction efficiently from weak biological sources.
Our recent research addressed the hardest obstacles, and for the first time managed to apply wave front shaping to image weak neurons deep inside scattering brain tissue. This was published in Nature Communication 2024.
The paper has raised a lot of excitement and a conference presentation we have given on it at the Optica imaging congress has won the best student paper award. Serious neurobiological groups are interested in applying it for their in-vivo brain imaging applications, and we are submitting an NIH proposal about this together with a team at Berkeley university.
Another recent paper from our lab, with Sagi Monin (recently uploaded to ArXiv) develops fast efficient algorithms for estimating such a wavefront shaping modulation correction.
Certain tissue components, in particular neurons in the brain, are fluorescent meaning that we can use illumination at a certain wavelength to excite them, and they emit light at a new wavelength. This new color allows biologists to distinguish them from the rest of the tissue. However, when fluorescent components are located deep inside the tissue the emitted light scatters on its way to the camera and one can only see very aberrated images. Fluorescent signals are usually very weak and the aberrated images also suffer from a very low SNR. As a result, methods that attempt to computationally analyze the aberrated images and restore them are usually inapplicable at such low SNR. A promising alternative attempts to apply aberration correction inside the optical path, by placing a deformable mirror or a spatial light modulator, that reshapes the light waves and directs the photons emerging from a single point inside the tissue into a single sensor pixel. Despite the large potential to revolutionize tissue imaging, before our work wavefront shaping was only applied to various synthetic targets and not on real biological signals. This has to do with 4 major challenges. First one needs to be able to find the desired shape of the modulation correction using noninvasive feedback from the sample. Second, modulation estimation needs to work with weak sources under a very low SNR. Third, previous modulation estimation approaches involved very slow computation, but modulation estimation needs to happen fast enough to support the fast dynamics of a living tissue. Finally, since the scattering happens in a 3D volume, the desired correction varies spatially very fast, and each modulation supports the correction of an extremely small tissue window.
The line of research from our group in the last years addresses all these hard challenges.
First the work of Aizik, nature communications 2024 proposed a wavefront shaping system that can correct aberrations using the weak fluorescent signal emitted from neurons deep inside brain tissue. After years of wavefront shaping research by a large research community, this is really the first time that wavefront shaping was successfully applied on biological data. The paper received a lot of interest from the research community, as evident by the fact that it won the best paper award, and by the fact that serious researchers are interested in applying the technique in their lab.
Another recent important paper from our lab (Monin 25) develops a fast algorithm for estimating the desired correction using the ideas of optical computing.
Finally, the work of Alterman CVPR 2025 studies fundamental theory beyond a 3D wavefront-shaping correction and attempts to understand how to extend them to correct wide field of view images.
The line of research from our group in the last years addresses all these hard challenges.
First the work of Aizik, nature communications 2024 proposed a wavefront shaping system that can correct aberrations using the weak fluorescent signal emitted from neurons deep inside brain tissue. After years of wavefront shaping research by a large research community, this is really the first time that wavefront shaping was successfully applied on biological data. The paper received a lot of interest from the research community, as evident by the fact that it won the best paper award, and by the fact that serious researchers are interested in applying the technique in their lab.
Another recent important paper from our lab (Monin 25) develops a fast algorithm for estimating the desired correction using the ideas of optical computing.
Finally, the work of Alterman CVPR 2025 studies fundamental theory beyond a 3D wavefront-shaping correction and attempts to understand how to extend them to correct wide field of view images.