Ultra-adaptive holographic computational imaging for seeing through random scatter, fibers, and around corners

Project Information

See-Beyond

Grant agreement ID: 101002406

DOI

10.3030/101002406

EC signature date 23 February 2021

Start date 1 October 2021

End date 30 September 2026

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 999 875,00

EU contribution

€ 1 999 875,00

Coordinated by

THE HEBREW UNIVERSITY OF JERUSALEM
Israel

Periodic Reporting for period 2 - See-Beyond (Ultra-adaptive holographic computational imaging for seeing through random scatter, fibers, and around corners)

Reporting period: 2023-04-01 to 2024-09-30

Summary of the context and overall objectives of the project

Optical imaging systems play an instrumental role in modern life, from smartphones and automotive cameras to microscopes that are critical for clinical diagnostics. However, even the most advanced imaging systems are still extremely limited in their penetration depth when imaging through common complex inhomogeneous media such as fog and biological tissues. The reason for this fundamental limitation is the inherent scattering of light in such complex samples. Examples for impacted application span many applicative fields, from microscopy, through laser-based imaging systems in autonomous vehicles and aircrafts, to astronomical observations, with important societal impacts. One common example is the requirement to perform invasive, and sometime dangerous, biopsies, where the suspected tissue is extracted to be studied with a conventional optical microscope, since it is currently not possible to obtain an optical image of the tissue at depth.

While light scattering is random, it is deterministic, and it can, in theory, be compensated for by appropriately reshaping the scattered light fields, in a concept termed ‘wavefront-shaping’. However, in practice, it is extremely challenging to impossible to perform such corrections for three fundamental reasons: first, one need to know the exact form of the correction pattern; second, it is experimentally impractical to control the scattered optical fields with the millions to billions of required control parameters; and third, physical wavefront shaping devices are two-dimensional (2D) flat devices that cannot correct volumetric three-dimensional (3D) scattering.

In this project, we propose to remove these fundamental barriers and unleash the full applicative potential of wavefront-shaping by shifting the burden from the physical hardware to a digital, naturally-parallelizable computational scattering compensation process. Our novel concept is made possible by the combination of our recent discovery that the wavefront-correction can be found by adaptive optimization of an image quality metric, similar to a multi-parameter ‘autofocus’, and the huge increase in computational power, allowing the unprecedented capability of rapidly digitally recording, storing, and processing large datasets of scattered optical fields. Specifically, we aim at developing a computational correction technique that is based on rapid holographic recording of several scattered light fields. The developed methodology would be utilized to correct optical scattering in several applications, ranging from optical micro-endoscopy, through microscopy, to imaging through diffusive barriers, and non-line-of-sight imaging using light reflected from rough surfaces.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During this reporting period, significant progress has been achieved across the various research domains outlined in our project plan, resulting in several publications in leading peer-reviewed journals. Below is a summary of the key advancements:

1) Lensless microendoscopy via ptychography: substantial progress has been made in developing an innovative method for imaging microscopic structures through thin, flexible fibers, without the conventional use of bulky lenses. The studied technique, known as ptychography, utilizes advanced computational algorithms to reconstruct complex light fields from conventional images captured by standard cameras. Additionally, unforeseen avenues for research have emerged during our exploration of the laser-scanning acquisition approach, as detailed below.
2) Algorithmic development: Two novel algorithms have been developed. One is based on the parallel optimization of millions of wavefront correction parameters, leveraging optimization tools that were developed for training large neural networks. In experiments and simulations the algorithm demonstrated exceptional robustness and optimization speed, enhancing imaging capabilities. The second developed algorithm is an adaptation and extension of the state of the art scattering compensation algorithm to image incoherent fluorescent samples, rather than reflective coherent samples considered so far. Another implemented advancement is a memory-efficient implementation of this state-of-the-art algorithm, which allows for the first time to reconstruct megapixel-scale images.
3) 3D lensless endoscopy and deep-tissue imaging via computational wavefront shaping: Successful experiments have been conducted in holographic computational imaging through miniature commercial fibers, and through multiple scattering samples, surpassing initial expectations. Imaging in complex volumetric scattering samples was demonstrated by either performing a computational volumetric correction, or by utilizing focused ultrasound waves to select specific small regions within the sample for imaging, simplifying the correction of distorted images and resulting in clearer representations of deep lying targets.
4) Non-line-of-sight (NLOS) imaging: we have been able to demonstrate computational localization of moving targets around the corner by recording random acoustic waves that are reflected off nearby walls.

In addition, our research yielded several unexpected discoveries:
1) The research on endoscopic ptychography led to the adaptation of a technique known as 'image scanning microscopy' to lensless fluorescence endoscopy, allowing improved resolution and signal collection efficiency.
2) A new mathematical formulation allowed us to extend computational scattering correction to incoherent fluorescence microscopy, and to photoacosutic tomography.
3) By placing a miniature partially-reflective mirror next to a commercial thin fiber distal end, we have been able to perform real-time holography through dynamically bent fibers, allowing video-rate lensless endoscopic imaging.
4) We have developed a novel unique device that is able to near-perfectly absorb complex light fields, surpassing the state-of-the-art “coherent perfect absorbers” systems by more than three orders of magnitude. This breakthrough may have interesting implications for detectors, modulators, and wireless energy transmission.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

We have been able to progress significantly beyond the state of the art in several aspects and directions. The notable advancements are:
1) Our image-guided holographic computational wavefront shaping approach has surpassed the state-of-the-art number of correction parameters (known as ‘degrees of freedom’ or ‘modes’) by more than an order of magnitude. Most importantly, this was achieved using a three orders of magnitude lower number of measurements (i.e. images), opening the path to rapid acquisitions, as required when tackling dynamic distortions, present in dynamically varying samples, such as biological tissues.
2) Our memory-efficient implementation of the state of the art scattering compensation algorithm has allowed correction and reconstruction of megapixel-scale images, more than two orders of magnitude larger pixel-count than the state of the art. In addition, our novel cross-correlation (covariance-matrix) based approach for fluorescence imaging through scattering layers allowed imaging of orders of magnitude more complex-shaped objects than the previous state-of-the-art, using a smaller number of measurements.
3) We have demonstrated lensless endoscopic microscopy of fluorescent targets with a light collection efficiency and resolution that surpass the current state-of-the-art by adapting the technique known as ‘image scanning microscopy’, originally developed for confocal microscopy.
4) Our ‘distal-holography’ technique allows imaging through dynamically bent lensless fibers without the need to compensate for wavefront distortions, using a much simpler experimental setup, and without any adaptive computational algorithm.
5) The novel ‘Massively degenerate coherent perfect absorber’ device that we have developed surpass the state-of-the-art systems by being able to near-perfectly absorb more than three orders of magnitude more complex optical fields. This advancement was recognized by being selected as one of the ‘top 10 breakthroughs in Physics for 2022’ by PhysicsWorld.

The expected results until the end of the project include the refinement and optimization of the above techniques in terms of acquisition and processing speed, and most importantly, in the ability to tackle volumetric scattering, which requires a more complex wavefront correction scheme. Once developed and matured, we will aim at testing the different techniques on various samples, including biological tissues.

See-Beyond computational correction of fluorescence microscopy through scattering layers