Practical Imaging and Inversion of Transient Light Transport

Project Information

ECHO

Grant agreement ID: 802192

DOI

10.3030/802192

Project closed

EC signature date 29 October 2018

Start date 1 December 2018

End date 30 November 2023

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 525 840,00

EU contribution

€ 1 525 840,00

Coordinated by

RHEINISCHE FRIEDRICH-WILHELMS-UNIVERSITAT BONN
Germany

Periodic Reporting for period 4 - ECHO (Practical Imaging and Inversion of Transient Light Transport)

Reporting period: 2023-06-01 to 2023-11-30

Summary of the context and overall objectives of the project

Imaging is a crucial component in almost any modern technical application. Common to all imaging techniques is the issue of visibility; objects that are not within the direct line of sight cannot be detected and characterized. For instance, a medical endoscope cannot see a tumor that is located one bend beyond the device's reach, and a self-driving automobile cannot sense and react to a pedestrian that is about to enter the street from behind a large truck.

The ECHO project was located at the intersection between fundamental research and applications engineering. Its focus was on emerging optical sensing technologies that provide new insights into the world around us and can even look "around the corner". ECHO's goal is to enable practical approaches to time-of-flight, or "transient" sensing, i.e. the recording of light and its propagation through space and time. While it has been shown that transient images can do astonishing imaging feats such as imaging objects that are only accessible through indirect reflections, most setups so far have been expensive and constrained to controlled laboratory settings. Overcoming these limitations could enable practical solutions to unsolved imaging challenges across all application fields, including medical, automotive, search/rescue and industrial imaging.

To work toward this goal, ECHO aimed to
- make transient imaging available at low cost and under practical conditions,
- further our understanding of transient images and their mathematical structure,
- use this insight to develop better, more efficient analysis techniques for transient data, and to
- develop low-cost, practical application demonstrators based on these developments and insights.

Over the course of the project, we have been able to achieve significant improvements regarding the practicality of transient imaging. Among other things, we have been able to demonstrate the use of recently emerging and extremely affordable consumer-grade single-photon sensors for
many traditional applications of transient images: range imaging, material
classification and non-line-of-sight object tracking.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

TI/CTI/MPI Databases
A central goal of ECHO is to deepen our understanding of the structure of transient image data, to develop novel analysis techniques (whether by expert design or by machine learning), and to assess the performance and problems of existing analysis techniques. It is therefore essential to have access to reference data that reflects the statistical behaviour of the addressed problem class.
Using an efficient simulation scheme for transient light transport (see following section), we have been able to generate large amounts of reference data for the non-line-of-sight reconstruction problem that draw from various sources of 3D scene models: shape databases, 3D scan databases. We further developed augmentation strategies for our clean synthetic data to reflect the statistical variation in real-world sensor data.
We further began the construction of a versatile experimental setup for capturing high=quality light echoes. This work has experienced significant delay due to lab access restrictions (COVID pandemic) but is currently being resumed.
Forward Modelling
We have developed an extremely performant forward simulation engine for transient light transport that takes into account optical effects such as self-occlusion in the hidden parts of the scene, or non-trivial reflectance distributions.
Non-Line-of-Sight Reconstruction
A relatively new and extremely promising use case for transient imaging lies in its ability to extend the direct line of sight. The research direction of non-line-of-sight imaging aims to devise sensing strategies and reconstruction techniques for objects that are located “around the corner” and only accessible through indirect reflections.
Using the forward models developed within ECHO, we were the first to demonstrate a non-line-of-sight reconstruction approach that is based on a physically accurate image formation model. We further trained a machine learning system to predict depth maps of non-line-of-sight scenes within a few milliseconds, alleviating for the first time the heavy time demand of existing reconstruction approaches.
Computer Graphics Models
In order to achieve the project goals, novel highly accurate and performant computer graphics had to be developed. In addition to their use within the project scope, these constitute significant improvements on the state of the art in the area of computer graphics and led to award-winning publications as well as inclusion in the most widely used open-source graphics software package (Blender).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

We have been able to demonstrate the first non-line-of-sight reconstruction technique that is based on a physically accurate image formation model [Iseringhausen and Hullin 2020]. We further used the same forward simulation, augmented by a parametric sensor model, to perform non-line-of-sight reconstructions within a few milliseconds.
We have introduced the first dedicated calibration scheme for non-line-of-sight sensing setups. We have demonstrated a sensor fusion approach to combine low-resolution transient images with high-resolution intensity images into high-resolution transient images.
We have constructed a hardware-software system based on affordable, consumer-grade sensors that can implement iconic applications for transient data, such as pointwise material classification, depth imaging, and object tracking around a corner.
After completion of the project, we have not only managed to make non-line-of-sight sensing systems significantly more accessible and more robust and performant due to improved signal formation models and better hardware designs. The detail models that were developed to achieve these goals have also found use as components in computer graphics systems.

Extremely small and affordable optical sensors can be used to implement challenging sensing tasks.