Polarized 3D Endoscopy

Project Information

P3D Endoscopy

Grant agreement ID: 101113391

DOI

10.3030/101113391

Project closed

EC signature date 6 March 2023

Start date 1 October 2023

End date 30 September 2025

Funded under

• European Research Council (ERC)

Total cost

No data

EU contribution

€ 150 000,00

Coordinated by

TEL AVIV UNIVERSITY
Israel

Periodic Reporting for period 1 - P3D Endoscopy (Polarized 3D Endoscopy)

Reporting period: 2023-10-01 to 2025-09-30

Summary of the context and overall objectives of the project

Motivation:
Endoscopy is a critical tool in modern medicine, enabling minimally invasive diagnosis and surgery for conditions ranging from colorectal cancer to gastrointestinal disorders. However, standard endoscopes are limited to capturing 2D intensity images (RGB), which often miss subtle tissue anomalies. Polarization information is a crucial missing modality; light polarization reveals properties of tissue microstructure—such as surface texture, collagen alignment, and early-stage lesions—that remain invisible to standard color cameras and the human eye. Adding polarization capabilities to endoscopy could significantly improve diagnostic accuracy and early detection rates.

Challenge:
a. The primary obstacle to integrating polarization imaging into clinical practice is physical size. Conventional polarization cameras rely on bulky optical assemblies, moving parts, or complex division-of-focal-plane sensors that sacrifice resolution. These solutions simply cannot fit into the constrained tip of a medical endoscope. To address this, our project turned to lensless imaging. By removing the bulky lens and replacing it with a simple, ultra-thin optical element (such as a diffuser or coded mask), the imaging system can be miniaturized to a scale suitable for insertion into the human body. This approach offers a potential path to the "Holy Grail" of a compact, disposable, and informative endoscopic sensor.

b. While lensless imaging solves the size constraint, it introduces a significant performance challenge. Lensless polarization reconstruction is an ill-posed computational problem; without a lens to focus light, the resulting images often suffer from low spatial resolution, noise, and a lack of high-frequency detail. On its own, a lensless polarization camera is "not good enough" to provide the reliable, high-fidelity imagery required for medical diagnosis.

Goal:
The core objective of this project was to overcome this quality barrier without sacrificing miniaturization. We observed that standard miniature RGB cameras are already mature, widely available, and easily integrated into endoscopes.
Consequently, we designed and built a prototype that combines a lensless polarization camera with a standard RGB camera. Our breakthrough lies in utilizing the high-quality structural information from the standard RGB image to "guide" the reconstruction of the polarization data. By fusing the detailed spatial data from the standard camera with the raw polarization signals from the lensless sensor, we can computationally recover high-fidelity polarization images.

Impact:
This approach allows us to generate a next-generation endoscope that provides rich polarization information alongside standard RGB video, with negligible impact on the device's size or cost. The successful development of this Guided Lensless Polarization Imaging system (validated in our CVPR 2026 acceptance) paves the way for advanced medical devices that can detect pathologies earlier and more accurately than ever before.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Work Performed and Main Achievements:
The primary technical objective of this project was to overcome the physical limitations of incorporating polarization imaging into endoscopy. To achieve this, the work was divided into three main technical pillars:
(1) Optical System Design and Miniaturization
(2) Deep Learning Algorithm Development
(3) Prototype Fabrication and Validation.

Outcomes of the Action:
The project has resulted in the following key technical outcomes:
a. A Validated "RGB-Guided" Architecture: We have established a new paradigm for medical imaging where a standard endoscopic camera acts as a structural guide for auxiliary, miniaturized sensors. This proves that high-quality polarization imaging does not require a large lens, provided a standard color image is available.
b. A Functional Lensless Polarization Prototype: We successfully fabricated a physical prototype that eliminates the need for bulky polarization optics. The device uses simple, low-cost optical elements (diffuser and film) that are scalable to mass manufacturing.
c. Advanced Reconstruction Software: A proprietary deep-learning pipeline was developed that solves the ill-posed problem of lensless imaging by fusing multi-modal data (RGB + Polarization).
d. Scientific Dissemination: The methodology and results were accepted for publication at CVPR 2026 findings. Note that CVPR is a top-tier computer vision conference, validating the scientific novelty and rigor of the approach.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Key Results:
1. Validated Prototype: We fabricated a functional prototype of a lensless polarization camera using a simple diffuser and a custom-patterned polarization mask. This hardware design eliminates the bulky lens assemblies found in traditional polarization cameras.
2. Reconstruction Algorithm: We developed a proprietary Deep Learning framework (published in CVPR 2026 findings) that uses a "guide" image from a standard color camera to reconstruct high-fidelity polarization data. This method demonstrated superior performance (higher PSNR and SSIM) compared to state-of-the-art physics-based solvers and unguided deep learning methods.
3. Proof of Generalization: The system was successfully tested on both synthetic datasets and real-world scenes (including plastic and metallic objects with internal stress), proving it can recover fine polarization details—such as surface texture and material composition—that are invisible to the naked eye.


Potential Impacts:

The results of this project hold significant potential to transform the field of medical endoscopy and diagnostic imaging.

1. Clinical Impact: Enhanced Diagnostic Capabilities:
The integration of polarization imaging into endoscopy provides a "new sense" for physicians. Polarization contrast can reveal tissue properties that standard RGB cameras miss, such as collagen fiber alignment, surface roughness, and early-stage pre-cancerous lesions. By providing this data in high resolution, the project paves the way for earlier detection of pathologies (e.g. colorectal polyps), better delineation of tumor margins during surgery, and a reduction in missed diagnoses.

2. Industrial Impact: Miniaturization and Cost-Efficiency:
The project successfully demonstrated that bulky optical hardware can be replaced by computational imaging. This has a profound impact on medical device manufacturing. We took into account the following important factors:
Size: The lensless sensor is thin and compact, making it viable for the tip of a standard endoscope (diameter < 10mm) or even smaller catheter-based devices.
Cost: The optical components (diffuser and polarization film) are passive and inexpensive to manufacture compared to precision micro-lenses. This ensures the technology can be adopted without significantly increasing the cost of medical devices.

3. Scientific Impact: AI-Driven Optical Design
The project validates a "hybrid" approach to imaging where hardware and software are co-designed. It demonstrates that standard sensors (RGB) can act as structural anchors for novel sensors (Polarization). This methodology can be extended to other modalities, such as hyperspectral or thermal imaging, influencing future research in computer vision and medical optics.

Photos of our designed system