Periodic Reporting for period 2 - NEURAL SPICING (Neural Spectral Image Decoding)
Okres sprawozdawczy: 2022-11-01 do 2024-04-30
Had we the capacity to image the molecular tissue composition in real time, without exposing the patient or clinical staff to radiation, then this would revolutionize healthcare. Spectral imaging techniques, such as multispectral diffuse reflectance imaging and photoacoustic imaging, have the potential to recover important tissue properties including oxygenation, temperature, and the concentration of water. However, decades of research invested in solving the inverse problem of reconstructing clinically relevant tissue properties from spectral measurements have failed to produce solutions that can quantify tissue parameters robustly in a clinical setting. Initial attempts to address the limitations of model-based approaches with machine learning were hampered by the absence of labeled reference data needed for supervised algorithm training. While the simulation of training data can potentially address this bottleneck, the domain gap between real and simulated images remains a large unsolved challenge.
This project (NEURAL SPICING) at the intersection of physics, biology, medicine, and data science bridges the domain gap with a "learning-to-simulate" approach and reframes the quantification problem as a decoding problem that can be tackled with physics-constrained neural networks. It is based on the hypothesis that unlabeled and weakly labeled measurement data can be leveraged to improve the realism of simulated spectral images and ultimately the quantification accuracy. In building upon modern machine learning techniques, the concept naturally allows different sources of uncertainty to be handled or even exploited and enables imaging systems to learn from their experience.
The proposed second generation of spectral imaging is inherently safe and low-cost and could thus pave the way for a new era in interventional healthcare, with clinical applications ranging from cancer diagnosis to the staging and therapy of cardiovascular and inflammatory diseases.
This project (NEURAL SPICING) at the intersection of physics, biology, medicine, and data science bridges the domain gap with a "learning-to-simulate" approach and reframes the quantification problem as a decoding problem that can be tackled with physics-constrained neural networks. It is based on the hypothesis that unlabeled and weakly labeled measurement data can be leveraged to improve the realism of simulated spectral images and ultimately the quantification accuracy. In building upon modern machine learning techniques, the concept naturally allows different sources of uncertainty to be handled or even exploited and enables imaging systems to learn from their experience.
The proposed second generation of spectral imaging is inherently safe and low-cost and could thus pave the way for a new era in interventional healthcare, with clinical applications ranging from cancer diagnosis to the staging and therapy of cardiovascular and inflammatory diseases.
Technical foundation: As the foundation of this project, we acquired and annotated in vivo spectral imaging data of various organs, pathologies and modalities across three species (humans, porcines, rats) in close collaboration with clinical partners from Heidelberg University Hospital and the Städtisches Klinikum Karlsruhe. Based on these data, we demonstrated that deep learning-based tissue discrimination is feasible with high accuracy for both hyperspectral imaging and multispectral photoacoustic imaging.
The acquired knowledge enabled us to build a highly realistic spectral image synthesis pipeline. As a core component, it comprises digital twins of various spectral imaging devices that imitate the behavior of the real devices in the virtual environment as closely as possible. Our software contribution “SIMPA: an open-source toolkit for simulation and image processing for photonics and acoustics” leverages the knowledge on tissue properties as well as the digital device twins for simulation of diffuse reflectance images and photoacoustic tomography images. As a particular highlight, we showed that enhancing traditional physics-based simulation pipelines with learnable components substantially reduces the domain gap between simulated and real spectral imaging data. Finally, we demonstrated that our synthetic data enables the training of uncertainty-aware neural networks that can regress important functional tissue parameters such as tissue oxygenation.
Clinical applications: The methodology developed so far already allowed us to explore first clinical applications. These ranged from fully-automatic tissue differentiation for autonomous robotics to the optimization of surgical techniques and real-time perfusion monitoring in the operating room. An unforeseen highlight in this context was related to our invertible neural network-based approach to uncertainty quantification in spectral imaging. Our out-of-distribution detection method turned out to be the enabling method for automatic real-time detection of ischemia in patients undergoing partial kidney resection. With this contribution, we are the first to provide real-time assistance in laparoscopic surgery on the basis of spectral imaging.
Validation framework: Flaws in machine learning validation are an underestimated global problem. In the process of desiging a validation strategy for this project that adequately reflects the underlying clinical needs, we created “Metrics Reloaded”, a recommendation framework that guides researchers through the process of selecting appropriate image analysis validation metrics in a problem-aware manner. While an initial focus was put on tissue discrimination and localization tasks, we are currently extending the framework to the tasks of image reconstruction and synthesis in close collaboration with the International Photoacoustic Standardization Consortium (IPASC).
The acquired knowledge enabled us to build a highly realistic spectral image synthesis pipeline. As a core component, it comprises digital twins of various spectral imaging devices that imitate the behavior of the real devices in the virtual environment as closely as possible. Our software contribution “SIMPA: an open-source toolkit for simulation and image processing for photonics and acoustics” leverages the knowledge on tissue properties as well as the digital device twins for simulation of diffuse reflectance images and photoacoustic tomography images. As a particular highlight, we showed that enhancing traditional physics-based simulation pipelines with learnable components substantially reduces the domain gap between simulated and real spectral imaging data. Finally, we demonstrated that our synthetic data enables the training of uncertainty-aware neural networks that can regress important functional tissue parameters such as tissue oxygenation.
Clinical applications: The methodology developed so far already allowed us to explore first clinical applications. These ranged from fully-automatic tissue differentiation for autonomous robotics to the optimization of surgical techniques and real-time perfusion monitoring in the operating room. An unforeseen highlight in this context was related to our invertible neural network-based approach to uncertainty quantification in spectral imaging. Our out-of-distribution detection method turned out to be the enabling method for automatic real-time detection of ischemia in patients undergoing partial kidney resection. With this contribution, we are the first to provide real-time assistance in laparoscopic surgery on the basis of spectral imaging.
Validation framework: Flaws in machine learning validation are an underestimated global problem. In the process of desiging a validation strategy for this project that adequately reflects the underlying clinical needs, we created “Metrics Reloaded”, a recommendation framework that guides researchers through the process of selecting appropriate image analysis validation metrics in a problem-aware manner. While an initial focus was put on tissue discrimination and localization tasks, we are currently extending the framework to the tasks of image reconstruction and synthesis in close collaboration with the International Photoacoustic Standardization Consortium (IPASC).
With this project, we are the first to phrase the spectral image quantification problem as a learning-to-simulate task. While prior work has focused on physics-based simulation pipelines, NEURAL SPICING has seen us pioneer a deep learning-based approach to spectral image synthesis that enables us to challenge the published prior knowledge on tissue properties and thus to continuously improve the realism of the synthetic images based on new data.
Clinically, based on our novel image analysis methodology, we were the first to explore the benefit of spectral imaging for several use cases ranging from the optimization of surgical techniques to the fully-automatic detection of ischemia in laparoscopic surgery.
The remainder of the project timeline will see us focus on including further learning components in each step of the spectral image simulation pipeline to achieve a fully differentiable pipeline that can continuously be improved based on new incoming data. This will enable us to explore further clinical applications which rely on the exact quantification of tissue parameters.
Clinically, based on our novel image analysis methodology, we were the first to explore the benefit of spectral imaging for several use cases ranging from the optimization of surgical techniques to the fully-automatic detection of ischemia in laparoscopic surgery.
The remainder of the project timeline will see us focus on including further learning components in each step of the spectral image simulation pipeline to achieve a fully differentiable pipeline that can continuously be improved based on new incoming data. This will enable us to explore further clinical applications which rely on the exact quantification of tissue parameters.