Periodic Reporting for period 4 - GlobalBioIm (Global integrative framework for Computational Bio-Imaging)
Berichtszeitraum: 2021-04-01 bis 2022-03-31
One needs vast amounts of acquired data (Fourier samples for MRI, projections for x-ray imaging) to reconstruct high-quality medical and biological images. Most imaging systems divide the data in batches (2D slices, typically), which are then reconstructed one at a time. To reconstruct the underlying signal in its entirety, we propose an alternative strategy in which we consider the data jointly, relying on a precise reallocation of the measurements in 3D space/time. This joint reconstruction results in a very large-scale optimization that exploits the redundancy of the signal (sparsity, spatiotemporal correlations) to improve the solution. Our goal was to demonstrate the feasibility of such a big-data integration by leveraging recent advances in optimization and data science, and to document its advantages in terms of quality and resolution, setting the stage for the design of future bioimaging systems.
We are proposing a unifying framework for the development of such next-generation reconstruction algorithms. It involves a clear separation between the physical (forward model) and signal-related (regularization, incorporation of prior constraints) aspects of the problem. The three pillars of our formulation are: an operator algebra with a corresponding set of fast linear solvers; a statistical framework for the principled derivation of reconstruction methods; advanced learning schemes for the optimization of algorithmic components. These core technologies are incorporated into a modular software library (GlobalBioIm) that features key components to implement and test iterative reconstruction algorithms. We are applying our framework to improve upon the state of the art in a variety of modalities: x-ray and optical projection tomography; various forms of optical microscopies; single-particle analysis in cryo-electron tomography; and real-time MRI.
In short, the two primary objectives of GlobalBioIm were
(i) to improve the resolution and quality of images obtained from a wide range of modalities using data integration and advanced processing techniques;
(ii) to provide imaging scientists with a unified software framework that facilitates and speeds up the development of state-of-the-art image-reconstruction algorithms.
Due to the widespread use of imaging in medicine and biology, the project should contribute to the advancement of science and to the improvement of image-based diagnostics (e.g. radiology).
We are proposing a unifying framework for the development of such next-generation reconstruction algorithms. It involves a clear separation between the physical (forward model) and signal-related (regularization, incorporation of prior constraints) aspects of the problem. The three pillars of our formulation are: an operator algebra with a corresponding set of fast linear solvers; a statistical framework for the principled derivation of reconstruction methods; advanced learning schemes for the optimization of algorithmic components. These core technologies are incorporated into a modular software library (GlobalBioIm) that features key components to implement and test iterative reconstruction algorithms. We are applying our framework to improve upon the state of the art in a variety of modalities: x-ray and optical projection tomography; various forms of optical microscopies; single-particle analysis in cryo-electron tomography; and real-time MRI.
In short, the two primary objectives of GlobalBioIm were
(i) to improve the resolution and quality of images obtained from a wide range of modalities using data integration and advanced processing techniques;
(ii) to provide imaging scientists with a unified software framework that facilitates and speeds up the development of state-of-the-art image-reconstruction algorithms.
Due to the widespread use of imaging in medicine and biology, the project should contribute to the advancement of science and to the improvement of image-based diagnostics (e.g. radiology).
The work is divided into two main parts: a conceptual/enabling component that takes the form of a unified software library (GlobalBioIm) which is common to all sub-projects; and an application-oriented part that is devoted to the development of specialized reconstruction algorithms for a variety of imaging modalities.
Our methods are made available to the community as part of an integrative software framework (GlobalBioIm) for the reconstruction of biomedical images (https://biomedical-imaging-group.github.io/GlobalBioIm/). The leading idea of our system is to decouple the physical aspect of the problem, which is application-specific, from the so-called “regularization” of the solution. Regularization dictates how to properly constrain the reconstruction and is often the same across modalities. Once equipped with these two components, the user has to specify a suitable cost functional, together with an adequate method of optimization. The desired image-reconstruction algorithm can then be automatically assembled by GlobalBioIm. Since the critical part of this approach is the choice of the regularization, we have also performed a theoretical investigation of its effect on image reconstruction. As an alternative to regularization, we have also made use of convolutional neural networks, which has enabled us to improve the quality of the reconstructed images, and this, far beyond the state of the art.
On the application side, our contributions include
- high-end reconstruction methods for x-ray CT and optical projection tomography;
- a new algorithm for structured-illumination microscopy that uses computational sectioning;
- a comparative assessment of the resolution of various forms of superresolution microscopies;
- a multiresolution framework for 3D reconstruction with joint refinement of the poses;
- a novel paradigm (CryoGAN) for the 3D reconstruction of single particles in cryo-EM without pose estimation;
- the improvement of the axial resolution of fluorescence microscopy through the combination of multiple views of particle replicates that differ by their orientation;
- a novel method for diffraction tomography based on the solution of the inverse scattering problem;
- a novel reconstruction framework for dynamic MRI based on deep prior that produces images of astonishing quality.
Our methods are made available to the community as part of an integrative software framework (GlobalBioIm) for the reconstruction of biomedical images (https://biomedical-imaging-group.github.io/GlobalBioIm/). The leading idea of our system is to decouple the physical aspect of the problem, which is application-specific, from the so-called “regularization” of the solution. Regularization dictates how to properly constrain the reconstruction and is often the same across modalities. Once equipped with these two components, the user has to specify a suitable cost functional, together with an adequate method of optimization. The desired image-reconstruction algorithm can then be automatically assembled by GlobalBioIm. Since the critical part of this approach is the choice of the regularization, we have also performed a theoretical investigation of its effect on image reconstruction. As an alternative to regularization, we have also made use of convolutional neural networks, which has enabled us to improve the quality of the reconstructed images, and this, far beyond the state of the art.
On the application side, our contributions include
- high-end reconstruction methods for x-ray CT and optical projection tomography;
- a new algorithm for structured-illumination microscopy that uses computational sectioning;
- a comparative assessment of the resolution of various forms of superresolution microscopies;
- a multiresolution framework for 3D reconstruction with joint refinement of the poses;
- a novel paradigm (CryoGAN) for the 3D reconstruction of single particles in cryo-EM without pose estimation;
- the improvement of the axial resolution of fluorescence microscopy through the combination of multiple views of particle replicates that differ by their orientation;
- a novel method for diffraction tomography based on the solution of the inverse scattering problem;
- a novel reconstruction framework for dynamic MRI based on deep prior that produces images of astonishing quality.
GlobalBioIm has succeeded in demonstrating that data integration coupled with advanced data processing can dramatically increase image quality. Equivalently, it has confirmed that it is possible to produce acceptable images from fewer (or noisier) measurements (compressed sensing), which translates into a corresponding reduction of the radiation dosage.
Another important point is that the methodologies that have been developed for a given modality are, in most cases, transposable to others. In fact, we have gone one step further by developing a set of universal computational modules that can be easily combined in order to produce a modality-specific reconstruction pipeline at the level of the state of the art, with minimal development effort.
Our experiments also demonstrate the great potential of machine-learning techniques (in particular, deep convolutional neural networks) for image reconstruction. This is undoubtedly a most promising direction of research that needs to be further explored and applied/tested with more diverse modalities.
Our theoretical investigation of the effect of regularization has resulted in a new “representer theorem” that characterizes the solution of a broad category of functional optimization problems in Banach spaces. Interestingly, these problems are not only relevant to imaging, but also to machine learning. In particular, the theorem allows us to infer the RKHS energy-minimization properties of classical kernel estimators, which underlie a central theme in classical machine learning. Much more surprising and rewarding is the fact that it allows us to identify sparsity-promoting energy functionals whose minimization results in parametric solutions that are compatible with the structure of neural networks, including the ones that use ReLU activations. This suggests the possibility of linking the design of neural architectures with functional optimization, which offers a novel point of view.
Another important point is that the methodologies that have been developed for a given modality are, in most cases, transposable to others. In fact, we have gone one step further by developing a set of universal computational modules that can be easily combined in order to produce a modality-specific reconstruction pipeline at the level of the state of the art, with minimal development effort.
Our experiments also demonstrate the great potential of machine-learning techniques (in particular, deep convolutional neural networks) for image reconstruction. This is undoubtedly a most promising direction of research that needs to be further explored and applied/tested with more diverse modalities.
Our theoretical investigation of the effect of regularization has resulted in a new “representer theorem” that characterizes the solution of a broad category of functional optimization problems in Banach spaces. Interestingly, these problems are not only relevant to imaging, but also to machine learning. In particular, the theorem allows us to infer the RKHS energy-minimization properties of classical kernel estimators, which underlie a central theme in classical machine learning. Much more surprising and rewarding is the fact that it allows us to identify sparsity-promoting energy functionals whose minimization results in parametric solutions that are compatible with the structure of neural networks, including the ones that use ReLU activations. This suggests the possibility of linking the design of neural architectures with functional optimization, which offers a novel point of view.