Medical Hyperspectral Image and Video Processing and Interpretation via Constrained Matrix and Tensor Factorization

Project Information

HyPPOCRATES

Grant agreement ID: 844290

DOI

10.3030/844290

Project closed

EC signature date 27 March 2019

Start date 7 October 2019

End date 6 October 2022

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 259 808,64

EU contribution

€ 259 808,64

Coordinated by

ETHNIKO ASTEROSKOPEIO ATHINON
Greece

Periodic Reporting for period 2 - HyPPOCRATES (Medical Hyperspectral Image and Video Processing and Interpretation via Constrained Matrix and Tensor Factorization)

Reporting period: 2021-10-07 to 2022-10-06

Summary of the context and overall objectives of the project

HyPPOCRATES project focused on the development and analysis of state-of-the-art efficient and reliable machine learning methods for processing multidimensional signals such as hyperspectral imaging data. Specifically, we emphasized the development of novel matrix and tensor decomposition methods that allow us to represent high-dimensional and large-scale signals such as hyperspectral images in such a way to capture the inherent structure of data. In doing so, we efficiently addressed several problems that show up in image processing and understanding, successfully carrying out difficult tasks such as denoising, compression, unmixing, clustering, etc. We came up with methods that process whole batches of data as well as methods amenable to handing data that are becoming available in a streaming fashion. The novelty of the proposed approach lies in the coupling of the decomposition task with model selection. This is implemented via the use of sophisticated regularization terms that are incorporated in newly formulated optimization problems. Those regularizers are carefully devised in such a way to capture the structure of the signals, thus allowing us to infer the ranks of matrices and tensors while learning the matrix/tensor factors.

Nowadays, massive amounts of imaging data are generated by various sources e.g. cellphone cameras, medical imaging devices, satellite imaging sensors, etc. The development of efficient algorithmic tools that will be capable of efficient processing and extracting valuable knowledge out of these data thus becomes a pressing need. At the same time, in several domains e.g. medical imaging, the reliability of algorithms is of utmost importance. Specifically, it is crucial to come up with some guarantees regarding the performance of the algorithms that will allow us to trust their results. HyPPOCRATES addressed all those challenges by proposing computationally efficient tools that perform these tasks while elaborating on a theoretical understanding of these algorithms by shedding light on important aspects such as recovery guarantees etc. HyPPOCRATES provided algorithms that can be applied to a wide range of imaging data.

The overall objective of the program was the development of a suite of machine learning algorithms that will be applied in various imaging tasks. The algorithms were built on state-of-the-art ideas of machine learning and nonconvex optimization theory and came up with theoretical guarantees as well strong empirical evidence that shows the efficiency of the derived methods. By incorporating information for the structure of imaging data such as spatial and spectral correlation which is an inherent characteristic of hyperspectral images, HyPPOCRATES algorithms can efficiently carry out tasks that are otherwise computationally expensive, requiring a significant amount of computational resources.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In the framework of HyPPOCRATES, we developed novel matrix and tensor decomposition methods that leverage low-rank representations to provide rank-revealing image representations. Specifically, we came up with a novel variational form of the Schatten-p quasi norm, which is a nonconvex heuristic that is used for enforcing a low-rank structure. In hyperspectral imaging (HSIs) low-rank structures allow us to capture the spatio-spectral correlation leading to meaningful representations of the HSIs. Since we are dealing with nonconvex optimization problems, understanding the optimization landscape is challenging, yet critical when it comes to providing guarantees for the performance of the derived algorithm. In our work, we analyzed the landscape of the nonconvex objective function showing that the nonconvex matrix factorization-based relaxation that we proposed comes with no additional spurious local minima induced in the derived optimization problem. That is to say, computational savings are coming without paying any price in the task of finding minimizers of the new problem. The new formulation has been also used in the problem of online robust principal component analysis, where it is assumed that data arrive in a streaming fashion.

The main contributions of HyPPOCRATES are summarized as follows:
1) We came up with a novel formulation for low-rank matrix factorization. With our approach we have demonstrated significant computational advantages over other state-of-the-art methods in problems such as hyperspectral image processing and matrix completion. Moreover, for the first time in the literature theoretical guarantees for the performance of the non-convex low-rank matrix factorization methods were developed.
2) We developed a suite of novel tensor decomposition algorithms that are suitably devised to efficiently process hyperspectral images and videos. Our work is the first that provides algorithms for performing tensor decomposition and model selection at the same time in a computationally efficient manner.
3) We developed a provably correct method for robust subspace recovery that can perform subspace estimation without requiring the knowledge of its rank. The resulting algorithm has been applied to the problem of subspace clustering of hyperspectral images offering significant advantages over other state-of-the-art methods. The results obtained during the action have been accepted and presented to top-tier prestigious peer-reviewed conferences in the fields of machine learning and signal processing, such as the International Conference on Learning Representations (ICLR), International Conference on Machine Learning (ICML), IEEE Transactions on Signal Processing, etc.
The ER has also disseminated the results of HyPPOCRATES by giving talks to research institutes (e.g. Swiss Data Science Center) and universities (e.g. JHU, Technical University of Crete, Ecole Polytechnique).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The results of HyPPOCRATES go beyond state-of-the-art relevant methods since they offer matrix and tensor decomposition approaches that simultaneously perform model selection. This is achieved by better capturing inherent structures of signals and images. Specifically, we have shown that the proposed methods outperform other state-of-the-art approaches in various hyperspectral imaging problems where spatio-spectral correlation is highly present, and the ranks of matrix factorization and tensor decomposition models correspond to physically meaningful quantities e.g. the number of materials (endmembers) depicted on the scene. The results obtained so far, show the great potential of matrix and tensor decomposition methods for processing hyperspectral images. In the framework of HyPPOCRATES we have devised tensor decomposition methods that process streaming data and thus are amenable to processing multispectral/hyperspectral videos. Moreover, we have focused on applying the derived matrix and tensor decomposition methods for processing hyperspectral data for detecting SARS-CoV-2 molecules. This application is challenging yet expected to have a great impact on the effort of humanity to address the COVID-19 pandemic crisis. Finally, the algorithms we have developed in the framework of HyPPOCRATES can be applied can offer significant benefits to various applications such as the analysis of FMRI data and EEG/ECG signals. Currently, we have been working towards publishing a general-purpose toolbox that will allow users to run matrix and tensor decomposition algorithms for analyzing data captured for various imaging modalities.

The Block Term Tensor Decomposition used for hypespectral image representation in HyPPOCRATES