UNCARIA: UNcertainty estimation in CARdiac Image Analysis

Project Information

UNCARIA

Grant agreement ID: 892297

DOI

10.3030/892297

Project closed

EC signature date 10 March 2020

Start date 1 September 2021

End date 29 February 2024

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 224 071,20

EU contribution

€ 224 071,20

Coordinated by

UNIVERSIDAD POMPEU FABRA
Spain

Periodic Reporting for period 2 - UNCARIA (UNCARIA: UNcertainty estimation in CARdiac Image Analysis)

Reporting period: 2023-03-01 to 2024-02-29

Summary of the context and overall objectives of the project

- What is the problem being addressed?
The overarching goal of the UNCARIA project was to develop mathematical and computational tools and systems for translating medical Machine Learning algorithms from highly accurate, large-scale, pattern recognition systems, into a more clinically usable technology. This is a greatly challenging problem that we have approached by building trustworthiness and transparency on modern DNNs via theoretical frameworks and practical tools for predictive uncertainty analysis.
Toward addressing the above research problem, we have developed new techniques to improve model calibration (the ability of a model to be certain when it is correct and uncertain when it tends to be incorrect), and attempt to construct also appropriate measures of success for uncertainty quantification applications. The theoretical approach is complemented by relevant applications: while the initial goal of the project was to focus on cardiac image applications, we have not limited ourselves to this particular area, but have developed methods applicable in most medical image analysis modalities.

- Why is it important for society?
As Artificial Intelligence adoption grows in different areas of our society, it becomes increasingly important to remain vigilant and aware of its unforeseeable behavior when faced with data far away from its training distribution.
In particular uncertainty quantification techniques are useful approaches to measure the true confidence placed by machine learning models on their predictions, which can be critical in medical applications like the ones considered in this project.

- What were the overall objectives?
The objectives stated next are a slight rectification of the ones initially formulated in the research proposal, as they were been adapted to the continuously changing research landscape of medical machine learning.
RO1 – Epistemic Uncertainty in Medical Image Analysis: New tools and methods
RO2 – Extension to Quantification of Uncertainty in Medical Image Segmentation
RO3– Aleatoric Uncertainty Analysis: Calibration in Medical Image Diagnosis

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project started remotely on September/2021 due to delays in obtaining an Australian visa arising from Australian pandemic restrictions. The fellow received visa approval and traveled to Australia in the end of Summer 2022, reducing the originally foreseen outgoing phase of 18 months down to a for a five-month stay. The work performed in the first half of the project is divided below into the first remote part and the second actual outgoing phase.

OUTGOING PHASE:
----------------------------
During the initial remote and outgoing part of the project, several research articles with the fellow as first author have been generated:
1) Balanced-mixup for highly imbalanced medical image classification, MICCAI 2021, Adrian Galdran el al.
2) On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness, MICCAIw 2022, Adrian Galdran el al.
3) Convolutional nets versus vision transformers for diabetic foot ulcer classification, MICCAIw 2021, Adrian Galdran el al.
4) Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation, Adrian Galdran el al., ICPR 2021
5) Test Time Transform Prediction for Open Set Histopathological Image Recognition, MICCAI 2022, Adrian Galdran el al.
6) Multi-Head Multi-Loss Model Calibration, MICCAI 2023, A Galdran et al.
7) Performance Metrics for Probabilistic Ordinal Classifiers, MICCAI 2023, A Galdran

The 6th work was developed in collaboration with Dr. Johan Verjans, a cardiologist who is in charge of supervising the secondment to the South Australian Medical Research Institute, in Adelaide. Since the proposed method can quantify aleatoric uncertainty in classification problems, this achievement contributes to the third objective of the project RO3. Further, the fellow won a public competition on segmentation of Multiple Sclerosis lesions from brain MRIs with uncertain data, extending the method in (*) to image segmentation, see: https://shifts.grand-challenge.org/ for competition details. This extended technique therefore contributes substantially to the state of the art in medical image segmentation, i.e. RO2. Besides, several research articles were produced in collaboration with other researchers. Finally, the fellow also carried out networking activities in Adelaide and collaborated in preparing research proposals with member of the AIML (Australian Institute of Machine Learning).


INCOMING PHASE:
---------------------------
The last 12 months of the project were spent at Universitat Pompeu Fabra in Barcelona (Spain). Besides continuing his substantial productivity in terms of scientifc publication, the fellow performed outreach activities (European Research Night presentation at the Science Museum in Barcelona, delivered a talk at Sick Children Hospital Sant Joan in Barcelona). The fellow also has taken up supervisiing and mentoring activities. At the moment, he is supervisiing two PhD students and two MSc students, which is a natural next step in his scientific career. Lastly, the fellow won a highly competitive talk for a Ramon y Cajal fellowship, a 5-year contract of a senior research position comparable to a tenure track, awarded by the Spanish Ministry of Science.

Below there is a lis of publications belonging to this last phase of the project. Additionally, another relevant outcome is the lead in organizing and holding UQinMIA - Uncertainty Quantification in Medical Image Analysis, a tutorial (advanced lectures on cutting-edge research topics) in Vancouver, celebrated jointly with MICCAI 2023.
1) Multi-Head Multi-Loss Model Calibration, Adrian Galdran, J. Verjans, G. Carneiro, MAG Ballester, MICCAI, 2023
2) Performance Metrics for Probabilistic Ordinal Classifiers, Adrian Galdran, MICCAI 2023
3) Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses, B. Liu, …, Adrian Galdran, …, I. B. Ayed, Medical Image Analysis, 2024
4) FUSeg: The foot ulcer segmentation challenge, C. Wang,... Adrian Galdran, …, Z. Yu, Information, 2024

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Work carried out during the project advanced the state-of-the-art in three directions:

1) Out-of-distribution detection of histological data: capability of machine learning models to discard unknown data when making predictions, which belongs to the area of Epistemic Uncertainty Quantification (RO1).
2) A multi-head multi-loss model ensemble technique for improving the calibration of diagnostic models, with a performance beyond current training-time calibration techniques, a valuable contribution to Aleatoric Uncertainty Quantification field (RO3).
3) Extension of the method in 2) for lesion segmentation on brain MRI scans, which resulted into a state-of-the-art approach to volumetric segmentation with proper uncertainties, contributing to RO2.

The second part of the project resulted in noticeable outreach activity, plus further contributions to the state-of-the-art in medical image segmentation with built-in uncertainty quantification. Furthermore, exploration of uncertainty quantification techniques for pancreatic lesion segmentation and diagnosis was carried out in collaboration with a SME in Barcelona (Sycai Medical), and a PhD student of this company was taken up by the fellow who now acts as her supervisor.

Project Image