Periodic Reporting for period 2 - UNCARIA (UNCARIA: UNcertainty estimation in CARdiac Image Analysis)
Periodo di rendicontazione: 2023-03-01 al 2024-02-29
- What is the problem being addressed?
The overarching goal of the UNCARIA project was to develop mathematical and computational tools and systems for translating medical Machine Learning algorithms from highly accurate, large-scale, pattern recognition systems, into a more clinically usable technology. This is a greatly challenging problem that we have approached by building trustworthiness and transparency on modern DNNs via theoretical frameworks and practical tools for predictive uncertainty analysis.
Toward addressing the above research problem, we have developed new techniques to improve model calibration (the ability of a model to be certain when it is correct and uncertain when it tends to be incorrect), and attempt to construct also appropriate measures of success for uncertainty quantification applications. The theoretical approach is complemented by relevant applications: while the initial goal of the project was to focus on cardiac image applications, we have not limited ourselves to this particular area, but have developed methods applicable in most medical image analysis modalities.
- Why is it important for society?
As Artificial Intelligence adoption grows in different areas of our society, it becomes increasingly important to remain vigilant and aware of its unforeseeable behavior when faced with data far away from its training distribution.
In particular uncertainty quantification techniques are useful approaches to measure the true confidence placed by machine learning models on their predictions, which can be critical in medical applications like the ones considered in this project.
- What were the overall objectives?
The objectives stated next are a slight rectification of the ones initially formulated in the research proposal, as they were been adapted to the continuously changing research landscape of medical machine learning.
RO1 – Epistemic Uncertainty in Medical Image Analysis: New tools and methods
RO2 – Extension to Quantification of Uncertainty in Medical Image Segmentation
RO3– Aleatoric Uncertainty Analysis: Calibration in Medical Image Diagnosis
The overarching goal of the UNCARIA project was to develop mathematical and computational tools and systems for translating medical Machine Learning algorithms from highly accurate, large-scale, pattern recognition systems, into a more clinically usable technology. This is a greatly challenging problem that we have approached by building trustworthiness and transparency on modern DNNs via theoretical frameworks and practical tools for predictive uncertainty analysis.
Toward addressing the above research problem, we have developed new techniques to improve model calibration (the ability of a model to be certain when it is correct and uncertain when it tends to be incorrect), and attempt to construct also appropriate measures of success for uncertainty quantification applications. The theoretical approach is complemented by relevant applications: while the initial goal of the project was to focus on cardiac image applications, we have not limited ourselves to this particular area, but have developed methods applicable in most medical image analysis modalities.
- Why is it important for society?
As Artificial Intelligence adoption grows in different areas of our society, it becomes increasingly important to remain vigilant and aware of its unforeseeable behavior when faced with data far away from its training distribution.
In particular uncertainty quantification techniques are useful approaches to measure the true confidence placed by machine learning models on their predictions, which can be critical in medical applications like the ones considered in this project.
- What were the overall objectives?
The objectives stated next are a slight rectification of the ones initially formulated in the research proposal, as they were been adapted to the continuously changing research landscape of medical machine learning.
RO1 – Epistemic Uncertainty in Medical Image Analysis: New tools and methods
RO2 – Extension to Quantification of Uncertainty in Medical Image Segmentation
RO3– Aleatoric Uncertainty Analysis: Calibration in Medical Image Diagnosis
The project started remotely on September/2021 due to delays in obtaining an Australian visa arising from Australian pandemic restrictions. The fellow received visa approval and traveled to Australia in the end of Summer 2022, reducing the originally foreseen outgoing phase of 18 months down to a for a five-month stay. The work performed in the first half of the project is divided below into the first remote part and the second actual outgoing phase.
OUTGOING PHASE:
----------------------------
During the initial remote and outgoing part of the project, several research articles with the fellow as first author have been generated:
1) Balanced-mixup for highly imbalanced medical image classification, MICCAI 2021, Adrian Galdran el al.
2) On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness, MICCAIw 2022, Adrian Galdran el al.
3) Convolutional nets versus vision transformers for diabetic foot ulcer classification, MICCAIw 2021, Adrian Galdran el al.
4) Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation, Adrian Galdran el al., ICPR 2021
5) Test Time Transform Prediction for Open Set Histopathological Image Recognition, MICCAI 2022, Adrian Galdran el al.
6) Multi-Head Multi-Loss Model Calibration, MICCAI 2023, A Galdran et al.
7) Performance Metrics for Probabilistic Ordinal Classifiers, MICCAI 2023, A Galdran
The 6th work was developed in collaboration with Dr. Johan Verjans, a cardiologist who is in charge of supervising the secondment to the South Australian Medical Research Institute, in Adelaide. Since the proposed method can quantify aleatoric uncertainty in classification problems, this achievement contributes to the third objective of the project RO3. Further, the fellow won a public competition on segmentation of Multiple Sclerosis lesions from brain MRIs with uncertain data, extending the method in (*) to image segmentation, see: https://shifts.grand-challenge.org/(si apre in una nuova finestra) for competition details. This extended technique therefore contributes substantially to the state of the art in medical image segmentation, i.e. RO2. Besides, several research articles were produced in collaboration with other researchers. Finally, the fellow also carried out networking activities in Adelaide and collaborated in preparing research proposals with member of the AIML (Australian Institute of Machine Learning).
INCOMING PHASE:
---------------------------
The last 12 months of the project were spent at Universitat Pompeu Fabra in Barcelona (Spain). Besides continuing his substantial productivity in terms of scientifc publication, the fellow performed outreach activities (European Research Night presentation at the Science Museum in Barcelona, delivered a talk at Sick Children Hospital Sant Joan in Barcelona). The fellow also has taken up supervisiing and mentoring activities. At the moment, he is supervisiing two PhD students and two MSc students, which is a natural next step in his scientific career. Lastly, the fellow won a highly competitive talk for a Ramon y Cajal fellowship, a 5-year contract of a senior research position comparable to a tenure track, awarded by the Spanish Ministry of Science.
Below there is a lis of publications belonging to this last phase of the project. Additionally, another relevant outcome is the lead in organizing and holding UQinMIA - Uncertainty Quantification in Medical Image Analysis, a tutorial (advanced lectures on cutting-edge research topics) in Vancouver, celebrated jointly with MICCAI 2023.
1) Multi-Head Multi-Loss Model Calibration, Adrian Galdran, J. Verjans, G. Carneiro, MAG Ballester, MICCAI, 2023
2) Performance Metrics for Probabilistic Ordinal Classifiers, Adrian Galdran, MICCAI 2023
3) Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses, B. Liu, …, Adrian Galdran, …, I. B. Ayed, Medical Image Analysis, 2024
4) FUSeg: The foot ulcer segmentation challenge, C. Wang,... Adrian Galdran, …, Z. Yu, Information, 2024
OUTGOING PHASE:
----------------------------
During the initial remote and outgoing part of the project, several research articles with the fellow as first author have been generated:
1) Balanced-mixup for highly imbalanced medical image classification, MICCAI 2021, Adrian Galdran el al.
2) On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness, MICCAIw 2022, Adrian Galdran el al.
3) Convolutional nets versus vision transformers for diabetic foot ulcer classification, MICCAIw 2021, Adrian Galdran el al.
4) Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation, Adrian Galdran el al., ICPR 2021
5) Test Time Transform Prediction for Open Set Histopathological Image Recognition, MICCAI 2022, Adrian Galdran el al.
6) Multi-Head Multi-Loss Model Calibration, MICCAI 2023, A Galdran et al.
7) Performance Metrics for Probabilistic Ordinal Classifiers, MICCAI 2023, A Galdran
The 6th work was developed in collaboration with Dr. Johan Verjans, a cardiologist who is in charge of supervising the secondment to the South Australian Medical Research Institute, in Adelaide. Since the proposed method can quantify aleatoric uncertainty in classification problems, this achievement contributes to the third objective of the project RO3. Further, the fellow won a public competition on segmentation of Multiple Sclerosis lesions from brain MRIs with uncertain data, extending the method in (*) to image segmentation, see: https://shifts.grand-challenge.org/(si apre in una nuova finestra) for competition details. This extended technique therefore contributes substantially to the state of the art in medical image segmentation, i.e. RO2. Besides, several research articles were produced in collaboration with other researchers. Finally, the fellow also carried out networking activities in Adelaide and collaborated in preparing research proposals with member of the AIML (Australian Institute of Machine Learning).
INCOMING PHASE:
---------------------------
The last 12 months of the project were spent at Universitat Pompeu Fabra in Barcelona (Spain). Besides continuing his substantial productivity in terms of scientifc publication, the fellow performed outreach activities (European Research Night presentation at the Science Museum in Barcelona, delivered a talk at Sick Children Hospital Sant Joan in Barcelona). The fellow also has taken up supervisiing and mentoring activities. At the moment, he is supervisiing two PhD students and two MSc students, which is a natural next step in his scientific career. Lastly, the fellow won a highly competitive talk for a Ramon y Cajal fellowship, a 5-year contract of a senior research position comparable to a tenure track, awarded by the Spanish Ministry of Science.
Below there is a lis of publications belonging to this last phase of the project. Additionally, another relevant outcome is the lead in organizing and holding UQinMIA - Uncertainty Quantification in Medical Image Analysis, a tutorial (advanced lectures on cutting-edge research topics) in Vancouver, celebrated jointly with MICCAI 2023.
1) Multi-Head Multi-Loss Model Calibration, Adrian Galdran, J. Verjans, G. Carneiro, MAG Ballester, MICCAI, 2023
2) Performance Metrics for Probabilistic Ordinal Classifiers, Adrian Galdran, MICCAI 2023
3) Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses, B. Liu, …, Adrian Galdran, …, I. B. Ayed, Medical Image Analysis, 2024
4) FUSeg: The foot ulcer segmentation challenge, C. Wang,... Adrian Galdran, …, Z. Yu, Information, 2024
Work carried out during the project advanced the state-of-the-art in three directions:
1) Out-of-distribution detection of histological data: capability of machine learning models to discard unknown data when making predictions, which belongs to the area of Epistemic Uncertainty Quantification (RO1).
2) A multi-head multi-loss model ensemble technique for improving the calibration of diagnostic models, with a performance beyond current training-time calibration techniques, a valuable contribution to Aleatoric Uncertainty Quantification field (RO3).
3) Extension of the method in 2) for lesion segmentation on brain MRI scans, which resulted into a state-of-the-art approach to volumetric segmentation with proper uncertainties, contributing to RO2.
The second part of the project resulted in noticeable outreach activity, plus further contributions to the state-of-the-art in medical image segmentation with built-in uncertainty quantification. Furthermore, exploration of uncertainty quantification techniques for pancreatic lesion segmentation and diagnosis was carried out in collaboration with a SME in Barcelona (Sycai Medical), and a PhD student of this company was taken up by the fellow who now acts as her supervisor.
1) Out-of-distribution detection of histological data: capability of machine learning models to discard unknown data when making predictions, which belongs to the area of Epistemic Uncertainty Quantification (RO1).
2) A multi-head multi-loss model ensemble technique for improving the calibration of diagnostic models, with a performance beyond current training-time calibration techniques, a valuable contribution to Aleatoric Uncertainty Quantification field (RO3).
3) Extension of the method in 2) for lesion segmentation on brain MRI scans, which resulted into a state-of-the-art approach to volumetric segmentation with proper uncertainties, contributing to RO2.
The second part of the project resulted in noticeable outreach activity, plus further contributions to the state-of-the-art in medical image segmentation with built-in uncertainty quantification. Furthermore, exploration of uncertainty quantification techniques for pancreatic lesion segmentation and diagnosis was carried out in collaboration with a SME in Barcelona (Sycai Medical), and a PhD student of this company was taken up by the fellow who now acts as her supervisor.