Periodic Reporting for period 4 - SelfDriving4DSR (Enabling Live-Cell 4D Super-Resolution Microscopy Guided by Artificial Intelligence)
Reporting period: 2024-10-01 to 2025-12-31
Summary of the context and overall objectives of the project
The SelfDriving4DSR project aims to address a significant challenge in the field of super-resolution microscopy - the inability to observe molecular-level cellular processes non-invasively in living samples for extended periods. This happens because high-intensity illumination used in super-resolution microscopy can compromise cell behaviour and prove damaging over time.
This issue is important to society as it directly affects our ability to understand the molecular basis of cell regulation, which is crucial for studying biological behaviour in health and disease. Observing these processes at relevant spatial and temporal scales could lead to significant advancements in biomedical research, potentially resulting in disease diagnosis, treatment, and prevention breakthroughs.
The overall objective of the SelfDriving4DSR project is to overcome these limitations by establishing self-driving microscopes. These microscopes are designed to adapt in real-time to the biological phenomenon under observation, enabling the capacity for unprecedented 4D imaging data optimised for content, resolution, and quality while remaining non-invasive for long periods of time.
To achieve this, the project proposes to bridge and evolve cutting-edge concepts in Computational Optical Microscopy and Machine Learning, effectively establishing Machine Learning Guided 4D Super-Resolution Microscopy. This novel approach challenges the assumption that microscopy needs to obey homogeneous temporal sampling and enables 4D live-cell nanoscopic imaging over record periods.
The project's enabling capacity will be demonstrated by visualising nanoscale cellular events previously unseen over hours, such as the molecular-level progression of viral infection. This will provide unprecedented insights into the mechanisms of viral infection and potentially lead to the development of more effective antiviral treatments.
The project's main objective is to advance the field of super-resolution microscopy by designing autonomous microscopes that can adjust to the biological phenomenon being studied in real-time. This will allow researchers to observe molecular-level cellular processes in living samples for prolonged periods, providing valuable insights into cell regulation and the development of diseases such as viral infections. The success of this initiative will result in significant progress in biomedical research and will have far-reaching implications for society.
This issue is important to society as it directly affects our ability to understand the molecular basis of cell regulation, which is crucial for studying biological behaviour in health and disease. Observing these processes at relevant spatial and temporal scales could lead to significant advancements in biomedical research, potentially resulting in disease diagnosis, treatment, and prevention breakthroughs.
The overall objective of the SelfDriving4DSR project is to overcome these limitations by establishing self-driving microscopes. These microscopes are designed to adapt in real-time to the biological phenomenon under observation, enabling the capacity for unprecedented 4D imaging data optimised for content, resolution, and quality while remaining non-invasive for long periods of time.
To achieve this, the project proposes to bridge and evolve cutting-edge concepts in Computational Optical Microscopy and Machine Learning, effectively establishing Machine Learning Guided 4D Super-Resolution Microscopy. This novel approach challenges the assumption that microscopy needs to obey homogeneous temporal sampling and enables 4D live-cell nanoscopic imaging over record periods.
The project's enabling capacity will be demonstrated by visualising nanoscale cellular events previously unseen over hours, such as the molecular-level progression of viral infection. This will provide unprecedented insights into the mechanisms of viral infection and potentially lead to the development of more effective antiviral treatments.
The project's main objective is to advance the field of super-resolution microscopy by designing autonomous microscopes that can adjust to the biological phenomenon being studied in real-time. This will allow researchers to observe molecular-level cellular processes in living samples for prolonged periods, providing valuable insights into cell regulation and the development of diseases such as viral infections. The success of this initiative will result in significant progress in biomedical research and will have far-reaching implications for society.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
The SelfDriving4DSR ERC Consolidator Grant project aims to develop a "self-driving" 4D super-resolution microscope guided by artificial intelligence (AI) for long-term live cell imaging with nanoscale resolution. The work performed can be related to the project 4 major aims:
AIM 1: Establish Real-Time 4D Super-Resolution Microscopy
The first aim is to establish real-time 4D super-resolution microscopy. This is being achieved through the development of the NanoPyx library (DOI:10.1101/2023.08.13.553080) a machine-learning-powered high-speed bioimage analysis engine designed for real-time use during data acquisition. This library has significantly advanced the field of super-resolution microscopy by enabling real-time analysis of 4D super-resolution microscopy data.
In addition, we developed eSRRF, a new super-resolution approach that empowers low-illumination and low-phototoxicity live-cell imaging (DOI:10.1038/s41592-023-02057-w). eSRRF uses data-driven machine-learning optimization of its own algorithm, extracting photophysical information from the data itself. This development has significantly contributed to the reduction of phototoxicity in live-cell imaging, a major challenge in the field.
In tandem, we continue to develop a prototype multi-modal reactive microscope, controled by a machine-learning agent able to adapt imaging modality (e.g low-resolution live-cell friendly observation vs damaging super-resolution imaging) and imaging parameters (e.g. illumination power, exposure time, etc.) in real-time. This prototype will be used to demonstrate the potential of the SelfDriving4DSR approach.
AIM 2: Machine-Learning Driven 4D SRM with Minimal Photodamage
The second aim is to develop a machine-learning-driven 4D super-resolution microscopy approach with minimal photodamage. This was achieved through the development of the ZeroCostDL4Mic (DOI:10.1038/s41467-021-22518-0) and DL4MicEverywhere (10.1101/2023.11.19.567606) platforms. These platforms provide a large library of deep learning models for microscopy, serving as the basis for interpreting and analyzing microscopy data.
Our work in this area has also led to significant advancements in reducing phototoxicity in live imaging. This was discussed in our paper "Harnessing Artificial Intelligence To Reduce Phototoxicity in Live Imaging" (DOI:10.48550/arXiv.2308.04387).
AIM 3: Learning from Quantitative Cellular Observation and Phototoxicity Assays
The third aim is to learn from quantitative cellular observation and phototoxicity assays. This is also being achieved through the development of the ZeroCostDL4Mic and DL4MicEverywhere platforms, which have provided a large library of deep learning models for microscopy. These platforms have significantly advanced our ability to interpret and analyze microscopy data, leading to new insights into cellular behavior and phototoxicity.
AIM 4: Spatiotemporal Mapping of HIV-1 Viral Infection at the Nanoscale
The fourth aim is to map HIV-1 viral infection at the nanoscale spatiotemporally. This was in part achieved through our observations of T-cell plasma membrane CD4 redistribution upon HIV-1 binding using single-molecule super-resolution imaging. These observations have provided new insights into the mechanisms of HIV-1 viral infection and have potential implications for developing new therapeutic strategies (DOI:10.3390/v13010142).
The project is already leading to significant advancements in infection cell biology, super-resolution microscopy, machine learning and bioimage analysis. The development of new technologies and methodologies has advanced our understanding of cellular behaviour and viral infection and provided new tools for the broader scientific community. The project's philosophy of making research reproducible, transparent, and open-source has ensured that these advancements are widely disseminated and accessible to the cell biology and biomedical research community.
AIM 1: Establish Real-Time 4D Super-Resolution Microscopy
The first aim is to establish real-time 4D super-resolution microscopy. This is being achieved through the development of the NanoPyx library (DOI:10.1101/2023.08.13.553080) a machine-learning-powered high-speed bioimage analysis engine designed for real-time use during data acquisition. This library has significantly advanced the field of super-resolution microscopy by enabling real-time analysis of 4D super-resolution microscopy data.
In addition, we developed eSRRF, a new super-resolution approach that empowers low-illumination and low-phototoxicity live-cell imaging (DOI:10.1038/s41592-023-02057-w). eSRRF uses data-driven machine-learning optimization of its own algorithm, extracting photophysical information from the data itself. This development has significantly contributed to the reduction of phototoxicity in live-cell imaging, a major challenge in the field.
In tandem, we continue to develop a prototype multi-modal reactive microscope, controled by a machine-learning agent able to adapt imaging modality (e.g low-resolution live-cell friendly observation vs damaging super-resolution imaging) and imaging parameters (e.g. illumination power, exposure time, etc.) in real-time. This prototype will be used to demonstrate the potential of the SelfDriving4DSR approach.
AIM 2: Machine-Learning Driven 4D SRM with Minimal Photodamage
The second aim is to develop a machine-learning-driven 4D super-resolution microscopy approach with minimal photodamage. This was achieved through the development of the ZeroCostDL4Mic (DOI:10.1038/s41467-021-22518-0) and DL4MicEverywhere (10.1101/2023.11.19.567606) platforms. These platforms provide a large library of deep learning models for microscopy, serving as the basis for interpreting and analyzing microscopy data.
Our work in this area has also led to significant advancements in reducing phototoxicity in live imaging. This was discussed in our paper "Harnessing Artificial Intelligence To Reduce Phototoxicity in Live Imaging" (DOI:10.48550/arXiv.2308.04387).
AIM 3: Learning from Quantitative Cellular Observation and Phototoxicity Assays
The third aim is to learn from quantitative cellular observation and phototoxicity assays. This is also being achieved through the development of the ZeroCostDL4Mic and DL4MicEverywhere platforms, which have provided a large library of deep learning models for microscopy. These platforms have significantly advanced our ability to interpret and analyze microscopy data, leading to new insights into cellular behavior and phototoxicity.
AIM 4: Spatiotemporal Mapping of HIV-1 Viral Infection at the Nanoscale
The fourth aim is to map HIV-1 viral infection at the nanoscale spatiotemporally. This was in part achieved through our observations of T-cell plasma membrane CD4 redistribution upon HIV-1 binding using single-molecule super-resolution imaging. These observations have provided new insights into the mechanisms of HIV-1 viral infection and have potential implications for developing new therapeutic strategies (DOI:10.3390/v13010142).
The project is already leading to significant advancements in infection cell biology, super-resolution microscopy, machine learning and bioimage analysis. The development of new technologies and methodologies has advanced our understanding of cellular behaviour and viral infection and provided new tools for the broader scientific community. The project's philosophy of making research reproducible, transparent, and open-source has ensured that these advancements are widely disseminated and accessible to the cell biology and biomedical research community.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
Over the first reporting period, we have made significant progress in developing machine learning (ML) guided 4D super-resolution microscopy (4D-SRM), pushing beyond the state of the art in multiple dimensions.
AIM1 - Establishing Real-Time 4D-SRM
We are engineering a novel adaptive 4D-SRM microscope capable of transitioning between different imaging modalities (SIM, SRRF, SMLM) in real-time using heterogeneous illumination controlled by adaptive optics. This has enabled us to achieve fast 3D super-resolution imaging with sub-100nm resolution in all axes. Key achievements include:
- Developing a multifocus detection system to capture multiple focal planes simultaneously, enabling high-speed 3D-SRRF and 3D-SMLM. This leverages concepts from multifocus microscopy (MFM).
- Generalizing the SRRF concept from 2D to 3D by optically characterising the multifocus system and developing analytical methods to reposition sample information accurately in 3D space.
- Achieving near instantaneous reconstructions of 4D datasets to enable real-time decision-making on adjusting the acquisition based on the observed sample.
Overall, we have pioneered a new paradigm in 4D-SRM that is optimised for ML control and can dynamically adapt its spatiotemporal sampling based on interpreting the biological phenomena under observation.
AIM2 - ML-Driven 4D-SRM with Minimal Photodamage
We have developed an analytical engine that controls 4D-SRM acquisition by predicting optimal imaging conditions while minimising light-induced cell stress. This includes:
- Classifiers that interpret biological state from the imaged data, including resolution, motility, image quality, and photodamage metrics.
- Predictors that determine the spatiotemporal resolution needed to capture cellular events while obeying sampling constraints to keep cells alive.
- Control frameworks to automate switching between imaging modalities based on the classifiers and predictors.
- Phototoxicity assays using ML-controlled microfluidics to characterise and optimise long-term live cell imaging schemes.
These methods enable microscopes to continuously adapt their acquisition to balance image content, quality and cell health.
AIM3 - Quantitative Cellular Observation and Phototoxicity Assays
We are currently using microfluidic systems capable of automated live-to-fixed cell imaging and spatial proteomics. These systems provide quantitative readouts on cell state transitions and molecular components regulating them. Additionally, we are using the same platforms to develop phototoxicity assays that enable us to engineer imaging schemes that reduce light-induced cell damage.
AIM4 – Tracking Viral Infection Dynamics at the Nanoscale
Our team is using our smart 4D-SRM technology to develop spatiotemporal tracking system for HIV-1 infection progression in CD4+ T cells. This will enable us to link early viral fusion events to downstream replication outcomes over 24 hours, and correlate the mode and timing of virus entry with the efficiency of infection. In the future, we plan to use transfer learning to differentiate stages of infection for adaptive imaging control, which will further enhance the accuracy and efficiency of our ML classifiers and predictors.
Expected Results Until the End of the Project
Over the final period, we expect additional breakthrough results across our aims:
AIM1 – We will finalise the 4D-SRM platform for release as an open-source microscope design that other groups can reproduce. This will include extensive documentation and training materials from workshops.
AIM2 – The analytical engine will be expanded with additional classifiers and predictors to handle more cell types, labels and imaging modes. We expect the system to approach human-level proficiency at inferring optimal acquisition parameters from the observed data.
AIM3 – The microfluidic live-to-fixed imaging system will be parallelised to increase throughput for spatial omics. New assays will quantify the transcriptomic effects of photodamage.
AIM4 – We will complete the HIV-1 infection timescale, correlating fusion to virion assembly over 24 hours. New machine learning predictors will be trained to forecast subsequent infection stages based on early imaging data. We will demonstrate the enabling potential of our technology by tackling open questions on the establishment of HIV-1 latency.
Beyond these aims, we are actively exploring the integration of our smart microscopes with other modalities like expansion microscopy. Additional pilot studies are also underway in collaboration with experts in cell cycle, differentiation and metastasis.
We remain on track to deliver a transformative leap in 4D live-cell SRM capabilities guided by artificial intelligence. This will open new horizons for connecting molecular processes to higher-level cell regulation across scales. We are committed to widespread dissemination through open-source distribution and hands-on training activities.
AIM1 - Establishing Real-Time 4D-SRM
We are engineering a novel adaptive 4D-SRM microscope capable of transitioning between different imaging modalities (SIM, SRRF, SMLM) in real-time using heterogeneous illumination controlled by adaptive optics. This has enabled us to achieve fast 3D super-resolution imaging with sub-100nm resolution in all axes. Key achievements include:
- Developing a multifocus detection system to capture multiple focal planes simultaneously, enabling high-speed 3D-SRRF and 3D-SMLM. This leverages concepts from multifocus microscopy (MFM).
- Generalizing the SRRF concept from 2D to 3D by optically characterising the multifocus system and developing analytical methods to reposition sample information accurately in 3D space.
- Achieving near instantaneous reconstructions of 4D datasets to enable real-time decision-making on adjusting the acquisition based on the observed sample.
Overall, we have pioneered a new paradigm in 4D-SRM that is optimised for ML control and can dynamically adapt its spatiotemporal sampling based on interpreting the biological phenomena under observation.
AIM2 - ML-Driven 4D-SRM with Minimal Photodamage
We have developed an analytical engine that controls 4D-SRM acquisition by predicting optimal imaging conditions while minimising light-induced cell stress. This includes:
- Classifiers that interpret biological state from the imaged data, including resolution, motility, image quality, and photodamage metrics.
- Predictors that determine the spatiotemporal resolution needed to capture cellular events while obeying sampling constraints to keep cells alive.
- Control frameworks to automate switching between imaging modalities based on the classifiers and predictors.
- Phototoxicity assays using ML-controlled microfluidics to characterise and optimise long-term live cell imaging schemes.
These methods enable microscopes to continuously adapt their acquisition to balance image content, quality and cell health.
AIM3 - Quantitative Cellular Observation and Phototoxicity Assays
We are currently using microfluidic systems capable of automated live-to-fixed cell imaging and spatial proteomics. These systems provide quantitative readouts on cell state transitions and molecular components regulating them. Additionally, we are using the same platforms to develop phototoxicity assays that enable us to engineer imaging schemes that reduce light-induced cell damage.
AIM4 – Tracking Viral Infection Dynamics at the Nanoscale
Our team is using our smart 4D-SRM technology to develop spatiotemporal tracking system for HIV-1 infection progression in CD4+ T cells. This will enable us to link early viral fusion events to downstream replication outcomes over 24 hours, and correlate the mode and timing of virus entry with the efficiency of infection. In the future, we plan to use transfer learning to differentiate stages of infection for adaptive imaging control, which will further enhance the accuracy and efficiency of our ML classifiers and predictors.
Expected Results Until the End of the Project
Over the final period, we expect additional breakthrough results across our aims:
AIM1 – We will finalise the 4D-SRM platform for release as an open-source microscope design that other groups can reproduce. This will include extensive documentation and training materials from workshops.
AIM2 – The analytical engine will be expanded with additional classifiers and predictors to handle more cell types, labels and imaging modes. We expect the system to approach human-level proficiency at inferring optimal acquisition parameters from the observed data.
AIM3 – The microfluidic live-to-fixed imaging system will be parallelised to increase throughput for spatial omics. New assays will quantify the transcriptomic effects of photodamage.
AIM4 – We will complete the HIV-1 infection timescale, correlating fusion to virion assembly over 24 hours. New machine learning predictors will be trained to forecast subsequent infection stages based on early imaging data. We will demonstrate the enabling potential of our technology by tackling open questions on the establishment of HIV-1 latency.
Beyond these aims, we are actively exploring the integration of our smart microscopes with other modalities like expansion microscopy. Additional pilot studies are also underway in collaboration with experts in cell cycle, differentiation and metastasis.
We remain on track to deliver a transformative leap in 4D live-cell SRM capabilities guided by artificial intelligence. This will open new horizons for connecting molecular processes to higher-level cell regulation across scales. We are committed to widespread dissemination through open-source distribution and hands-on training activities.