Periodic Reporting for period 4 - BUNGEE-TOOLS (Building Next-Generation Computational Tools for High Resolution Neuroimaging Studies)
Okres sprawozdawczy: 2021-03-01 do 2022-02-28
Recent advances in magnetic resonance imaging (MRI) scanners are providing us with images of the human brain of increasing detail. While these images hold promise to greatly increase our understanding of the human brain works, researchers still analyze the images with old software tools that are over a decade old, and were designed to work with older, less detailed images. In particular, these tools do not consider smaller brain regions that are visible in present-day scans, but not in older MRI images. This inability to capitalize on the vast improvement of MRI is slowing down progress in brain research.
In this project, we propose to build next-generation software tools that will enable researchers to take full advantage of the increasing image quality of modern MRI scanners. The main idea is that the tools capitalize on "ex vivo" data from brains of dead donors. Such brains have the advantage that the can be stained with different dyes and looked under the microscope, allowing us to build maps of superior detail. Our initial plan was to build these tools around a high-resolution map (“atlas”) of the human brain. However, the advent of deep machine learning in 2015-2016 changed our plans and we ended up building them around convolutional neural networks instead. The tools can be used to analyze MRI scans of living people, enabling researchers to extract more detailed information and more subtle changes due to e.g. aging or disease. Moreover, the tools can be used to analyze clinical MRI scans acquired in hospitals with diagnostic purposes, which have much less resolution than the scans that are normally acquired in research. This capability was not present in previous tools, and will enable researchers to conduct studies with millions of scans that already exist in picture archiving systems in hospitals around the world.
All our developed tools are freely available to the scientific community, so we expect them to have a tremendous impact on the quest to understand the human brain (in health and in disease), and ultimately on public health and the economy.
In this project, we propose to build next-generation software tools that will enable researchers to take full advantage of the increasing image quality of modern MRI scanners. The main idea is that the tools capitalize on "ex vivo" data from brains of dead donors. Such brains have the advantage that the can be stained with different dyes and looked under the microscope, allowing us to build maps of superior detail. Our initial plan was to build these tools around a high-resolution map (“atlas”) of the human brain. However, the advent of deep machine learning in 2015-2016 changed our plans and we ended up building them around convolutional neural networks instead. The tools can be used to analyze MRI scans of living people, enabling researchers to extract more detailed information and more subtle changes due to e.g. aging or disease. Moreover, the tools can be used to analyze clinical MRI scans acquired in hospitals with diagnostic purposes, which have much less resolution than the scans that are normally acquired in research. This capability was not present in previous tools, and will enable researchers to conduct studies with millions of scans that already exist in picture archiving systems in hospitals around the world.
All our developed tools are freely available to the scientific community, so we expect them to have a tremendous impact on the quest to understand the human brain (in health and in disease), and ultimately on public health and the economy.
1. Obtaining the images that are necessary to build our atlas: We obtained the brain samples through the UCL Queen Square Brain Bank (QSBB). We have designed a protocol for MRI scanning of the samples. We had technicians working at QSBB that routinely sectioned, stained, and digitized the tissue, with help from our collaborators at QSBB. They followed a pipeline that we have developed in this project, and which facilitates solving the “jigsaw puzzle” of figuring out where in the brain is each section that we have cut (“3D histology reconstruction”).
2. Manually tracing the boundaries of brain structures on the stained sections: we also hired technicians to trace these boundaries, which is necessary in order to build the maps of the brain, as we need to know where we are at every image location. Manual delineation is a very repetitive, time-consuming process, so we have developed computer programs to assist the technicians and speed up the process. Specifically, we have created a program called SmartInterpol, which enables a used to delineated one section every few, and fills in the boundaries in between.
3. Designing computer algorithms to solve the 3D histology reconstruction problem: This is a very difficult problem. We have developed joint image registration methods, which, combined with image synthesis methods, have enabled us to produce highly accurate 3D reconstructions without human intervention.
4. Building the atlas: as explained above, we decided to replace the atlas by convolutional neural networks (CNNs).
5. Building programs that apply the tools to the automated analysis of MRI scans of living people: During the second half of the project, we moved away from atlases and Bayesian techniques and instead developed methods based on CNNs that also borrowed ideas from the Bayesian literature, enabling us to segment brain MRI scans of any orientation, resolution, and contrast, including heterogeneous clinical scans acquired at hospitals.
6. Our colleagues from the UCL Dementia Research Centre are using our software to try to improve our understanding of the different variants of this disease – which will hopefully lead to improved diagnosis and treatment of the disease.
7. Scientific output: The project has so far yielded a large number of publications, which have been made freely available (“open access”). A list can be found in this report. In addition, we have: organized workshops and symposia (3D histology reconstruction, joint meetings between imaging laboratories from UCL and Imperial College London); given seminars on how to apply for European grants (ERC and Marie Curie programs); been part of the MICCAI mentorship program (“Connecting early-career researchers with experienced peers who share their experiences in academia, industry or entrepreneurship, to help develop the next generation of entrepreneurs, and innovators”); and co-organized the summer school of our laboratory at UCL.
8. Growth of mentees: research assistants hired for this project moved on a PhD position at the Institute of Cancer Research in the UK, and to the School of Medicine at the University of Cambridge. One of our PhD students moved on to a postdoc at MIT and the other is now a research scientist in industry. Two of our the postdocs won prestigious, highly competitive fellowships to continue their academic careers elsewhere; the other two obtained machine learning positions in industry.
9. Public engagement: We have taken dissemination very seriously. Engagement activities have included: participation in the In2ScienceUK program (hosting two high-school students from disadvantaged backgrounds for two weeks); articles in the newsletters of the UCL Medical Physics department and of the UCL QS Brain Bank; live demonstrations at schools in the London area as part of the STEM Ambassador Program; and live demonstrations to schoolchildren at the University College Hospital open day. Unfortunately, public engagement activities were seriously slowed down by the COVID-19 pandemic.
10. Data: we are making all the raw and processed data publicly available to other researchers, using mechanisms provided by University College London. In addition, we have created a website that anyone can use to explore the histological data: https://github-pages.ucl.ac.uk/BrainAtlas/#/atlas
11. Code: our tools are implemented in the widespread neuroimaging package FreeSurfer and can be easily used after installing FreeSurfer. The source code is also available on GitHub.
12. Further funding: results from this project have been instrumental to writing new proposals that have recently been funded, e.g. NIH grants 1R01AG070988 and 1RF1MH123195.
2. Manually tracing the boundaries of brain structures on the stained sections: we also hired technicians to trace these boundaries, which is necessary in order to build the maps of the brain, as we need to know where we are at every image location. Manual delineation is a very repetitive, time-consuming process, so we have developed computer programs to assist the technicians and speed up the process. Specifically, we have created a program called SmartInterpol, which enables a used to delineated one section every few, and fills in the boundaries in between.
3. Designing computer algorithms to solve the 3D histology reconstruction problem: This is a very difficult problem. We have developed joint image registration methods, which, combined with image synthesis methods, have enabled us to produce highly accurate 3D reconstructions without human intervention.
4. Building the atlas: as explained above, we decided to replace the atlas by convolutional neural networks (CNNs).
5. Building programs that apply the tools to the automated analysis of MRI scans of living people: During the second half of the project, we moved away from atlases and Bayesian techniques and instead developed methods based on CNNs that also borrowed ideas from the Bayesian literature, enabling us to segment brain MRI scans of any orientation, resolution, and contrast, including heterogeneous clinical scans acquired at hospitals.
6. Our colleagues from the UCL Dementia Research Centre are using our software to try to improve our understanding of the different variants of this disease – which will hopefully lead to improved diagnosis and treatment of the disease.
7. Scientific output: The project has so far yielded a large number of publications, which have been made freely available (“open access”). A list can be found in this report. In addition, we have: organized workshops and symposia (3D histology reconstruction, joint meetings between imaging laboratories from UCL and Imperial College London); given seminars on how to apply for European grants (ERC and Marie Curie programs); been part of the MICCAI mentorship program (“Connecting early-career researchers with experienced peers who share their experiences in academia, industry or entrepreneurship, to help develop the next generation of entrepreneurs, and innovators”); and co-organized the summer school of our laboratory at UCL.
8. Growth of mentees: research assistants hired for this project moved on a PhD position at the Institute of Cancer Research in the UK, and to the School of Medicine at the University of Cambridge. One of our PhD students moved on to a postdoc at MIT and the other is now a research scientist in industry. Two of our the postdocs won prestigious, highly competitive fellowships to continue their academic careers elsewhere; the other two obtained machine learning positions in industry.
9. Public engagement: We have taken dissemination very seriously. Engagement activities have included: participation in the In2ScienceUK program (hosting two high-school students from disadvantaged backgrounds for two weeks); articles in the newsletters of the UCL Medical Physics department and of the UCL QS Brain Bank; live demonstrations at schools in the London area as part of the STEM Ambassador Program; and live demonstrations to schoolchildren at the University College Hospital open day. Unfortunately, public engagement activities were seriously slowed down by the COVID-19 pandemic.
10. Data: we are making all the raw and processed data publicly available to other researchers, using mechanisms provided by University College London. In addition, we have created a website that anyone can use to explore the histological data: https://github-pages.ucl.ac.uk/BrainAtlas/#/atlas
11. Code: our tools are implemented in the widespread neuroimaging package FreeSurfer and can be easily used after installing FreeSurfer. The source code is also available on GitHub.
12. Further funding: results from this project have been instrumental to writing new proposals that have recently been funded, e.g. NIH grants 1R01AG070988 and 1RF1MH123195.
The two main contributions of this project that have significantly advanced the state of the art are:
1. The acquisition and curation of one of the most (if not the most) comprehensive histological dataset of the human brain to date.
2. The development of machine learning methods that can analyze brain MRI scans of any resolution and contrast, including clinical scans. These methods enable, for the first time, analysis of highly heterogeneous clinical MRI scans for neuroimaging studies with millions of subjects.
1. The acquisition and curation of one of the most (if not the most) comprehensive histological dataset of the human brain to date.
2. The development of machine learning methods that can analyze brain MRI scans of any resolution and contrast, including clinical scans. These methods enable, for the first time, analysis of highly heterogeneous clinical MRI scans for neuroimaging studies with millions of subjects.