Periodic Reporting for period 1 - STARE (Machine learning based Software Toolkit for Automated identification in atomic-REsolution operando nanoscopy)
Reporting period: 2021-01-01 to 2022-06-30
The main objective of this ERC-PoC is to demonstrate the feasibility of a software toolkit with machine learning functionality for electron microscopy and to investigate its commercialization potential.
Electron microscopy technology has evolved through the years and the major developments are in two areas, namely advanced detector technology and in situ experimental capacity. These developments have allowed the Materials Science community to study dynamic processes in materials down to the atomic level. Due to the complexity, these new technologies are usually supplied by specialized technology providers, this includes both hardware and software. As a result, conducting advanced electron microscopy experiments is time-consuming and difficult. Parallel to this, machine learning has shown to be able to help human beings in many fronts, and it is also gaining popularity in the microscopy community for tasks such as data analysis. Yet, machine learning has largely been used for electron microscopy in an off-line fashion.
It is clear that a streamlined version of a software package that is capable of performing experimental control and machine learning will be beneficial for the community.
In this Proof-of-concept project, we demonstrated the above idea through a piece of software written in Python. On the technical level, the software encompasses different functional blocks for microscope control, detector control, in situ experiment control and machine learning.
The main framework is developed by our team at the Technical University Darmstadt and it contains the following aspects:
(i) A detector control part was implemented through the technical support from Quantum Detectors (QD). This part has two main functions, commanding the detector and receiving data stream returned by the detector.
(ii) A data visualization part that allows the user to set visualization parameters, and it is responsible for generating the final images.
(iii) The machine learning part contains two functions phase extraction and atom identification. The phase extraction method is based on previous research result in our group; the atom identification is based on open source Python package AtomAI. These two methods represent two different use cases of machine learning on real space image and reciprocal space data. The phase extraction method will generate a view of the identified unique phases of imaging materials; the atom identification method will generate a view of the real space image with atom locations identified. The results of these methods can be passed down to further analysis defined by users.
During the course of this project, it has come to our realization that the experimental control part can be expanded and enhanced with machine learning. This is not in the scope of the originally planned activities. Therefore, we plan to submit a follow-up grant proposal to further develop this idea and explore a possible extension in an EIC Transition Grant context.
Electron microscopy technology has evolved through the years and the major developments are in two areas, namely advanced detector technology and in situ experimental capacity. These developments have allowed the Materials Science community to study dynamic processes in materials down to the atomic level. Due to the complexity, these new technologies are usually supplied by specialized technology providers, this includes both hardware and software. As a result, conducting advanced electron microscopy experiments is time-consuming and difficult. Parallel to this, machine learning has shown to be able to help human beings in many fronts, and it is also gaining popularity in the microscopy community for tasks such as data analysis. Yet, machine learning has largely been used for electron microscopy in an off-line fashion.
It is clear that a streamlined version of a software package that is capable of performing experimental control and machine learning will be beneficial for the community.
In this Proof-of-concept project, we demonstrated the above idea through a piece of software written in Python. On the technical level, the software encompasses different functional blocks for microscope control, detector control, in situ experiment control and machine learning.
The main framework is developed by our team at the Technical University Darmstadt and it contains the following aspects:
(i) A detector control part was implemented through the technical support from Quantum Detectors (QD). This part has two main functions, commanding the detector and receiving data stream returned by the detector.
(ii) A data visualization part that allows the user to set visualization parameters, and it is responsible for generating the final images.
(iii) The machine learning part contains two functions phase extraction and atom identification. The phase extraction method is based on previous research result in our group; the atom identification is based on open source Python package AtomAI. These two methods represent two different use cases of machine learning on real space image and reciprocal space data. The phase extraction method will generate a view of the identified unique phases of imaging materials; the atom identification method will generate a view of the real space image with atom locations identified. The results of these methods can be passed down to further analysis defined by users.
During the course of this project, it has come to our realization that the experimental control part can be expanded and enhanced with machine learning. This is not in the scope of the originally planned activities. Therefore, we plan to submit a follow-up grant proposal to further develop this idea and explore a possible extension in an EIC Transition Grant context.