Periodic Reporting for period 1 - DECIPHER (Dynamic Electron Imaging with Phase Retrieval)
Période du rapport: 2022-06-01 au 2024-11-30
The development of light-activated functional nanodevices is dependent on observing the out-of-equilibrium transport of energy, charge, and spin throughout nanomaterials. Two-dimensional functionalized nanoparticle (NP) supracrystals represent a key example of how functional activity and collective responses can be triggered by an external laser pulse. To fully understand the dynamics of these systems, new imaging techniques are needed, characterized by high spatial and temporal resolution and sensitivity to both light and heavy elements. DECIPHER proposes to bring together novel advances in electron sources with lensless phase retrieval imaging to create a microscope capable of directly visualizing the nanoscale dynamics of functionalized materials with fs-ps temporal resolution and Å-nm spatial resolution. Coherent diffractive imaging (CDI) is an effective route to study these systems as it enables the recovery of the unique nanostructure of an object encoded in the fine structure of the speckle pattern produced when fully coherent light is scattered off a sample. Ultrafast electron sources can effectively probe these nanosystems and enable the full-field imaging of functional intra- and inter-NP supracrystals. With recent breakthroughs in electron source and detector technologies, it is now possible to deploy these imaging methods in 4D (x, y, z, time). The three primary aims of DECIPHER are: (i) to develop an advanced ultrafast electron diffraction imaging system, (ii) to employ state-of-the-art phase retrieval techniques that facilitate comprehensive coherent imaging at multiple length scales, with the ability to detect both heavy and light elements, and (iii) to utilize these novel techniques to directly observe the light-induced functionality of NP supracrystals, with exceptional precision in determining their elemental composition, as well as their 2D and 3D surface features.
The Action deployed at the Laboratory for Ultrafast X-Ray and Electron Microscopy (LUXEM), University of Pavia (UniPV, IT), with the strategic collaboration of LACUS EPFL (CH) and DrXWorks (NL). The beginning of the Action in June 2022 coincided with the post-installation phase of the state-of-the-art ultrafast high-peak pulse intensity Ti:sapphire laser system in the Laboratory for Ultrafast X-ray and Electron Microscopy (LUXEM). Beginning in June and continuing until the end of the Action, we continuously worked to design and build in-house monitoring solutions to ensure that the laser system operated at target specifications. These include a custom beamline to provide pulse duration and beam mode monitoring of the output amplified laser pulses. Additionally, we modified the system itself to allow us to monitor the laser pulses generated by the Ti:sapphire oscillator driving the amplifier system. We defined operating procedures and performance benchmarks to monitor the laser behavior over time and began an optimization of the laser system performance through a series of diagnostic tests of both the laser and the climate (temperature + humidity) controlled laboratory space to ensure the amplifier system can operate at peak performance for long periods of time. This involved coordination with onsite staff at UniPV and offsite vendors (KMLabs, USA) to acquire, parse, analyze, and assess the system performance.
When the Action began, I had already initiated my discussion with the Consortium member Dr. X Works, who we are collaborating closely with for the design of our ultrafast electron source. In June 2022, further collaboration with this strategic partner led to the definition of the technical requirements and specifications of the ultrafast electron source, including the physical dimensions, brightness, emittance, and spatial coherence. The latter three are linked through a set of equations that allows the characteristics of the electron source to be tuned depending on the experimental need. With this information, we were able to complete the design of the Ultrafast Electron Diffraction Imaging (UEDI) beamline by mid-August 2022, including the required optics for laser beam delivery to the ultrafast electron source, beam steering and delay line for the pump arm of the setup, electron steering and focusing elements (steering plates and solenoid design), sample manipulation, and experimental chamber. The detection system for the electrons is provided by a single electron counting QUADRO detector manufactured by Dectris. We have collaborated closely with the technical designers at Dectris regarding the detection system and our experimental design to perform phase retrieval microscopy using scatter patterns obtained with the detector. With the design set, purchases have been placed for the necessary components and in house experts at UniPV have been solicited to support the Action with the design and construction of custom mechanical and electronic components for the experiment.
In addition to the experimental design of the UEDI system, I have also been pursuing new avenues for phase retrieval-based image reconstruction algorithms for use with this system. LUXEM began a collaboration with Prof. Giovanni Pellegrini in the Physics Department at UniPV to investigate the application of deep learning to the problem of phase retrieval with partially coherent sources, both electron- and photon-based. We are leveraging cutting-edge research on physics-informed convolutional neural networks to model a more complex forward model to account for partial coherence effects from the electrons. Specifically, we are incorporating previous methods such as Deep Image Prior, which uses the structure of the deep network itself to aid in the image recovery process. This method frees the reconstruction method from needing large datasets to train the algorithm, letting the final image be recovered using only the data acquired during an experiment. Initial tests are promising, showing that the method’s performance can match traditional gradient descent-based optimization methods for traditional coherent ptychographic image reconstruction, while simultaneously solving for the backward model of the imaging system, something that is not done in conventional phase retrieval.
When the Action began, I had already initiated my discussion with the Consortium member Dr. X Works, who we are collaborating closely with for the design of our ultrafast electron source. In June 2022, further collaboration with this strategic partner led to the definition of the technical requirements and specifications of the ultrafast electron source, including the physical dimensions, brightness, emittance, and spatial coherence. The latter three are linked through a set of equations that allows the characteristics of the electron source to be tuned depending on the experimental need. With this information, we were able to complete the design of the Ultrafast Electron Diffraction Imaging (UEDI) beamline by mid-August 2022, including the required optics for laser beam delivery to the ultrafast electron source, beam steering and delay line for the pump arm of the setup, electron steering and focusing elements (steering plates and solenoid design), sample manipulation, and experimental chamber. The detection system for the electrons is provided by a single electron counting QUADRO detector manufactured by Dectris. We have collaborated closely with the technical designers at Dectris regarding the detection system and our experimental design to perform phase retrieval microscopy using scatter patterns obtained with the detector. With the design set, purchases have been placed for the necessary components and in house experts at UniPV have been solicited to support the Action with the design and construction of custom mechanical and electronic components for the experiment.
In addition to the experimental design of the UEDI system, I have also been pursuing new avenues for phase retrieval-based image reconstruction algorithms for use with this system. LUXEM began a collaboration with Prof. Giovanni Pellegrini in the Physics Department at UniPV to investigate the application of deep learning to the problem of phase retrieval with partially coherent sources, both electron- and photon-based. We are leveraging cutting-edge research on physics-informed convolutional neural networks to model a more complex forward model to account for partial coherence effects from the electrons. Specifically, we are incorporating previous methods such as Deep Image Prior, which uses the structure of the deep network itself to aid in the image recovery process. This method frees the reconstruction method from needing large datasets to train the algorithm, letting the final image be recovered using only the data acquired during an experiment. Initial tests are promising, showing that the method’s performance can match traditional gradient descent-based optimization methods for traditional coherent ptychographic image reconstruction, while simultaneously solving for the backward model of the imaging system, something that is not done in conventional phase retrieval.
The main result from the Action, although interrupted prematurely for personal reasons, were two-fold:
1 - Design and early implementation of the UEDI set-up - unique in its capabilities.
2 - Development of Unsupervised Depp NN strategies supporting ptychographic image reconstruction from partially coherent sources - first time implementation.
The achievement of these milestones was made it possible by the unique know-how at LUXEM and its consortium of partners which, before the start of the DECIPHER Action, had already led to the following seeding output:
• Goldberg, et al., “Single-pulse, reference-free, spatiospectral measurement of ultrashort pulse-beams,” Optica, 9 (8), 894-902 (2022).
Publisher website: https://opg.optica.org/optica/fulltext.cfm?uri=optica-9-8-894&id=488520(s’ouvre dans une nouvelle fenêtre)
Trusted OA repository: https://iris.unipv.it/handle/11571/1477249(s’ouvre dans une nouvelle fenêtre)
• Bevis, et al., "Multi-wavelength Ultrafast Ptychography: A Flexible Beamline with a Compact High Harmonic Source," in The International Conference on Ultrafast Phenomena (UP) 2022, Technical Digest Series (Optica Publishing Group, 2022), paper Th4A.41.
Publisher website: https://opg.optica.org/abstract.cfm?URI=UP-2022-Th4A.41(s’ouvre dans une nouvelle fenêtre)
Trusted OA repository: https://iris.unipv.it/handle/11571/1477538(s’ouvre dans une nouvelle fenêtre)
Peer-reviewed published output - Results of the Action:
• Publication accepted in January 2024 -where DECIPHER is acknowledged- but not yet published, hence not yet on IRIS repository under CC BY (links to the publisher’s website and to the trusted open access repository will be provided as soon as published):
M. Pancaldi, F. Guzzi, C. Bevis, M. Manfredda, J. Barolak, S. Bonetti, I. Bykova, D. De Angelis, G. de Ninno, M. Fanciulli, L. Novinec, E. Pedersoli, A. Ravindran, B. Rösner, C. David, T. Ruchon, A. Simoncig, M. Zangrando, D. Adams, P.Vavassori M. Sacchi, G. Kourousias, G. Mancini, F. Capotondi “High-resolution ptychographic imaging at a seeded free-electron laser source using OAM beams”, accepted in Optica (2024). Also in arXiv as CCBY at link: https://arxiv.org/abs/2310.11773(s’ouvre dans une nouvelle fenêtre)
• Abstracts at international conferences submitted -where DECIPHER is acknowledged- currently under review. Please note peer-reviewed abstracts will be published upon acceptance at the conferences. Hence these works are not yet published and hence not yet appearing on IRIS repository under CC BY (links to the publisher’s website and to the trusted open access repository will be provided as soon as published):
HILAS 2024
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, C. Svetina and G. F. Mancini, "Compact Beamline for High-Throughput, Multiscale, Ultrafast Extreme Ultraviolet Ptychography", submitted and accepted for oral presentation (2023)
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, G. F. Mancini and G.Pellegrini "Unsupervised Deep Learning for Ptychography", submitted and accepted for poster presentation (2023)
CLEO 2024
C. Grova, C. S. Bevis, N.Giani D. E. Adams, C. Svetina and G. F. Mancini, "A Flexible Beamline for Ultrafast High-Throughput Microscopy with Extreme Ultraviolet Radiation", submitted (2023)
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, G. F. Mancini and G. Pellegrini, "Towards Real-Time Ultrafast Ptychography with Unsupervised Deep Learning", submitted (2023)
UP2024
C. Grova, D. Karpov, N. Giani, C. S. Bevis, D. E. Adams, C. Svetina and G. F. Mancini, "High-Throughput, Tabletop, Ultrafast EUV Ptychography across Lengthscales", submitted (2024)
C. Grova, N. Giani, , C. S. Bevis, D. E. Adams, G. F. Mancini and G. Pellegrini, "Enabling Real-Time Ultrafast Ptychographic Imaging through Unsupervised Deep Learning", submitted (2024)
Please note that because these publications, currently accepted or under review, and already acknowledging EU funding, will be made available ‘open access’ as soon as possible.
During the timeframe of the Action, I also embed myself within the community of the Physics Department of the University of Pavia and LUXEM.
Teaching:
I gave three guest lectures in the Department’s course on Ultrafast Laser Physics regarding ultrafast pulse characterization. I also gave a comprehensive course on Coherent Diffractive Imaging and phase retrieval to members of LUXEM during the fall of 2022.
Outreach & Dissemination:
I was a participant in the European Researchers Night through the SHARPER program in Pavia, IT, where the university setup more than 40 booths at in the Castello Visconteo in downtown Pavia, where the research activities of scientists throughout the University of Pavia were showcased to the general public at the “Scienza in Castello” event. Moreover, I was invited to join the “Sumo Science” outreach program, which bridges the world of research and high school throughout Italy.
As part of my outreach and dissemination activities, I drafted an original Blog article for Dectris. Through the collaboration with Dectris on ptychographic lensless imaging with electron sources, Prof. Mancini and I were invited to contribute a blog article about the potential uses and future of the ptychography imaging technique in this new and burgeoning field. The article appears on the Blog section of the Dectris’s website regarding electron microscopy in September 2022, titled “Ptychography – With a ‘P’ As in ‘Pterodactyl”. This blog is Open Access; DECIPHER and the MSCA Action are acknowledged therein in compliance with the OpenAIRE guidelines. Available at the following address: https://www.dectris.com/company/news/blog/electron-microscopy/the-emerging-possibilities-in-ptychography/(s’ouvre dans une nouvelle fenêtre)
Professional Development:
We continued strengthening the close collaborative ties between LUXEM and members of the Consortium through two onsite visits to the newly constructed laboratory spaces LUXEM has in the Physics Department. We hosted Prof. Daniel Adams and one of his graduate students from the Colorado School of Mines in Golden, CO, USA during the last week of June. We took this opportunity to work in person on several research projects we are actively engaged in, the most prominent being the investigation into new reconstruction techniques and data processing methods for ultrafast pump-probe experiments. Prof. Adams gave a research seminar to the department where he summarized ongoing research efforts his lab is engaged in, and we were able to discuss knowledge transfer between his lab and LUXEM. Then, in October, we hosted Prof. Fabrizio Carbone from EPFL and a member of the DECIPHER Consortium. He gave a seminar to the members of LUXEM and we discussed the new ultrafast laser source, gained feedback on the experimental design of the UEDI system, and potential avenues for collaboration in the future.
In July 2022, I travelled internationally to Montreal, CA to participate in the International Conference on Ultrafast Phenomena, where my original contribution was accepted for a poster presentation. For both poster and associated peer-reviewed proceeding the visibility of the EU flag and to acknowledge EU funding was ensured. Here, I networked with peers and presented original research on the novel computational imaging system we designed to be housed in the state-of-the-art LUXEM lab space.
1 - Design and early implementation of the UEDI set-up - unique in its capabilities.
2 - Development of Unsupervised Depp NN strategies supporting ptychographic image reconstruction from partially coherent sources - first time implementation.
The achievement of these milestones was made it possible by the unique know-how at LUXEM and its consortium of partners which, before the start of the DECIPHER Action, had already led to the following seeding output:
• Goldberg, et al., “Single-pulse, reference-free, spatiospectral measurement of ultrashort pulse-beams,” Optica, 9 (8), 894-902 (2022).
Publisher website: https://opg.optica.org/optica/fulltext.cfm?uri=optica-9-8-894&id=488520(s’ouvre dans une nouvelle fenêtre)
Trusted OA repository: https://iris.unipv.it/handle/11571/1477249(s’ouvre dans une nouvelle fenêtre)
• Bevis, et al., "Multi-wavelength Ultrafast Ptychography: A Flexible Beamline with a Compact High Harmonic Source," in The International Conference on Ultrafast Phenomena (UP) 2022, Technical Digest Series (Optica Publishing Group, 2022), paper Th4A.41.
Publisher website: https://opg.optica.org/abstract.cfm?URI=UP-2022-Th4A.41(s’ouvre dans une nouvelle fenêtre)
Trusted OA repository: https://iris.unipv.it/handle/11571/1477538(s’ouvre dans une nouvelle fenêtre)
Peer-reviewed published output - Results of the Action:
• Publication accepted in January 2024 -where DECIPHER is acknowledged- but not yet published, hence not yet on IRIS repository under CC BY (links to the publisher’s website and to the trusted open access repository will be provided as soon as published):
M. Pancaldi, F. Guzzi, C. Bevis, M. Manfredda, J. Barolak, S. Bonetti, I. Bykova, D. De Angelis, G. de Ninno, M. Fanciulli, L. Novinec, E. Pedersoli, A. Ravindran, B. Rösner, C. David, T. Ruchon, A. Simoncig, M. Zangrando, D. Adams, P.Vavassori M. Sacchi, G. Kourousias, G. Mancini, F. Capotondi “High-resolution ptychographic imaging at a seeded free-electron laser source using OAM beams”, accepted in Optica (2024). Also in arXiv as CCBY at link: https://arxiv.org/abs/2310.11773(s’ouvre dans une nouvelle fenêtre)
• Abstracts at international conferences submitted -where DECIPHER is acknowledged- currently under review. Please note peer-reviewed abstracts will be published upon acceptance at the conferences. Hence these works are not yet published and hence not yet appearing on IRIS repository under CC BY (links to the publisher’s website and to the trusted open access repository will be provided as soon as published):
HILAS 2024
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, C. Svetina and G. F. Mancini, "Compact Beamline for High-Throughput, Multiscale, Ultrafast Extreme Ultraviolet Ptychography", submitted and accepted for oral presentation (2023)
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, G. F. Mancini and G.Pellegrini "Unsupervised Deep Learning for Ptychography", submitted and accepted for poster presentation (2023)
CLEO 2024
C. Grova, C. S. Bevis, N.Giani D. E. Adams, C. Svetina and G. F. Mancini, "A Flexible Beamline for Ultrafast High-Throughput Microscopy with Extreme Ultraviolet Radiation", submitted (2023)
C. Grova, C. S. Bevis, N. Giani, D. E. Adams, G. F. Mancini and G. Pellegrini, "Towards Real-Time Ultrafast Ptychography with Unsupervised Deep Learning", submitted (2023)
UP2024
C. Grova, D. Karpov, N. Giani, C. S. Bevis, D. E. Adams, C. Svetina and G. F. Mancini, "High-Throughput, Tabletop, Ultrafast EUV Ptychography across Lengthscales", submitted (2024)
C. Grova, N. Giani, , C. S. Bevis, D. E. Adams, G. F. Mancini and G. Pellegrini, "Enabling Real-Time Ultrafast Ptychographic Imaging through Unsupervised Deep Learning", submitted (2024)
Please note that because these publications, currently accepted or under review, and already acknowledging EU funding, will be made available ‘open access’ as soon as possible.
During the timeframe of the Action, I also embed myself within the community of the Physics Department of the University of Pavia and LUXEM.
Teaching:
I gave three guest lectures in the Department’s course on Ultrafast Laser Physics regarding ultrafast pulse characterization. I also gave a comprehensive course on Coherent Diffractive Imaging and phase retrieval to members of LUXEM during the fall of 2022.
Outreach & Dissemination:
I was a participant in the European Researchers Night through the SHARPER program in Pavia, IT, where the university setup more than 40 booths at in the Castello Visconteo in downtown Pavia, where the research activities of scientists throughout the University of Pavia were showcased to the general public at the “Scienza in Castello” event. Moreover, I was invited to join the “Sumo Science” outreach program, which bridges the world of research and high school throughout Italy.
As part of my outreach and dissemination activities, I drafted an original Blog article for Dectris. Through the collaboration with Dectris on ptychographic lensless imaging with electron sources, Prof. Mancini and I were invited to contribute a blog article about the potential uses and future of the ptychography imaging technique in this new and burgeoning field. The article appears on the Blog section of the Dectris’s website regarding electron microscopy in September 2022, titled “Ptychography – With a ‘P’ As in ‘Pterodactyl”. This blog is Open Access; DECIPHER and the MSCA Action are acknowledged therein in compliance with the OpenAIRE guidelines. Available at the following address: https://www.dectris.com/company/news/blog/electron-microscopy/the-emerging-possibilities-in-ptychography/(s’ouvre dans une nouvelle fenêtre)
Professional Development:
We continued strengthening the close collaborative ties between LUXEM and members of the Consortium through two onsite visits to the newly constructed laboratory spaces LUXEM has in the Physics Department. We hosted Prof. Daniel Adams and one of his graduate students from the Colorado School of Mines in Golden, CO, USA during the last week of June. We took this opportunity to work in person on several research projects we are actively engaged in, the most prominent being the investigation into new reconstruction techniques and data processing methods for ultrafast pump-probe experiments. Prof. Adams gave a research seminar to the department where he summarized ongoing research efforts his lab is engaged in, and we were able to discuss knowledge transfer between his lab and LUXEM. Then, in October, we hosted Prof. Fabrizio Carbone from EPFL and a member of the DECIPHER Consortium. He gave a seminar to the members of LUXEM and we discussed the new ultrafast laser source, gained feedback on the experimental design of the UEDI system, and potential avenues for collaboration in the future.
In July 2022, I travelled internationally to Montreal, CA to participate in the International Conference on Ultrafast Phenomena, where my original contribution was accepted for a poster presentation. For both poster and associated peer-reviewed proceeding the visibility of the EU flag and to acknowledge EU funding was ensured. Here, I networked with peers and presented original research on the novel computational imaging system we designed to be housed in the state-of-the-art LUXEM lab space.