Periodic Reporting for period 2 - NANOVR (Nanoscale Design using Virtual Reality)
Okres sprawozdawczy: 2023-11-01 do 2025-04-30
Atoms and molecules are the building blocks of matter, making up everything that surrounds us. Much of the mechanics of the natural world occurs at the level of the atoms and molecules, in a realm that is invisible to our eyes. We are accustomed to imagining molecules as they are shown in textbooks, leading to a false impression that the molecular world is static. In fact, the molecular world is dynamic, with its atomic and molecular building blocks perpetually engaged in a tightly correlated dance that depends on how a molecule’s energy field interacts with that of its surroundings. As Richard Feynmann famously stated, “all of biology can be explained by the wigglings and jigglings of atoms”.
As scientists have made progress engineering the structures of molecular systems at the nano-scale, a new fundamental challenge has emerged: namely, our ability to understand and engineer molecular dynamics and flexibility. Understanding molecular dynamics and flexibility – on the level of cells, molecules, atoms, and electrons – has impacts for our everyday lives, and allows us to make strides developing important technologies in areas like medicine, energy, and the environment. The NANOVR research program aims to build computational tools which increase our understanding of the microscopic processes that guide molecular dynamics and flexibility, allowing us to predict, control, and design matter with atomic and molecular precision – helping us to tackle environmental, energy, and health challenges.
Understanding the process of molecular transformation is difficult in part because it involves complex 3D structures changing into other complex 3D structures. The latest advances in virtual reality (VR) offer exciting new opportunities for visualizing and understanding the dynamics of complex high dimensional systems. By combining the latest in VR technology with the state-of-the-art in high performance computing [HPC], and Artificial Intelligence (AI), the EU-funded NANOVR project aims to develop new approaches for nano-scale design, engineering, and simulation which are implemented within NanoVer, a software framework which enables research specialists and citizen scientists to use VR-enabled interactive simulations to literally 'reach out and touch' real-time molecular simulations, steering their dynamical pathways in real-time, and then training AIs which can offer new suggestions for the microscopic processes associated with transformation on the level of single molecules. Through coordination and collaboration across an international network of NANOVR ‘nodes’, this framework will provide a new way to understand the dynamics and behavior of complex 3D molecular systems, which researchers and citizen scientists can apply to a range of problems – for example understanding the molecular mechanism of drug molecules, biochemistry, materials chemistry, & catalysis.
As scientists have made progress engineering the structures of molecular systems at the nano-scale, a new fundamental challenge has emerged: namely, our ability to understand and engineer molecular dynamics and flexibility. Understanding molecular dynamics and flexibility – on the level of cells, molecules, atoms, and electrons – has impacts for our everyday lives, and allows us to make strides developing important technologies in areas like medicine, energy, and the environment. The NANOVR research program aims to build computational tools which increase our understanding of the microscopic processes that guide molecular dynamics and flexibility, allowing us to predict, control, and design matter with atomic and molecular precision – helping us to tackle environmental, energy, and health challenges.
Understanding the process of molecular transformation is difficult in part because it involves complex 3D structures changing into other complex 3D structures. The latest advances in virtual reality (VR) offer exciting new opportunities for visualizing and understanding the dynamics of complex high dimensional systems. By combining the latest in VR technology with the state-of-the-art in high performance computing [HPC], and Artificial Intelligence (AI), the EU-funded NANOVR project aims to develop new approaches for nano-scale design, engineering, and simulation which are implemented within NanoVer, a software framework which enables research specialists and citizen scientists to use VR-enabled interactive simulations to literally 'reach out and touch' real-time molecular simulations, steering their dynamical pathways in real-time, and then training AIs which can offer new suggestions for the microscopic processes associated with transformation on the level of single molecules. Through coordination and collaboration across an international network of NANOVR ‘nodes’, this framework will provide a new way to understand the dynamics and behavior of complex 3D molecular systems, which researchers and citizen scientists can apply to a range of problems – for example understanding the molecular mechanism of drug molecules, biochemistry, materials chemistry, & catalysis.
We have carried out work to develop a new ‘human-in-the-loop’ approach which enables researchers to use virtual reality to obtain insight into the dynamics of molecular change at the molecular level. The approach combines interactive molecular dynamics in virtual reality (iMD‐VR) with free energy (FE) calculation methods, combining human spatial and chemical intuition on the one hand with machine automation on the other. We have shown that the iMD‐VR‐FE method allows researchers to obtain both qualitative and quantitative insight into the mechanics by which drug-like molecules bind to proteins.
iMD-VR depends on a state-of-the-art multi-person VR software called ‘NanoVer’, whose development has been made possible with ERC funding, and which is available open source on GitHub along with extensive documentation to facilitate its usage, development, and application by scientists, educators, software developers, and VR enthusiasts (https://nanover.org/(odnośnik otworzy się w nowym oknie)). Currently, NanoVer is the only available open-source framework for running multi-person iMD-VR using the latest generation of commodity wireless ‘standalone’ AR/VR headsets such as the Meta Quest 2 and Meta Quest 3. The open-source nature of the NanoVer software means that it is being utilized by scientists to support a wide range of applications in areas related to human health including drug design, enzyme catalysis, protein design, and biomolecular signaling. More information at www.nanover.org
Design, development, and testing of the NanoVer software has been made possible by establishing a distributed network of NANOVR ‘nodes’, which are physical spaces equipped with VR hardware that runs the NanoVer software. The distributed node network enables us to test NanoVer’s ability to facilitate remote collaboration within the same virtual space despite users being remotely distributed in different physical spaces. The NANOVR node network enabled us to carry out our first ‘citizen-science’ gamified iMD-VR study (https://arxiv.org/abs/2409.07836(odnośnik otworzy się w nowym oknie)) measuring specialist and non-specialist users’ abilities to perceive subtle differences in the stiffness of molecular structures via molecular manipulation using handheld wireless controllers and hand-tracking. The results suggest that iMD-VR provides a novel, embodied approach for understanding molecular systems, enabling researchers to ‘sense’ molecular properties, and paving the way for future citizen science data collection efforts.
Given the extent to which Artificial intelligence (AI) is making possible advances across both science and industry, we have similarly made progress in understanding how iMD-VR can be combined with the latest AI advances to drive progress. Specifically, we have obtained proof-of-principle results illustrating that AIs can learn from human guided iMD-VR sampling strategies https://arxiv.org/abs/2409.07189(odnośnik otworzy się w nowym oknie). The rich datasets generated from iMD-VR – which capture human experts’ spatial insight regarding molecular structure and dynamics – can be used to train AI agents via a strategy called imitation learning (IL). This is an exciting result, because it suggests that AI agents can learn from human expertise in order to efficiently navigate vast conformational spaces, and provide insight that drives advances in important domains like materials science and biochemistry.
iMD-VR depends on a state-of-the-art multi-person VR software called ‘NanoVer’, whose development has been made possible with ERC funding, and which is available open source on GitHub along with extensive documentation to facilitate its usage, development, and application by scientists, educators, software developers, and VR enthusiasts (https://nanover.org/(odnośnik otworzy się w nowym oknie)). Currently, NanoVer is the only available open-source framework for running multi-person iMD-VR using the latest generation of commodity wireless ‘standalone’ AR/VR headsets such as the Meta Quest 2 and Meta Quest 3. The open-source nature of the NanoVer software means that it is being utilized by scientists to support a wide range of applications in areas related to human health including drug design, enzyme catalysis, protein design, and biomolecular signaling. More information at www.nanover.org
Design, development, and testing of the NanoVer software has been made possible by establishing a distributed network of NANOVR ‘nodes’, which are physical spaces equipped with VR hardware that runs the NanoVer software. The distributed node network enables us to test NanoVer’s ability to facilitate remote collaboration within the same virtual space despite users being remotely distributed in different physical spaces. The NANOVR node network enabled us to carry out our first ‘citizen-science’ gamified iMD-VR study (https://arxiv.org/abs/2409.07836(odnośnik otworzy się w nowym oknie)) measuring specialist and non-specialist users’ abilities to perceive subtle differences in the stiffness of molecular structures via molecular manipulation using handheld wireless controllers and hand-tracking. The results suggest that iMD-VR provides a novel, embodied approach for understanding molecular systems, enabling researchers to ‘sense’ molecular properties, and paving the way for future citizen science data collection efforts.
Given the extent to which Artificial intelligence (AI) is making possible advances across both science and industry, we have similarly made progress in understanding how iMD-VR can be combined with the latest AI advances to drive progress. Specifically, we have obtained proof-of-principle results illustrating that AIs can learn from human guided iMD-VR sampling strategies https://arxiv.org/abs/2409.07189(odnośnik otworzy się w nowym oknie). The rich datasets generated from iMD-VR – which capture human experts’ spatial insight regarding molecular structure and dynamics – can be used to train AI agents via a strategy called imitation learning (IL). This is an exciting result, because it suggests that AI agents can learn from human expertise in order to efficiently navigate vast conformational spaces, and provide insight that drives advances in important domains like materials science and biochemistry.
To date, the NANOVR project has advanced the state-of-the-art in a number of ways:
1) iMD‐VR‐FE is the first published methodology which allows researchers to obtain both qualitative and quantitative insight into the mechanics and free energy curves which characterize biomolecular dynamics, e.g. for the binding of drug-like molecules to proteins. Moving forward, we aim to expand the methodological toolset for performing quantitative analysis of human-sampled iMD-VR results, with demonstrations on various protein-ligand binding systems, including for example systems which are important for:
a. designing new antibiotics (e.g. the binding of small proteins to the PatGMac enzyme)
b. understanding the mechanisms of anti-anxiety medications (specifically the binding of serotonin analogues to the human HT2A-serotonin receptor)
c. treating flu epidemics (the binding of Tamiflu to the neuraminidase protein)
2) As the only open-source framework for running multi-person iMD-VR using the latest generation of commodity wireless ‘standalone’ AR/VR headsets such as the Meta Quest 2 and Meta Quest 3, NanoVer is being utilized by scientists to support a wide range of applications in areas related to human health including drug design (e.g. for diseases like tuberculosis), enzyme catalysis, protein design, and biomolecular energy-transport. Moving forward, we plan to work with our NANOVR node network to expand the visualization and interaction features which are available within NanoVer, and thereby increase its utility for researchers and educators. For example, we are just now undertaking a collaboration with developers of UnityMol in order to combine the visualization algorithms available in UnityMol with the underlying multi-person platform and interaction algorithms available in NanoVer. Moving forward, we aim to improve the existing cloud infrastructure which enables real-time simulation.
3) We carried out the first ever ‘citizen-science’ gamified iMD-VR study, in which we measured the ability of specialist and non-specialist users to perceive subtle differences in the stiffness of molecular structures in VR. The results are the first of their kind, suggesting that iMD-VR provides a novel, embodied approach for understanding molecular systems, enabling researchers to ‘sense’ molecular properties. Moving forward we intend to improve the NanoVer user experience in order to improve its intuitive usability by non-specialist ‘citizen-scientists’, with the idea that they can then help us address the applications detailed above. We also intend to investigate the most efficient ways in which humans can express their intuition (and the extent to which drawing in VR might be an easy way to express 3d spatial intuition), and also eventually work toward an app which is easily on the Meta Quest app store.
4) Our results showing that AIs can learn from human guided iMD-VR sampling are the first-of-their-kind, and pave the way for AI agents learn from human expertise expressed in VR order to efficiently navigate vast conformational spaces in important domains like materials science and biochemistry. Moving forward we aim to expand these results beyond their current ‘proof-of-principle’ stage, in order to investigate whether they can be generalized to a broader range of systems, with increasing levels of complexity. Using data obtained from both research experts and citizen scientists, we aim to construct high-quality datasets for training AIs that can help provide insight into the applications mentioned above.
1) iMD‐VR‐FE is the first published methodology which allows researchers to obtain both qualitative and quantitative insight into the mechanics and free energy curves which characterize biomolecular dynamics, e.g. for the binding of drug-like molecules to proteins. Moving forward, we aim to expand the methodological toolset for performing quantitative analysis of human-sampled iMD-VR results, with demonstrations on various protein-ligand binding systems, including for example systems which are important for:
a. designing new antibiotics (e.g. the binding of small proteins to the PatGMac enzyme)
b. understanding the mechanisms of anti-anxiety medications (specifically the binding of serotonin analogues to the human HT2A-serotonin receptor)
c. treating flu epidemics (the binding of Tamiflu to the neuraminidase protein)
2) As the only open-source framework for running multi-person iMD-VR using the latest generation of commodity wireless ‘standalone’ AR/VR headsets such as the Meta Quest 2 and Meta Quest 3, NanoVer is being utilized by scientists to support a wide range of applications in areas related to human health including drug design (e.g. for diseases like tuberculosis), enzyme catalysis, protein design, and biomolecular energy-transport. Moving forward, we plan to work with our NANOVR node network to expand the visualization and interaction features which are available within NanoVer, and thereby increase its utility for researchers and educators. For example, we are just now undertaking a collaboration with developers of UnityMol in order to combine the visualization algorithms available in UnityMol with the underlying multi-person platform and interaction algorithms available in NanoVer. Moving forward, we aim to improve the existing cloud infrastructure which enables real-time simulation.
3) We carried out the first ever ‘citizen-science’ gamified iMD-VR study, in which we measured the ability of specialist and non-specialist users to perceive subtle differences in the stiffness of molecular structures in VR. The results are the first of their kind, suggesting that iMD-VR provides a novel, embodied approach for understanding molecular systems, enabling researchers to ‘sense’ molecular properties. Moving forward we intend to improve the NanoVer user experience in order to improve its intuitive usability by non-specialist ‘citizen-scientists’, with the idea that they can then help us address the applications detailed above. We also intend to investigate the most efficient ways in which humans can express their intuition (and the extent to which drawing in VR might be an easy way to express 3d spatial intuition), and also eventually work toward an app which is easily on the Meta Quest app store.
4) Our results showing that AIs can learn from human guided iMD-VR sampling are the first-of-their-kind, and pave the way for AI agents learn from human expertise expressed in VR order to efficiently navigate vast conformational spaces in important domains like materials science and biochemistry. Moving forward we aim to expand these results beyond their current ‘proof-of-principle’ stage, in order to investigate whether they can be generalized to a broader range of systems, with increasing levels of complexity. Using data obtained from both research experts and citizen scientists, we aim to construct high-quality datasets for training AIs that can help provide insight into the applications mentioned above.