People navigate, manipulate and reason about the physical world around them with remarkable ease. When watching someone change a car tire or follow a cooking recipe, a person can later perform the same task themselves, drawing on a lifetime of embodied experience with everyday objects, physical cause and effect, and three-dimensional (3D) space. Today’s artificial intelligence systems show remarkable performance on tasks once considered challenging by many people — such as writing computer programs, generating images or holding conversations — yet other tasks that humans find easy and perform every day, such as perceiving and physically interacting with the surrounding world, remain difficult for machines. For example, commonly used industrial robots are pre-programmed for one specific task but are unable to easily perform other tasks in the changing 3D world.
The FRONTIER project aims to develop a new generation of AI models for systems that have a physical body, operate in a dynamic 3D world, and interact with their surroundings to achieve complex goals. Examples of such embodied systems include a robot as a home assistant, a robot in a factory, a self-driving car, or even a robotic surgeon. All these systems have a physical body, need to perceive the world around them, reason about it, and interact with it — capabilities that current large neural models are only just beginning to master.
To achieve this, FRONTIER addresses three scientific challenges. First, developing new neural architectures that combine large-scale neural networks with the geometry and physics of the 3D world, so that models can reason about how objects move, deform and contact one another. Second, developing learning methods that scale to vast amounts of uncurated visual data — for example, the millions of hours of instructional and tutorial videos available online — while incorporating physical and geometric structure, with the aim of bringing machines closer to humans, who can often learn from only a few available examples. Third, developing learning methods that allow many embodied systems to share and accumulate learning experiences across different environments and tasks, thereby achieving new levels of scale, accuracy and robustness not achievable by learning in any individual system alone.
Breakthrough progress on these problems would have major impact on our daily lives. The findings could lead to safer cars that learn from each other, intelligent production lines that collaboratively adapt to new workflows, or a new generation of intelligent assistive robots that automatically learn new skills from the Internet and each other. The first two years have additionally opened a new high-impact research direction, adapting the FRONTIER methodology to biomolecular systems in protein engineering and drug design. Biomolecular interactions are a natural analog of physical interactions between a robot and surrounding objects: both are governed by 3D geometry, forces, and dynamics, and both require learning from limited labeled data under strong structural priors. This extension does not change the goals of the proposal; the objectives remain as planned, and gain a parallel application domain of major scientific and societal importance.
