Model-based Reinforcement Learning for Versatile Robots in the Real World

Project Information

REAL-RL

Grant agreement ID: 101045454

DOI

10.3030/101045454

EC signature date 30 August 2022

Start date 1 January 2023

End date 31 December 2027

Funded under

• European Research Council (ERC)

Total cost

€ 1 998 500,00

EU contribution

€ 1 998 500,00

Coordinated by

EBERHARD KARLS UNIVERSITAET TUEBINGEN
Germany

Periodic Reporting for period 1 - REAL-RL (Model-based Reinforcement Learning for Versatile Robots in the Real World)

Reporting period: 2023-01-01 to 2025-06-30

Summary of the context and overall objectives of the project

The REAL-RL project aims to make robots smarter and more helpful in our everyday lives by teaching them to learn from their own experiences, much like humans do. Today’s robots are usually designed to do only one specific job, and making them work properly often requires very detailed knowledge about exactly how they should interact with the world around them. This makes it hard – or sometimes impossible – to use them for new or unexpected tasks.

REAL-RL takes a new approach that focuses on helping robots figure things out for themselves by trial and error. Most current methods in this area require robots to practice a huge number of times, which takes too long and can wear out real robots. As a shortcut, scientists use computer simulations, but these still need a lot of knowledge about every possible situation the robot might run into.

Our project is different: we want robots to learn their own "common sense" about how the world works, building up knowledge from their own experiences. With this knowledge, robots can quickly plan what to do and adapt when something unexpected happens—without needing tons of practice or super-detailed instructions.

We are tackling important scientific and technical challenges that stand in the way of this vision, including:
- How robots can build useful models of the world from experience.
- How they can safely explore and learn without making mistakes that could cause damage.
- How they can think ahead and make plans that take into account risks and different possible futures.

We’ve already made progress on some of the building blocks needed to make this happen, like computer programs that can guess what might happen next, or fast ways for robots to plan their actions.

Ultimately, if successful, this project will help build a new generation of robots that can use what they’ve learned to help humans with a wider range of tasks—from walking and carrying, to assembling items, to assisting around the house or workplace. These robots will not just follow instructions, but actually get better with practice, making them much more useful and adaptable in the future.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Over the past period, our team has been working on new ways to make robots learn from experience, so they can become more flexible and useful in the real world.

1. Helping robots understand how the world works.
We are teaching robots to build their own "mental models" of how they and their surroundings work, so they can predict what might happen next. For example, this could help a robot know how it will move when a certain force is applied. We tried using both traditional, physics-based tricks and newer, data-driven tricks—like learning directly from video footage. Early results are promising: we have working prototypes, and our research is being submitted to top scientific journals.

2. Making learning safer and more robust
For robots to learn without putting themselves or others at risk, we studied new ways to help them plan their actions more safely. We found that it’s not just about giving robots more data, but about giving them the right kind of data. We developed smarter ways for robots to learn from “what would have happened if…” scenarios, all while keeping them safe from risky mistakes. Several of our solutions are being published and shared with other researchers.

3. Teaching robots to make better decisions
We made improvements to how robots can plan ahead for tasks that involve complex decisions, like moving objects or navigating cluttered spaces.
Our new mathematical methods let computers "think ahead" during difficult choices, making planning much faster and more reliable. We have also begun exploring how large AI models, like those used in chatbots, can help robots reason through more complex situations.5. Powerful new tools

4. We are modifying an existing robot simulator to allow for a different way of obtaining control strategies for robots.
We’re also building a new type of simulator that learns directly from video, which could one day let robots “watch and learn” from what they see around them.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

One of our biggest successes has been creating ways for robots to learn about objects directly from regular video footage, making it easier for robots to understand what's in their environment. The method works on real-world videos and we foresee major advantages when combined this with vision-language-action models, the new large-scale neural networks for controlling robots.

Our modifications to the robot simulator allows using them as flexible world models for mental simulation and allows for efficient controller generation. If successful, we are much closer to our goal of a versatile learning robot.

Overview of the components of the project