Periodic Reporting for period 1 - INTERACT (Intuitive interaction for robots among humans)
Reporting period: 2022-09-01 to 2025-02-28
Thanks to developments in computing, sensing and communication technology, great progress has been made in the control of robots affixed to the factory floor, mobile robots in controlled environments, and self-driving cars in medium-complexity environments. However, mobile robots are not yet capable of seamless interaction in human populated environments, primarily due to the high complexity of modeling and reasoning over the effect that robot actions have on other robots and humans, and the inherent uncertainty. Humans rely on intuition, namely the ability to understand interaction instinctively, without the need for conscious reasoning. However, for control systems it is notoriously difficult to model intuition, and to do so with safety guarantees with respect to physical constraints. The goal of INTERACT is to provide mobile robots with the ability to reason about their coordination with other agents and the associated risks, take appropriate actions and continuously reassess.
In INTERACT we will reach this goal by developing an innovative approach across the boundaries of Motion Planning, Multi-robot Task Assignment and Machine Learning. In INTERACT we propose a holistic view on the interaction of mobile robots and humans, where we consider multiple spatio-temporal granularities ranging from individual interactions to the interaction of a robot fleet with the humans in a city, and from short term (local) to long term (global) effects of the interaction. Robots will use past experience to learn local and global intuition models of their interaction with the environment. These intuition models will be integrated in novel uncertainty-aware optimization methods to compute safe interaction-aware trajectories, task assignments and routes for mobile robots.
INTERACT will lay the foundation for intuitive multi-robot interaction, make it possible for teams of mobile robots to safely interact in human-centric environments and enable a new level of automation in factories and cities.
In INTERACT we will reach this goal by developing an innovative approach across the boundaries of Motion Planning, Multi-robot Task Assignment and Machine Learning. In INTERACT we propose a holistic view on the interaction of mobile robots and humans, where we consider multiple spatio-temporal granularities ranging from individual interactions to the interaction of a robot fleet with the humans in a city, and from short term (local) to long term (global) effects of the interaction. Robots will use past experience to learn local and global intuition models of their interaction with the environment. These intuition models will be integrated in novel uncertainty-aware optimization methods to compute safe interaction-aware trajectories, task assignments and routes for mobile robots.
INTERACT will lay the foundation for intuitive multi-robot interaction, make it possible for teams of mobile robots to safely interact in human-centric environments and enable a new level of automation in factories and cities.
The goal of INTERACT is to provide mobile robots, and fleets thereof, with the cognitive ability to reason about their interaction with other agents in their environment. We have contributed in several research fronts:
Motion planning:
Game-theory provides a natural mathematical tool for robot motion planning in interactive multi-agent settings. However, tractable algorithms usually rely on a strong assumption, namely that the objectives of all players in the scene are known. To make such tools applicable for ego-centric planning with only local information, we developed an adaptive model-predictive game solver [Liu2023], which jointly infers other players’ objectives online and computes a corresponding generalized Nash equilibrium strategy. Further, we addressed shortcomings of existing maximum likelihood estimation (MLE) approaches that solve inverse games that provide only point estimates of unknown parameters. Since they cannot quantify uncertainty, their performance suffers when many parameter values explain the observed behavior. To address these limitations, we take a Bayesian perspective and construct posterior distributions of game parameters. To render inference tractable, we employ a variational autoencoder (VAE) with an embedded differentiable game solver [Liu2024]. This structured VAE can be trained from an unlabeled dataset of observed interactions, naturally handles continuous, multi-modal distributions, and supports efficient sampling from the inferred posteriors without computing game solutions at runtime.
Uncertainty of interactions is captured in the problem of contingency planning, wherein an agent generates a set of possible plans. We extended our game-theoretic approach to contingency planning, tailored to multi-agent scenarios in which a robot’s actions impact the decisions of other agents and vice versa [Peters2024]. The resulting contingency game allows the robot to efficiently interact with other agents by generating strategic motion plans conditioned on multiple possible intents for other actors in the scene.
Lastly, micro-interaction is also a key problem in real-time motion planning for multiple robotic manipulators that operate in proximity. We build upon the concept of dynamic fabrics and extend them to multi-robot systems, referred to as Multi-Robot Dynamic Fabrics (MRDF) [Bakker2023]. This geometric method enables a very high planning frequency for high-dimensional systems at the expense of being reactive and prone to deadlocks. To detect and resolve deadlocks, we propose Rollout Fabrics where MRDF are forward simulated in a decentralized manner. We are now extending fabrics to include manipulation [Merva2024] and learning from demonstration [Bakker2024].
Task assignment:
Task and Motion Planning (TAMP) has made strides in complex manipulation tasks, yet the execution robustness of the planned solutions remains overlooked. We developed a method for reactive TAMP to cope with runtime uncertainties and disturbances. We combine an Active Inference planner (AIP) for adaptive high-level action selection and a novel Multi-Modal Model Predictive Path Integral controller (M3P2I) for low-level control [Zhang2024]. This results in a scheme that simultaneously adapts both high-level actions and low-level motions. The AIP generates alternative symbolic plans, each linked to a cost function for M3P2I. This idea enables blending different robot skills for fluid and reactive plan execution, accommodating plan adjustments at both the high and low levels to cope, for instance, with dynamic obstacles or disturbances that invalidate the current plan.
A common challenge in TAMP remains the required simplifications and abstractions, such that the resulting plans often do not account for robot dynamics, nor complex contacts. They also often ignore the effect of the low-level controllers on the optimality and/or feasibility of the plan's realizations. We are investigating the use of a parallelized physics simulator to compute realizations of the plan with a motion controller, realistic dynamics, and considering contacts with the environment [Matoses2024]. Using cross-entropy optimization, we sample the parameters used by the controllers, or actions, to obtain low-cost solutions implementable in the robot.
Multi-robot mobile manipulation:
Lastly, we have built a system formed by two mobile manipulators, see https://youtu.be/AldMFKnlW3M(opens in new window)
Motion planning:
Game-theory provides a natural mathematical tool for robot motion planning in interactive multi-agent settings. However, tractable algorithms usually rely on a strong assumption, namely that the objectives of all players in the scene are known. To make such tools applicable for ego-centric planning with only local information, we developed an adaptive model-predictive game solver [Liu2023], which jointly infers other players’ objectives online and computes a corresponding generalized Nash equilibrium strategy. Further, we addressed shortcomings of existing maximum likelihood estimation (MLE) approaches that solve inverse games that provide only point estimates of unknown parameters. Since they cannot quantify uncertainty, their performance suffers when many parameter values explain the observed behavior. To address these limitations, we take a Bayesian perspective and construct posterior distributions of game parameters. To render inference tractable, we employ a variational autoencoder (VAE) with an embedded differentiable game solver [Liu2024]. This structured VAE can be trained from an unlabeled dataset of observed interactions, naturally handles continuous, multi-modal distributions, and supports efficient sampling from the inferred posteriors without computing game solutions at runtime.
Uncertainty of interactions is captured in the problem of contingency planning, wherein an agent generates a set of possible plans. We extended our game-theoretic approach to contingency planning, tailored to multi-agent scenarios in which a robot’s actions impact the decisions of other agents and vice versa [Peters2024]. The resulting contingency game allows the robot to efficiently interact with other agents by generating strategic motion plans conditioned on multiple possible intents for other actors in the scene.
Lastly, micro-interaction is also a key problem in real-time motion planning for multiple robotic manipulators that operate in proximity. We build upon the concept of dynamic fabrics and extend them to multi-robot systems, referred to as Multi-Robot Dynamic Fabrics (MRDF) [Bakker2023]. This geometric method enables a very high planning frequency for high-dimensional systems at the expense of being reactive and prone to deadlocks. To detect and resolve deadlocks, we propose Rollout Fabrics where MRDF are forward simulated in a decentralized manner. We are now extending fabrics to include manipulation [Merva2024] and learning from demonstration [Bakker2024].
Task assignment:
Task and Motion Planning (TAMP) has made strides in complex manipulation tasks, yet the execution robustness of the planned solutions remains overlooked. We developed a method for reactive TAMP to cope with runtime uncertainties and disturbances. We combine an Active Inference planner (AIP) for adaptive high-level action selection and a novel Multi-Modal Model Predictive Path Integral controller (M3P2I) for low-level control [Zhang2024]. This results in a scheme that simultaneously adapts both high-level actions and low-level motions. The AIP generates alternative symbolic plans, each linked to a cost function for M3P2I. This idea enables blending different robot skills for fluid and reactive plan execution, accommodating plan adjustments at both the high and low levels to cope, for instance, with dynamic obstacles or disturbances that invalidate the current plan.
A common challenge in TAMP remains the required simplifications and abstractions, such that the resulting plans often do not account for robot dynamics, nor complex contacts. They also often ignore the effect of the low-level controllers on the optimality and/or feasibility of the plan's realizations. We are investigating the use of a parallelized physics simulator to compute realizations of the plan with a motion controller, realistic dynamics, and considering contacts with the environment [Matoses2024]. Using cross-entropy optimization, we sample the parameters used by the controllers, or actions, to obtain low-cost solutions implementable in the robot.
Multi-robot mobile manipulation:
Lastly, we have built a system formed by two mobile manipulators, see https://youtu.be/AldMFKnlW3M(opens in new window)
Our work so far contributes to the advancement of the robotics research field beyond the state-of-the art. The most significant ones to date are:
• We have introduced the concept of contingency games, which combines game theory with planning under uncertainty [Peters2024].
• We have proposed a unique task and motion planning method that combines the strengths of behavior trees and sampling based MPC (MPPI) [Zhang2024].
• We have proposed multi-robot geometric fabrics for the first time [Bakker2023].
• We have created a demonstrator with two mobile manipulators.
• We have introduced the concept of contingency games, which combines game theory with planning under uncertainty [Peters2024].
• We have proposed a unique task and motion planning method that combines the strengths of behavior trees and sampling based MPC (MPPI) [Zhang2024].
• We have proposed multi-robot geometric fabrics for the first time [Bakker2023].
• We have created a demonstrator with two mobile manipulators.