The goal of INTERACT is to provide mobile robots, and fleets thereof, with the cognitive ability to reason about their interaction with other agents in their environment. We have contributed in several research fronts:
Motion planning:
Game-theory provides a natural mathematical tool for robot motion planning in interactive multi-agent settings. However, tractable algorithms usually rely on a strong assumption, namely that the objectives of all players in the scene are known. To make such tools applicable for ego-centric planning with only local information, we developed an adaptive model-predictive game solver [Liu2023], which jointly infers other players’ objectives online and computes a corresponding generalized Nash equilibrium strategy. Further, we addressed shortcomings of existing maximum likelihood estimation (MLE) approaches that solve inverse games that provide only point estimates of unknown parameters. Since they cannot quantify uncertainty, their performance suffers when many parameter values explain the observed behavior. To address these limitations, we take a Bayesian perspective and construct posterior distributions of game parameters. To render inference tractable, we employ a variational autoencoder (VAE) with an embedded differentiable game solver [Liu2024]. This structured VAE can be trained from an unlabeled dataset of observed interactions, naturally handles continuous, multi-modal distributions, and supports efficient sampling from the inferred posteriors without computing game solutions at runtime.
Uncertainty of interactions is captured in the problem of contingency planning, wherein an agent generates a set of possible plans. We extended our game-theoretic approach to contingency planning, tailored to multi-agent scenarios in which a robot’s actions impact the decisions of other agents and vice versa [Peters2024]. The resulting contingency game allows the robot to efficiently interact with other agents by generating strategic motion plans conditioned on multiple possible intents for other actors in the scene.
Lastly, micro-interaction is also a key problem in real-time motion planning for multiple robotic manipulators that operate in proximity. We build upon the concept of dynamic fabrics and extend them to multi-robot systems, referred to as Multi-Robot Dynamic Fabrics (MRDF) [Bakker2023]. This geometric method enables a very high planning frequency for high-dimensional systems at the expense of being reactive and prone to deadlocks. To detect and resolve deadlocks, we propose Rollout Fabrics where MRDF are forward simulated in a decentralized manner. We are now extending fabrics to include manipulation [Merva2024] and learning from demonstration [Bakker2024].
Task assignment:
Task and Motion Planning (TAMP) has made strides in complex manipulation tasks, yet the execution robustness of the planned solutions remains overlooked. We developed a method for reactive TAMP to cope with runtime uncertainties and disturbances. We combine an Active Inference planner (AIP) for adaptive high-level action selection and a novel Multi-Modal Model Predictive Path Integral controller (M3P2I) for low-level control [Zhang2024]. This results in a scheme that simultaneously adapts both high-level actions and low-level motions. The AIP generates alternative symbolic plans, each linked to a cost function for M3P2I. This idea enables blending different robot skills for fluid and reactive plan execution, accommodating plan adjustments at both the high and low levels to cope, for instance, with dynamic obstacles or disturbances that invalidate the current plan.
A common challenge in TAMP remains the required simplifications and abstractions, such that the resulting plans often do not account for robot dynamics, nor complex contacts. They also often ignore the effect of the low-level controllers on the optimality and/or feasibility of the plan's realizations. We are investigating the use of a parallelized physics simulator to compute realizations of the plan with a motion controller, realistic dynamics, and considering contacts with the environment [Matoses2024]. Using cross-entropy optimization, we sample the parameters used by the controllers, or actions, to obtain low-cost solutions implementable in the robot.
Multi-robot mobile manipulation:
Lastly, we have built a system formed by two mobile manipulators, see
https://youtu.be/AldMFKnlW3M(opens in new window)
