In this project, we have developed several approaches that forecast human motion, human actions, and recognize errors induced by humans. An example application is a driving assistant system that forecasts the behaviour of other traffic participants to warn the driver before an accident happens. We thus developed an approach that takes videos from multiple cameras, which are mounted at the vehicle and monitor the surroundings, as input. It then detects other traffic participants in the videos and forecasts their motions for the next two to eight seconds simultaneously (Li et al., IJCAI 2023). Another work focuses on forecasting the motion of multiple socially interacting persons. To this end, we developed the first approach that was able to forecast the motion of multiple persons even for long-time horizons of 40 seconds and that models the uncertainty of the forecast motion (Tanke et al., ICCV 2023). We furthermore developed a metric that measures whether the forecast motion is socially plausible. In order to advance the state of the art further, we introduced a benchmark for evaluating approaches that forecast the motion of multiple persons in a natural working environment. The so-called “Humans in Kitchens” dataset (Tanke et al., NeurIPS 2023) is a large-scale multi-person 3D human motion dataset with annotated 3D human poses, scene geometry, and per-person activities. Overall, it consists of more than 4M annotated human poses of 90 individuals.
In order to improve the efficiency of forecasting approaches, we developed an approach, called TaylorSwiftNet, that forecasts continuously frames (Saber er al., BMVC 2022). TaylorSwiftNet is a novel approach that takes full advantage of a continuous representation of motion. In contrast to RNNs that forecast the future frame-by-frame or PDE-based approaches that discretize PDEs to solve them numerically, we infer a continuous function over time from the observations. This avoids discretization artifacts and provides an analytical function that can be swiftly evaluated for any future continuous point and allows to forecast future frames at a higher sampling rate than the observed frames, which are very useful properties for practical applications.
In order to detect human errors in the context of human-robot collaborations, we introduced a dataset and approach to recognize failures or errors that are caused by humans in human-robot collaborations (Thoduka et al., ICRA 2024). The proposed “Handover Failure Detection” dataset contains failures that are caused by the human participant in both robot-to-human and human-to-robot handovers. The dataset includes multimodal data such as video, robot joint state, and readings from a force-torque sensor. We also present a temporal action segmentation approach for jointly classifying the actions of the human participant and the robot, as well as recognizing failures. Since some human errors depend on an incorrect number of executions of certain actions, i.e. some actions are repeated too often and some not often enough, we also developed an approach for counting repetitive actions in videos (Luo et al., ICIP 2024). The approach is action-agnostic and can be used to detect human errors due to a wrong number of repetitions.
