We developed a hierarchical model that models complex activities at different granularities. At the top level, complex activities like “preparing pancakes” or “preparing a fruit salad” are modelled. These complex activities require several sub-activities that need to be executed like “take egg”, “crack egg”, or “stir dough”. These sub-activities are the intermediate representation of the hierarchy. At the lowest level, fine-granular activities or motion primitives are modelled. For instance, cracking eggs involves a sequence of human movements. The hierarchical model processes continuous video streams and predicts for each frame what sub-activity is executed as well as the overall complex activity.
In order to learn the parameters of the model, annotated videos are required. The developed model has the advantage that it can be trained in two ways. In the first setting, we assume that the videos have been annotated in the same way as the model is expected to analyze the videos. This means that for each frame the ongoing sub-activity is annotated. This setting is known as learning with full supervision. Providing such a frame-wise labeling of videos, however, is an enormous effort and can be too expensive for practical applications. We therefore developed learning procedures that allow to learn the model with less supervision, i.e. weak supervision. We investigated different types of weak supervision including video tags or protocols. While video tags only summarize what actions are occurring in each video, protocols provide the temporal order of the actions occurring in each video. For protocols with timestamps, we achieved up to 97% of the accuracy compared to fully supervised learning while reducing the annotation cost by factor 6.
While the activities occurring in a video is an important aspect of a video, they do not describe the full content of a video. We therefore moved beyond the commonly studied activity recognition task and introduced a novel, holistic view on video understanding. Instead of recognizing only the activities, holistic video understanding aims also at recognizing the objects and their attributes that are involved, the scenery where the video has taken, as well as the general context in which the activities are happening. As pioneering work, we released a dataset consisting of about 580,000 videos annotated by about 3100 different categories organized in a hierarchically taxonomy, which resulted in about 7.5 million annotations. We also demonstrated that training a network on such richly annotated dataset improves the action recognition accuracy on other datasets.
We also addressed the problem how a model that is trained on one domain can be adapted to recognize activities in another domain. For instance, we might have a model trained on videos from YouTube, but we want that it recognizes activities from a video that is captured by a camera mounted on a service robot. Since the videos the model has been trained on and the videos the model has to analyze look different, the model needs to be adapted to handle the differences. We thus developed approaches that successfully adapt models to different modalities or domains.
The results of the project have been disseminated by 8 publications in journals, 36 publications in peer-reviewed conferences, and 7 publications in peer-reviewed workshops. Furthermore, 9 symposia, workshops, and tutorials have been organized. The source code and data of several publications have been publicly released.
