WP1 focused on developing end-to-end deep learning architectures that combine the previously separated areas of object detection, segmentation, and tracking under the label "Video Object Segmentation (VOS)". We were able to significantly push ahead this research direction through a series of high-impact publications that advanced the state-of-the-art in VOS, including OnAVOS [BMVC'17], PReMVOS [ACCV'18], FeelVOS [CVPR'19], UnOVOST [WACV'20], STEm-Seg [ECCV'20], HODOR [CVPR'22 oral], and TarVIS [CVPR'23 highlight]. Those approaches created important milestones in the progression of the VOS field and are highly cited. In addition, our group achieved top results in 6 VOS challenges in 2018 and 2019, including 1st places in the CVPR'18 and CVPR'19 DAVIS Challenge competitions and the ECCV'18 and ICCV'19 YouTube-VOS Challenges.
WP2 aimed at building end-to-end deep learning approaches for single- and multi-object tracking. We made good progress towards this goal and developed state-of-the-art approaches for both tasks (SIAM-RCNN [CVPR'20], T2R-R2T [ICRA'20]). Moreover, we defined the task of "Multi-Object Tracking and Segmentation" (MOTS) [CVPR'19] and established it in the computer vision community, together with an annotated benchmark dataset and evaluation methodology (MOTS-Challenge). As previous tracking evaluation methods suffered from systematic problems, we developed a novel tracking evaluation methodology, Higher-Order Tracking Accuracy (HOTA) [IJCV'20], which has by now become a new standard for tracking evaluation.
WP3 focused on developing deep learning approaches for human motion analysis. Main outcomes were novel approaches for 3D body pose estimation (MeTrO [FG'20], MeTrAbs [IEEE TBIOM'21]) that won competitions at ECCV'18 and ECCV'20. Finally, we developed a principled approach for multi-dataset training of body pose estimation models [WACV'23] that resulted in a new state of the art in 3D human pose estimation quality.
The goal of WP4 was to interface the geometry-centric representations from traditional Computer Vision methods with the semantic analysis capabilities of deep learning based vision modules. For this, we worked on developing deep learning representations for 3D point clouds and 3D geometry, resulting in novel approaches for 3D semantic segmentation (DualConvMesh-Net [CVPR'20]), 3D instance segmentation (3DBEVIS [GCPR'19], 3D-MPA [CVPR'20]), 3D data augmentation (Mix3D [3DV'21]), and representation learning (Point2Vec, [GCPR'23]). Finally, we developed a novel mask transformer based architecture for 3D semantic segmentation tasks (Mask3D [ICRA'23]).
WP5 addressed the problem of learning deep networks with reduced human supervision. Towards this goal, we developed 1) automated tools for interactive, human-assisted image segmentation (ITIS [BMVC'18], DynaMITe [ICCV'23]); 2) an automated workflow for human-assisted segmentation annotation of entire video datasets [WACV'21]; 3) category-agnostic multi-object tracking approaches (4D-GVT [ICRA'20], OWT [CVPR'22]) that can be used for automatic mining of candidate object tracks in large video collections [ICRA'19]. Those tools significantly reduce the manual annotation effort in the creation of large-scale training and evaluation datasets.
WP6 was originally intended to focus on dataset collection and annotation. Due to ethical and data protection concerns during the Ethical Review phase, the plan to record and collect new datasets was not pursued further. Instead, we focused our efforts on creating novel and detailed segmentation annotations for existing (and already publicly available) benchmark datasets using the partially automated annotation tools and workflows developed in WP5. This resulted in the creation of the MOTS-Challenge [CVPR'19], the TAO-OWT [CVPR'22 oral] , and the BURST [WACV'23] benchmark datasets.
