3D understanding is crucial for autonomous systems to estimate distances for obstacle avoidance and to understand object shapes for interaction. Before 2020, most 3D modeling relied on traditional methods that did not involve deep learning. With the emergence of differentiable neural rendering techniques, 3D modeling experienced a revolution-like shift, almost a decade after the deep learning revolution in other areas. Neural Radiance Fields (NeRF) were the first to enable the reconstruction of highly detailed 3D models, which had not been possible with traditional approaches. More recently, 3D Gaussian Splatting (3DGS) achieved comparable quality while being significantly faster. These models are trained on images captured from multiple viewpoints and can render high-quality images from novel viewpoints that were not seen during training.
In our work, we first showed that despite their impressive rendering quality, these models often lack a reliable geometric understanding of the scene. This limitation poses a potential risk for autonomous systems interacting with the physical world. We proposed a method to improve the geometric understanding of 3DGS models. In follow-up work, we developed a training-free approach to quantify uncertainty by exploiting inconsistencies in their geometric predictions. This uncertainty can be used for exploration, for example, by guiding autonomous systems to further investigate parts of the scene that are deemed unreliable.
Another key capability for autonomous systems is temporal processing over long time horizons. Humans can process videos spanning minutes or even hours, whereas many current AI systems operate on individual frames or have limited temporal context. Only recently has research begun to address tasks that require processing hundreds of frames, corresponding to just a few minutes of video. A representative example of long-term temporal processing is point tracking. Given a point in the first frame of a video, the task is to track that point over hundreds of frames, even when its appearance changes or it temporarily disappears. Early approaches relied on access to the entire video, which is incompatible with real-world robotic systems that must operate online, using only past information.
To address this challenge, we proposed a simple architecture with an explicit memory mechanism that enables online long-term point tracking. This memory allows the system to relate information across time while processing frames sequentially. To further improve suitability for autonomous systems, we focused on reducing computational cost and memory usage, making the approach more efficient for real-time deployment.
For safe decision-making, AI models must also be aware of their own limitations. This requires explicit uncertainty modeling. One fundamental source of uncertainty arises from unknown object categories. While models are typically trained to recognize a fixed set of known classes, they must also indicate uncertainty when encountering objects that fall outside this set. We studied this problem for both segmentation, which assigns object labels at the pixel level, and detection, which localizes objects using bounding boxes. Building on advances in foundation models, we developed a specialized uncertainty module for segmentation that contrasts known and unknown categories. Detecting unknown objects proved more challenging due to the risk of forgetting known categories. Inspired by human cognition, which emphasizes shared properties of known objects, we proposed an approach that enables the localization of unknown objects without degrading performance on known classes.
Finally, we developed driving-specific methods addressing both representation learning and action. For building static maps of the environment, including roads, sidewalks, and pedestrian crossings, we proposed a method that adapts to geometric variations such as road slope while filtering out irrelevant visual information, such as vehicles and pedestrians. For action, we investigated the use of large transformer-based models for driving. We adapted architectures inspired by LLMs, which operate on discrete tokens, by representing visual input in an object-centric manner. Each object is encoded as a structured unit, or “slot,” providing a compact summary of the scene. We showed that these object-centric representations support both effective action prediction and accurate forecasting of future object locations.
