Spatial 3D Semantic Understanding for Perception in the Wild

Projektinformationen

SpatialSem

ID Finanzhilfevereinbarung: 101076253

DOI

10.3030/101076253

EK-Unterschriftsdatum 1 Juni 2023

Startdatum 1 Oktober 2023

Enddatum 30 September 2028

Finanziert unter

• European Research Council (ERC)

Gesamtkosten

€ 1 500 000,00

EU-Beitrag

€ 1 500 000,00

Koordiniert durch

TECHNISCHE UNIVERSITAET MUENCHEN
Germany

Periodic Reporting for period 1 - SpatialSem (Spatial 3D Semantic Understanding for Perception in the Wild)

Berichtszeitraum: 2023-10-01 bis 2026-03-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Understanding the 3D spatial semantics of real-world environments is central to visual perception, scene reasoning, and 3D content generation. Real-world spaces are inherently three-dimensional, and robust 3D perception enables machines not only to comprehend spatial structures, object functions, and interactions, but also to generate, reconstruct, and edit plausible 3D scenes. Despite its importance, learning comprehensive 3D representations remains challenging due to limited 3D data, noisy or partial observations, and the high-dimensional and multimodal nature of the problem.
The primary objective of this project is to develop algorithmic approaches that advance 3D visual understanding and generation by learning robust, generalizable, and increasingly controllable representations of objects, scenes, and interactions. For increased generalizability, we exploit strong general priors learned from large-scale data in other modalities (e.g. text, 2D images, and video) and transfer them into 3D reasoning, particularly in settings where direct 3D supervision is scarce, such as interaction modeling and dynamic scene understanding. The research focuses on three complementary goals:
- Developing robust 3D semantic understanding under partial and noisy observations, enabling general reasoning about scene structure and relationships, while exploiting cross-modal priors to improve generalization.
- Introducing efficient, compact 3D-based representations and operators to encode and generate objects and scenes, leveraging intrinsic spatial properties to enable large-scale synthesis and reconstruction across diverse modalities and environments.
- Modeling functionality and interactions in 3D scenes, enabling spatio-temporal reasoning and instruction-driven manipulation of dynamic scenarios, with emphasis on transferring interaction knowledge from video and language data.
The expected impact potential of this work is substantial. By combining 3D understanding with generative modeling, the project will enable next-generation technologies in machine perception, immersive communications, mixed reality, architectural and industrial modeling, and robotics. It establishes a paradigm of spatially-grounded, 3D-consistent semantic understanding capable of both interpreting and generating complex environments, supporting machines that can perceive, reason about, and interact intelligently with the 3D world.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

During the reporting periods, the project has made substantial progress toward robust, scalable, and expressive 3D scene understanding, representation, and interaction modeling, resulting in numerous publications at top-tier computer vision and graphics venues.

We developed methods enabling reliable 3D perception under challenging conditions (e.g. noisy, partial, or limited data), including uncertainty-aware scene completion, unsupervised 3D instance segmentation, and approaches bridging 2D observations and 3D representations via probabilistic multi-modal pretraining for single-image 3D reconstruction. We further advanced structure-aware models for articulated object reconstruction and category-agnostic techniques for non-rigid shape correspondence, demonstrating strong generalization beyond class-specific settings.

We have also introduced efficient and expressive 3D representations, including dictionary-based neural fields and data-driven reconstruction approaches that significantly accelerate optimization while improving geometric fidelity. These advances were complemented by new generative methods for 3D content creation and editing, enabling interactive, sketch-based mesh generation as well as animation of 3D humanoid characters using motion priors from video diffusion models.

We also strengthened semantic understanding through hierarchical and expressive 3D captioning, enabling object- and part-level descriptions with increased granularity. Finally, the project advanced functional and interaction-aware reasoning, from zero-shot human-scene interaction synthesis to instruction-driven editing of complex 3D scenes by integrating hierarchical representations, language-based reasoning, and physical constraints.

Overall, the project has progressed from foundational methods toward integrated 3D intelligence systems that jointly address geometry, semantics, generation, and interaction in real-world settings.

Major achievements with support of this SpatialSem ERC project (all acknowledged):
- 15 CVPR/ICCV papers in 2024 and 2025
- 2 Siggraph/Siggraph Asia papers (2024) and 2 NeurIPS/ICLR papers (2024/2026)

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

The project has introduced several key advances that push the boundaries of 3D scene understanding, generation, and interaction modeling beyond the state of the art. GenZI (Li et al., CVPR’24) represents a conceptual breakthrough, enabling zero-shot synthesis of 3D human–scene interactions without requiring any 3D interaction data. By departing from conventional supervised pipelines, it demonstrates that functional interactions can be inferred through optimization-based reasoning guided by cross-modal priors, opening a fundamentally new direction for interaction modeling under severe data scarcity.

Building on this idea of leveraging generative and external priors for 3D reasoning, LT3SD (Meng et al., CVPR’25) advances large-scale 3D scene synthesis by introducing a latent tree representation that supports coherent generation of arbitrarily large scenes via diffusion-based outpainting, enabling for the first time the creation of effectively unbounded 3D environments while preserving global geometric consistency. In a complementary direction, DNF (Zhang et al., CVPR’25) bridges classical structured representations, such as SVD-based dictionary learning, with modern neural fields and diffusion models, providing an efficient representation for encoding and synthesizing complex 4D deformable objects.

Extending these advances toward structured interaction and controllable scene manipulation, ScanEdit (Boudjoghra et al., ICCV'25) introduces instruction-driven editing of real-world 3D scans by combining hierarchical scene representations with large language model reasoning and physically grounded constraints. This enables functional, language-guided modifications of complex scenes, unifying semantic understanding, structural decomposition, and generative editing within a single framework.

Together, these contributions demonstrate substantial methodological advances in 3D understanding, generation, and interaction, establishing new paradigms for spatial, functional, and dynamic reasoning in complex 3D environments, while increasingly exploiting optimization-based inference and cross-modal generative priors.