Spatial 3D Semantic Understanding for Perception in the Wild

Información del proyecto

SpatialSem

Identificador del acuerdo de subvención: 101076253

DOI

10.3030/101076253

Fecha de la firma de la CE 1 Junio 2023

Fecha de inicio 1 Octubre 2023

Fecha de finalización 30 Septiembre 2028

Financiado con arreglo a

• European Research Council (ERC)

Coste total

€ 1 500 000,00

Aportación de la UE

€ 1 500 000,00

Coordinado por

TECHNISCHE UNIVERSITAET MUENCHEN
Germany

Periodic Reporting for period 1 - SpatialSem (Spatial 3D Semantic Understanding for Perception in the Wild)

Período documentado: 2023-10-01 hasta 2026-03-31

Resumen del contexto y de los objetivos generales del proyecto

Understanding the 3D spatial semantics of real-world environments is central to visual perception, scene reasoning, and 3D content generation. Real-world spaces are inherently three-dimensional, and robust 3D perception enables machines not only to comprehend spatial structures, object functions, and interactions, but also to generate, reconstruct, and edit plausible 3D scenes. Despite its importance, learning comprehensive 3D representations remains challenging due to limited 3D data, noisy or partial observations, and the high-dimensional and multimodal nature of the problem.
The primary objective of this project is to develop algorithmic approaches that advance 3D visual understanding and generation by learning robust, generalizable, and increasingly controllable representations of objects, scenes, and interactions. For increased generalizability, we exploit strong general priors learned from large-scale data in other modalities (e.g. text, 2D images, and video) and transfer them into 3D reasoning, particularly in settings where direct 3D supervision is scarce, such as interaction modeling and dynamic scene understanding. The research focuses on three complementary goals:
- Developing robust 3D semantic understanding under partial and noisy observations, enabling general reasoning about scene structure and relationships, while exploiting cross-modal priors to improve generalization.
- Introducing efficient, compact 3D-based representations and operators to encode and generate objects and scenes, leveraging intrinsic spatial properties to enable large-scale synthesis and reconstruction across diverse modalities and environments.
- Modeling functionality and interactions in 3D scenes, enabling spatio-temporal reasoning and instruction-driven manipulation of dynamic scenarios, with emphasis on transferring interaction knowledge from video and language data.
The expected impact potential of this work is substantial. By combining 3D understanding with generative modeling, the project will enable next-generation technologies in machine perception, immersive communications, mixed reality, architectural and industrial modeling, and robotics. It establishes a paradigm of spatially-grounded, 3D-consistent semantic understanding capable of both interpreting and generating complex environments, supporting machines that can perceive, reason about, and interact intelligently with the 3D world.

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

During the reporting periods, the project has made substantial progress toward robust, scalable, and expressive 3D scene understanding, representation, and interaction modeling, resulting in numerous publications at top-tier computer vision and graphics venues.

We developed methods enabling reliable 3D perception under challenging conditions (e.g. noisy, partial, or limited data), including uncertainty-aware scene completion, unsupervised 3D instance segmentation, and approaches bridging 2D observations and 3D representations via probabilistic multi-modal pretraining for single-image 3D reconstruction. We further advanced structure-aware models for articulated object reconstruction and category-agnostic techniques for non-rigid shape correspondence, demonstrating strong generalization beyond class-specific settings.

We have also introduced efficient and expressive 3D representations, including dictionary-based neural fields and data-driven reconstruction approaches that significantly accelerate optimization while improving geometric fidelity. These advances were complemented by new generative methods for 3D content creation and editing, enabling interactive, sketch-based mesh generation as well as animation of 3D humanoid characters using motion priors from video diffusion models.

We also strengthened semantic understanding through hierarchical and expressive 3D captioning, enabling object- and part-level descriptions with increased granularity. Finally, the project advanced functional and interaction-aware reasoning, from zero-shot human-scene interaction synthesis to instruction-driven editing of complex 3D scenes by integrating hierarchical representations, language-based reasoning, and physical constraints.

Overall, the project has progressed from foundational methods toward integrated 3D intelligence systems that jointly address geometry, semantics, generation, and interaction in real-world settings.

Major achievements with support of this SpatialSem ERC project (all acknowledged):
- 15 CVPR/ICCV papers in 2024 and 2025
- 2 Siggraph/Siggraph Asia papers (2024) and 2 NeurIPS/ICLR papers (2024/2026)

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

The project has introduced several key advances that push the boundaries of 3D scene understanding, generation, and interaction modeling beyond the state of the art. GenZI (Li et al., CVPR’24) represents a conceptual breakthrough, enabling zero-shot synthesis of 3D human–scene interactions without requiring any 3D interaction data. By departing from conventional supervised pipelines, it demonstrates that functional interactions can be inferred through optimization-based reasoning guided by cross-modal priors, opening a fundamentally new direction for interaction modeling under severe data scarcity.

Building on this idea of leveraging generative and external priors for 3D reasoning, LT3SD (Meng et al., CVPR’25) advances large-scale 3D scene synthesis by introducing a latent tree representation that supports coherent generation of arbitrarily large scenes via diffusion-based outpainting, enabling for the first time the creation of effectively unbounded 3D environments while preserving global geometric consistency. In a complementary direction, DNF (Zhang et al., CVPR’25) bridges classical structured representations, such as SVD-based dictionary learning, with modern neural fields and diffusion models, providing an efficient representation for encoding and synthesizing complex 4D deformable objects.

Extending these advances toward structured interaction and controllable scene manipulation, ScanEdit (Boudjoghra et al., ICCV'25) introduces instruction-driven editing of real-world 3D scans by combining hierarchical scene representations with large language model reasoning and physically grounded constraints. This enables functional, language-guided modifications of complex scenes, unifying semantic understanding, structural decomposition, and generative editing within a single framework.

Together, these contributions demonstrate substantial methodological advances in 3D understanding, generation, and interaction, establishing new paradigms for spatial, functional, and dynamic reasoning in complex 3D environments, while increasingly exploiting optimization-based inference and cross-modal generative priors.