Skip to main content
Go to the home page of the European Commission (opens in new window)
English English
CORDIS - EU research results
CORDIS

Learning Generative 3D Scene Models for Training and Validating Intelligent Systems

Project description

Training computers to see

Computer vision is an area of artificial intelligence (AI). The goal of computer vision is to equip machines with a visual understanding of their environment, ultimately enabling computers to identify objects in images and videos just like humans do. Much of the recent progress in computer vision builds on machine learning techniques that learn visual representations from large human annotated datasets. However, labeling data for training deep models is expensive and existing photo-realistic simulators do not provide the required variety and fidelity. The EU-funded project LEGO-3D will tackle this problem by developing probabilistic models capable of synthesizing 3D scenes jointly with photo-realistic 2D projections from arbitrary viewpoints and with full control over the scene elements. It will devise algorithms for automatic decomposition of real and synthetic scenes into latent 3D representations capturing geometry, material, light and motion.

Objective

Recently, the field of computer vision has witnessed a major transformation away from expert designed shallow models towards more generic deep representation learning. However, collecting labeled data for training deep models is costly and existing simulators with artist-designed scenes do not provide the required variety and fidelity. Project LEGO-3D will tackle this problem by developing probabilistic models capable of synthesizing 3D scenes jointly with photo-realistic 2D projections from arbitrary viewpoints and with full control over the scene elements. Our key insight is that data augmentation, while hard in 2D, becomes considerably easier in 3D as physical properties such as viewpoint invariances and occlusion relationships are captured by construction. Thus, our goal is to learn the entire 3D-to-2D simulation pipeline. In particular, we will focus on the following problems:

(A) We will devise algorithms for automatic decomposition of real and synthetic scenes into latent 3D primitive representations capturing geometry, material, light and motion.

(B) We will develop novel probabilistic generative models which are able to synthesize large-scale 3D environments based on the primitives extracted in project (A). In particular, we will develop unconditional, conditioned and spatio-temporal scene generation networks.

(C) We will combine differentiable and neural rendering techniques with deep learning based image synthesis, yielding high-fidelity 2D renderings of the 3D representations generated in project (B) while capturing ambiguities and uncertainties.

Project LEGO-3D will significantly impact a large number of application areas. Examples include vision systems which require access to large amounts of annotated data, safety-critical applications such as autonomous cars that rely on efficient ways for training and validation, as well as the entertainment industry which seeks to automate the creation and manipulation of 3D content.

Fields of science (EuroSciVoc)

CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.

You need to log in or register to use this function

Keywords

Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)

Programme(s)

Multi-annual funding programmes that define the EU’s priorities for research and innovation.

Topic(s)

Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.

Funding Scheme

Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.

ERC-STG - Starting Grant

See all projects funded under this funding scheme

Call for proposal

Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.

(opens in new window) ERC-2019-STG

See all projects funded under this call

Host institution

EBERHARD KARLS UNIVERSITAET TUEBINGEN
Net EU contribution

Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.

€ 1 467 500,00
Address
GESCHWISTER-SCHOLL-PLATZ
72074 Tuebingen
Germany

See on map

Region
Baden-Württemberg Tübingen Tübingen, Landkreis
Activity type
Higher or Secondary Education Establishments
Links
Total cost

The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.

€ 1 467 500,00

Beneficiaries (1)

My booklet 0 0