3D scene understanding in two glances

Projektinformationen

SEGMENT

ID Finanzhilfevereinbarung: 101086774

DOI

10.3030/101086774

EK-Unterschriftsdatum 23 März 2023

Startdatum 1 November 2023

Enddatum 31 Oktober 2028

Finanziert unter

• European Research Council (ERC)

Gesamtkosten

€ 2 083 091,25

EU-Beitrag

€ 2 083 091,25

Koordiniert durch

TECHNISCHE UNIVERSITAT DARMSTADT
Germany

Periodic Reporting for period 1 - SEGMENT (3D scene understanding in two glances)

Berichtszeitraum: 2023-11-01 bis 2026-04-30

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

If the sun appears from behind a cloud, some physical properties (light, shadow) of the scene drastically change, but our understanding of its structure does not. The first step of visual scene understanding is segmentation, in which our brain tries to infer which parts of the scene belong to which objects. Adults can do this in photographs – but photographs are not how we learned to see. We learned to see by moving around in a 3D world. The way that scenes project into our eyes, how light is affected by the optics of our eyes, how our photoreceptors sample the light, and how we move our eyes all provide rich information about our environment. However, we do not know how adults combine all this information to perceive segmented scenes, nor how important these different information sources are for learning. The goals of SEGMENT are to understand how adults use the rich information present in active 3D vision to perform segmentation, and to understand how this might be learned. We are developing a new display device and experimental methods to study how adults segment scenes when realistic visual information is available. In addition, we are developing new technologies using advanced computer graphics and machine learning to simulate the inputs to the visual system from early development to adulthood. We are conducting in silico experiments in artificial neural networks to understand segmentation learning, by systematically restricting or manipulating different factors.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

To study segmentation behaviour in human adults, we have developed new visual stimuli and perceptual decision-making tasks. These stimuli and tasks have allowed us to quantify how much different sources of visual information (e.g. colour and motion) contribute to human segmentation decisions. In tandem, we are developing a visual display system capable of presenting light to both eyes at real physical depths while participants’ gaze position is tracked, allowing us to measure optical and eye movement contributions to segmentation decisions and information gathering behaviour. In the next phase of the project, we will combine our newly developed tasks with our display system to include more and varied sources of information for segmentation learning, coming closer to approximating the retinal inputs of real environments under controlled laboratory conditions.

To support our experiments with artificial neural networks, we have developed a rendering pipeline that allows us to generate photorealistic yet abstract three-dimensional scenes, in which different sources of visual information can be systematically manipulated. In the next phase of our project we will examine the representations learned in artificial neural networks trained with different sources of information available, to better understand the potential importance of these information sources for visual representation learning.

Finally, we have developed an interdisciplinary method to better understand and explain the representations learned in artificial neural networks, using recent advances in computer graphics (physically based rendering). This new method allows us to find sets of 3D scenes that artificial neural networks interpret in the same way, which can be used to better understand what information networks are sensitive to and what they are insensitive to.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

Our method for finding physical scenes that artificial networks “see” as the same will be useful for understanding visual inference models in general. With further development, this method can be used with a human or animal “in the loop” to understand perceptual or neural representations in physical terms. Our display system development will be documented publicly and the software to run the system will be made available, allowing others to build from our development work.