Uncovering the core dimensions of visual object representations

Project Information

COREDIM

Grant agreement ID: 101039712

DOI

10.3030/101039712

EC signature date 7 June 2022

Start date 1 July 2022

End date 30 June 2027

Funded under

• European Research Council (ERC)

Total cost

€ 1 500 000,00

EU contribution

€ 1 500 000,00

Coordinated by

JUSTUS-LIEBIG-UNIVERSITAET GIESSEN
Germany

Periodic Reporting for period 1 - COREDIM (Uncovering the core dimensions of visual object representations)

Reporting period: 2022-07-01 to 2024-12-31

Summary of the context and overall objectives of the project

Immediately when we open our eyes, we are able to see a rich visual world. To make sense of our surroundings and interact with them in a meaningful way, it is widely believed that our brain identifies important properties - or dimensions - of the objects around us. These could be visual-perceptual dimensions (e.g. the color or the shape of an object), but also conceptual - or semantic - dimensions (e.g. is this object related to animals, is it related to fire or heat). Two key challenges have limited our understanding of these core dimensions that underlie our visual representations. First, the visual world is very complex, and it is not possible to capture this complexity using traditional small-scale experiments. This has made it hard to "connect the dots" and gain a more complete understanding of the dimensions that shape our cognitive and neural representations. Second, our visual perception and our semantic knowledge about the things around us are deeply intertwined (e.g. animals tend to be curvy, manmade things tend to have straight lines). Thus, it has been challenging to discriminate the contributions of pure visual input from the meaning we attach to objects.

COREDIM addresses these challenges with two primary objectives. First, the project aims to uncover the core dimensions underlying visual object representations in the so-called ventral visual system, a set of brain regions critical for processing visual information. Beyond identifying these core dimensions, the second objective is to untangle the roles of visual information and semantic knowledge in shaping how we understand objects.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

To address the first objective of identifying the core dimensions underlying our neural representation of objects, we have collected a large-scale representatively sampled dataset of brain responses to natural images, tailored to capture the breadth of our visual experiences. Specifically, we have first developed a sampling strategy that has allowed for broad coverage of the space of natural stimuli, distilling around 30.000 natural images to be representative of a set of 120 million natural photographs. Using a smaller dataset, we have already successfully identified the brain responses related to core object dimensions derived from behavior. Building on this work, we have used advanced brain imaging technique (7 Tesla functional and structural MRI, 3 Tesla Connectom MRI) to collect data from over 40 neuroimaging sessions per participant, providing the deepest sampling not only of retinotopic responses, anatomical and structural connectivity, but also functional responses to thousands of natural images. Our next step is to apply computational modeling to this data to reveal core dimensions underlying human neural representation of objects. Together, this promises a much deeper understanding of how our brain allows us to interpret the visual world.

The second objective of COREDIM is to distinguish the role of visual perception and semantic knowledge in our object representations. To address this, we have used innovative experimental designs using speeded similarity judgments. In a first step, we have compared different similarity tasks to each other, identifying the unique and shared aspects of mental representations measured by these tasks. In the next step, we will use speeded similarity judgments in a visual search task. This will identify core dimensions underlying fast representations which also serve as the basis for visual search without a predefined target. We will further use unrecognizable images to distinguish the impact of visual dimensions from the semantic knowledge we have about objects. In addition, this part of the project involves comparing human visual processing with that of non-human primates who lack the same level of semantic knowledge, where we have identified both commonalities and differences in their visual processing strategies. Together, this work promises a much better understanding of how strongly our object processing relies on vision alone as compared to our object knowledge.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

By tackling these fundamental challenges, COREDIM promises to provide a much deeper understanding of how our brains interpret the visual world. A key impact of this work has already emerged: Rather than viewing the high-level visual system as being either dominated by category or continuous dimensions, our work reveals that it is likely a combination of both, where category is expressed through multiple overlapping continuous tuning functions. Further, our results reveal the importance not only of high-level visual cortex but of all cortical processing stages in the representation of behaviorally-relevant information. By clarifying the relationship between visual and semantic information, this research has the potential to drive significant advances not only in visual and cognitive neuroscience but also in artificial intelligence, inspiring new approaches to machine vision. The large datasets and innovative methods generated by this project have been made or will continue to be made publicly available, providing highly valuable resources for future research in both human and machine vision. By unlocking new insights into how we see, how we interpret our surroundings, and how we interact with our visual world, COREDIM opens up exciting possibilities for advances in psychology, neuroscience, and machine intelligence.