Using deep neural networks to understand functional specialization in the human visual cortex

Project Information

DEEPFUNC

Grant agreement ID: 101117441

DOI

10.3030/101117441

EC signature date 18 December 2023

Start date 1 January 2024

End date 31 December 2028

Funded under

• European Research Council (ERC)

Total cost

€ 1 494 750,00

EU contribution

€ 1 494 750,00

Coordinated by

JUSTUS-LIEBIG-UNIVERSITAET GIESSEN
Germany

Periodic Reporting for period 1 - DEEPFUNC (Using deep neural networks to understand functional specialization in the human visual cortex)

Reporting period: 2024-01-01 to 2026-06-30

Summary of the context and overall objectives of the project

Understanding why the human visual cortex is organized into functionally specialized regions is one of the central open questions in cognitive neuroscience. While decades of research have documented selective responses for faces, bodies, and scenes, we still lack a principled account of why certain visual categories recruit specialized neural circuitry while others do not. Existing approaches have largely relied on contrast-based neuroimaging and descriptive accounts of selectivity, leaving unresolved whether functional specialization reflects computational constraints, developmental input statistics, or intrinsic neural biases. DEEPFUNC addresses this gap by introducing a unified computational–neuroscientific framework to explain the origins of functional specialization. Focusing on the human ventral visual pathway, the project combines large-scale natural visual input, hypothesis- and data-driven analyses, deep neural networks, and behavioral and neural data. This integrative approach allows us to move beyond descriptive mapping toward mechanistic explanation. The project is structured around three core objectives. First (WP1 – What), we will precisely characterize which properties of visual categories give rise to functional specialization. Second (WP2 – How), we will determine how natural visual input statistics and experience shape the development of specialization. Third (WP3 – Why), we will identify the computational and feature-level principles that determine where and why specialized representations emerge in human visual cortex. By systematically linking deep neural networks with human neuroimaging and behavior, DEEPFUNC establishes a testable computational account of functional specialization. This framework enables us to evaluate competing hypotheses, generate falsifiable predictions, and uncover principles that generalize beyond vision. The expected impact of DEEPFUNC is threefold. Scientifically, it advances the field from descriptive characterization to mechanistic explanation of functional specialization. Methodologically, it provides scalable tools for analyzing and comparing biological and artificial neural systems. Conceptually, it informs the design of intelligent machines by identifying when and why specialized processing architectures are advantageous. More broadly, clarifying how and why the brain organizes information into specialized systems contributes to a deeper understanding of the architecture of the human mind and brain.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the current reporting period, substantial scientific progress has been achieved across all three work packages. In WP1, we combined large-scale brain imaging data with computational modeling to investigate how the human visual system organizes information. Rather than viewing category-selective brain regions as single-purpose modules, we showed that these regions are composed of multiple interpretable representational dimensions. Building on this insight, we developed a new quantitative method to measure category selectivity across hundreds of object categories in both human visual cortex and artificial neural networks. This large-scale analysis confirmed well-known selectivities (such as for faces and scenes) and revealed additional selective patterns across object space. Importantly, our results indicate that category selectivity is better described as a graded phenomenon rather than a simple all-or-none property. We are currently investigating which visual features and computational principles give rise to these selectivity gradients. In parallel, we are synthesizing artificial visual categories that systematically vary in their degree of specialization in order to experimentally test how graded selectivity influences learning and neural processing. In WP2, we examine how visual experience shapes specialization across development. We began analyzing large-scale egocentric video datasets capturing visual input during childhood and adulthood. We developed tools to quantify how often different visual categories appear and where they occur in the visual field. In addition, we created a method to identify images that most clearly reveal differences between computational models trained on different types of visual experience. Ongoing work uses these tools to test how developmental input shapes visual representations. In WP3, we investigate why specialized processing emerges where it does in the brain. We systematically examined how functional specialization can be reliably measured in artificial neural networks and how different measurement approaches relate to causal interventions. We are now extending this work to track how specialized representations emerge during training and to test how modifying visual features and input statistics influences cortical organization. Together, these activities are advancing the project from descriptive mapping toward understanding why and how functional specialization emerges in the human brain.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The results of DEEPFUNC advance the field of cognitive neuroscience in several ways. First, the large-scale analysis of category selectivity across hundreds of object categories, combined with data-driven voxel decomposition, challenges the traditional view of category-selective brain regions as discrete modules. By demonstrating that specialization is multidimensional and graded rather than binary, DEEPFUNC advances our understanding of functional organization in the human visual cortex. Second, the direct comparison between human brain representations and artificial neural networks revealed systematic divergences in how selectivity emerges. Identifying this dissociation provides concrete guidance for improving computational architectures and evaluation strategies in artificial intelligence, thereby strengthening the neuroscience–AI feedback loop. Third, DEEPFUNC delivers publicly available methodological infrastructure that supports reproducible research at the interface of neuroscience and AI. The open-source tools developed within the project standardize dataset management, causal analysis, and representational comparison, thereby lowering barriers for interdisciplinary research and strengthening methodological rigor across the field. In the longer term, the project has broader impact potential. Understanding how and why functional specialization emerges can inform the design of artificial systems that balance specialized and general-purpose processing. The framework is also relevant for developmental and clinical research, where altered specialization patterns are implicated in neurodevelopmental and perceptual disorders. Overall, DEEPFUNC contributes a computational and testable account of functional specialization, new large-scale analytical tools, and open infrastructure that advance both neuroscience and artificial intelligence research.