Periodic Reporting for period 4 - CRACK (Cracking the neural code of human object vision)
Okres sprawozdawczy: 2023-11-01 do 2025-01-31
With every blink of our eyes, our brain effortlessly distills out of the stream of photons hitting the retina a conscious percept of the world. This percept consists of enclosed and meaningful entities – objects – and determines how we interact with the world. Over the course of about one hundred years, psychology and neuroscience have identified the core part of the brain mediating object vision. Understanding human object recognition is important for several reasons. It can give an answer to why do we perceive the way we do. Understanding the mechanisms also bears promise in clinical application, e.g. restitution of vision after brain damage. It is also of strong interest to industry and society as the biological brain remains the most exciting inspiration for building artificial vision systems. Yet, the mechanisms that mediate the human ability to recognize objects remain incompletely understood.
The overall goal of CRACK was to provide answers to three fundamental, long-standing and open questions about visual object recognition that build on each other: 1) How does each of the core cortical regions active during vision represent objects, 2) how do those regions communicate information, and 3) how does the observed dynamic neural activity mediate adaptive behavior? The first objective was to unravel the unique importance and role of each core cortical region of the visual brain involved in object recognition. We have controlled, isolated, and further described the nature of representations in visual cortex. The second objective was to clarify how those core cortical regions communicate with each other. Using layer-specific fMRI and time-frequency resolved EEG we have described feedforward and recurrent communication streams. The third objective was to link the observed mechanisms to behavior. We related neural activity to behavior using multiple methods, highlighting the aspects of visual representations suitably formatted to support behavior. Fulfilling these objectives through an orchestrated and interdisciplinary effort CRACK provides key empirical pieces of evidence for an updated theory of visual object recognition in the human cortex.
The overall goal of CRACK was to provide answers to three fundamental, long-standing and open questions about visual object recognition that build on each other: 1) How does each of the core cortical regions active during vision represent objects, 2) how do those regions communicate information, and 3) how does the observed dynamic neural activity mediate adaptive behavior? The first objective was to unravel the unique importance and role of each core cortical region of the visual brain involved in object recognition. We have controlled, isolated, and further described the nature of representations in visual cortex. The second objective was to clarify how those core cortical regions communicate with each other. Using layer-specific fMRI and time-frequency resolved EEG we have described feedforward and recurrent communication streams. The third objective was to link the observed mechanisms to behavior. We related neural activity to behavior using multiple methods, highlighting the aspects of visual representations suitably formatted to support behavior. Fulfilling these objectives through an orchestrated and interdisciplinary effort CRACK provides key empirical pieces of evidence for an updated theory of visual object recognition in the human cortex.
We have now addressed all three objectives. Concerning objective 1, we have a) developed and applied a computational approach to probe the function of human brain regions, b) conducted a series of experiments detailing how object representations are influenced by the typical, structured scene environments in which they appear, b) shown how fundamental object properties category and location are represented and d) investigated the representation of animacy. Concerning objective 2, we have disentangled information flow in feedforward and feedback direction a) in the spatial domain across cortical layers, and b) in the temporal domain in oscillatory channels. Concerning objective 3, we have identified visual representations suitably formatted to guide behaviour using several techniques.
The results of this project were and further will be disseminated through peer-reviewed articles, onference presentations, tutorials, in invited talks, and through the development of an online-prediction challenge called the Algonauts Project. The results have further significantly contributed to the development of several new research directions that are being continued by previous team members.
The results of this project were and further will be disseminated through peer-reviewed articles, onference presentations, tutorials, in invited talks, and through the development of an online-prediction challenge called the Algonauts Project. The results have further significantly contributed to the development of several new research directions that are being continued by previous team members.
Our work goes beyond the state-of-the-art in several aspects. For example, the computational approach elucidating the functions of visual brain regions from a new algorithmic perspective that previous approaches did not do to comparable depth. Second, our work on object category and location representations invites revision of the standard theory of visual processing. Third, a challenge we conducted proposes a new way of how to do science in a way that ensures efficiency and openness in cognitive neuroscience. Fourth, our work on how visual regions communicate demonstrate the methodology to resolve feedforward and feedback information flow in both space and in time and provide novel insight into the mechanisms of communication in both domains. Fifth, we provided a large neuroimaging data set for video that we analyzed using novel methods, highlighting in the temporal dimension of visual experience neglected typically in object recognition research.