Periodic Reporting for period 1 - TIME (It's about time: Towards a dynamic account of natural vision.)
Période du rapport: 2022-07-01 au 2024-12-31
The visual world around us is a source of rich semantic information that guides our higher-level cognitive processes and actions. To tap into this resource, the brain’s visual system engages in complex, intertwined computations to actively sample, extract, and integrate information across space and time. Surprisingly however, the integrative, dynamic, and active nature of vision hardly plays a role in the way we approach it in experimentation and computational modelling. Hence, despite significant advances in our general understanding of the neural mechanisms underlying vision, the brain’s natural modus operandi remains poorly understood. A new approach is needed.
TIME uses more natural viewing paradigms in both experimentation and modelling to close this gap. First, we simultaneously record high-resolution brain activity and eye-movements of participants while they explore and summarise natural visual scenes. The resulting large-scale resource will enable us to better understand the intricate mechanisms at play. In particular, it will enable us to study when and how the brain decides to move on from a given fixation location to the next, and how information from one fixation may affect the processing of the next, i.e. how information is integrated across fixations. In parallel to these analyses, we develop computational models that mirror the overall process. Here, we make use of recent developments in artificial intelligence to derive image-computable, large-scale models. A famous quote by Richard Feynman states that “What I cannot create, I do not understand”. This statement highlights the importance of computational models in science. Observing a system in experimentation and theorising about its inner mechanisms requires models to be built that put our hypotheses to the test. Therefore, in addition to building an AI-based model system, a final work package will test the alignment between brain data and the computational model.
In summary, using an interdisciplinary approach, TIME will establish when, where, and how visual semantic understanding emerges in the brain as it actively samples and integrates information from the world in a continuously updating and dynamic decision process. This approach marks a paradigm shift in how we study the visual system, and promises implications not only for neuroscience, but also for computer vision and artificial intelligence.
TIME uses more natural viewing paradigms in both experimentation and modelling to close this gap. First, we simultaneously record high-resolution brain activity and eye-movements of participants while they explore and summarise natural visual scenes. The resulting large-scale resource will enable us to better understand the intricate mechanisms at play. In particular, it will enable us to study when and how the brain decides to move on from a given fixation location to the next, and how information from one fixation may affect the processing of the next, i.e. how information is integrated across fixations. In parallel to these analyses, we develop computational models that mirror the overall process. Here, we make use of recent developments in artificial intelligence to derive image-computable, large-scale models. A famous quote by Richard Feynman states that “What I cannot create, I do not understand”. This statement highlights the importance of computational models in science. Observing a system in experimentation and theorising about its inner mechanisms requires models to be built that put our hypotheses to the test. Therefore, in addition to building an AI-based model system, a final work package will test the alignment between brain data and the computational model.
In summary, using an interdisciplinary approach, TIME will establish when, where, and how visual semantic understanding emerges in the brain as it actively samples and integrates information from the world in a continuously updating and dynamic decision process. This approach marks a paradigm shift in how we study the visual system, and promises implications not only for neuroscience, but also for computer vision and artificial intelligence.
While the ERC project TIME is still ongoing, we have already arrived at multiple milestones, reported below. As of August 2024, these are:
(1) The collection of a one-of-its-kind large-scale dataset, in which we combine magnetoencephalography with high-resolution eyetracking while participants explore natural scenes has been completed. Moreover, we finished preprocessing of all data by all participants. An alpha release of the data is ongoing.
(2) Given the new dataset, we derived a new multivariate analysis technique based on which we could show that fixation durations are largely explained by ongoing memory encoding.
(3) We have developed, implemented, and trained the first iteration of generative symbiotic networks, which combine two subnetworks, akin to the dorsal and ventral streams of the human visual system. The resulting system selects by itself where to look and how to process and integrate the information present.
(4) We have further extended artificial neural networks from AI to act as models of visual information processing in the brain. In particular, we developed network layers that are topographic in nature, contrasting the gold standard in the field. We demonstrated that networks equipped with this new network layer are more energy efficient and capture human behaviour more accurately.
(1) The collection of a one-of-its-kind large-scale dataset, in which we combine magnetoencephalography with high-resolution eyetracking while participants explore natural scenes has been completed. Moreover, we finished preprocessing of all data by all participants. An alpha release of the data is ongoing.
(2) Given the new dataset, we derived a new multivariate analysis technique based on which we could show that fixation durations are largely explained by ongoing memory encoding.
(3) We have developed, implemented, and trained the first iteration of generative symbiotic networks, which combine two subnetworks, akin to the dorsal and ventral streams of the human visual system. The resulting system selects by itself where to look and how to process and integrate the information present.
(4) We have further extended artificial neural networks from AI to act as models of visual information processing in the brain. In particular, we developed network layers that are topographic in nature, contrasting the gold standard in the field. We demonstrated that networks equipped with this new network layer are more energy efficient and capture human behaviour more accurately.
Despite the short duration of the grant, we have already been able to push the envelope of cognitive computational neuroscience with various findings. For example, we have opened a new avenue of research by linking fixation durations to memory processes. As another example, we have successfully implemented and trained an artificial agent that selects where to collect information (in terms of eye-movements) and how to integrate the novel information into the existing understanding of its surrounding (via recurrent connectivity profiles). This is of interest to both, cognitive neuroscientists interested in studying how the brain selects and processes visual information during active vision, as well as the computer vision community. Third, the view that visual processing is linearly aligned with representations in large language models from AI is providing an entirely novel perspective on the goals of visual information processing in the brain.