Periodic Reporting for period 4 - NATVIS (Characterizing neural mechanisms underlying the efficiency of naturalistic human vision)
Reporting period: 2022-03-01 to 2023-08-31
The efficient detection of goal-relevant objects in our environment is of critical importance in daily life. For example, the majority of road accidents are caused by insufficient attention to relevant objects (e.g. pedestrians) in scenes. Our daily-life visual environments, such as city streets and living rooms, contain a multitude of objects. Out of this overwhelming amount of sensory information, our brains must efficiently select and recognize those objects that are relevant for current goals. Visual and attention systems have developed and evolved to optimally perform real-world tasks like these, as reflected in the remarkable efficiency of naturalistic object detection. It is increasingly appreciated that the brain makes use of a wide range of available information to facilitate object detection in real scenes. This project aims to characterize the neural mechanisms that contribute to the efficiency of goal-directed naturalistic human vision.
The brain systems underlying the detection of task-relevant information in cluttered displays have primarily been studied using artificial and highly simplified displays. While these studies have been fundamentally important for revealing basic neural mechanisms involved in perception and attention, they fall short in fully explaining how the brain so rapidly detects familiar objects in complex but meaningful real-world scenes. A large body of behavioural work has shown that the detection of goal-relevant objects in real-world scenes is dramatically more efficient than the detection of targets in apparently much simpler artificial displays. The goal of this project is to understand why this is so, using psychophysics, fMRI, MEG, and TMS to improve our understanding of the neural mechanisms underlying the efficiency of object detection in natural scenes.
The brain systems underlying the detection of task-relevant information in cluttered displays have primarily been studied using artificial and highly simplified displays. While these studies have been fundamentally important for revealing basic neural mechanisms involved in perception and attention, they fall short in fully explaining how the brain so rapidly detects familiar objects in complex but meaningful real-world scenes. A large body of behavioural work has shown that the detection of goal-relevant objects in real-world scenes is dramatically more efficient than the detection of targets in apparently much simpler artificial displays. The goal of this project is to understand why this is so, using psychophysics, fMRI, MEG, and TMS to improve our understanding of the neural mechanisms underlying the efficiency of object detection in natural scenes.
The project investigated the neural mechanisms underlying the efficiency of naturalistic vision. By investigating four factors contributing to the efficiency of naturalistic vision, we have achieved several main results:
Attention in real-world scenes
Although attentional selection has been extensively studied in laboratory settings, where objects are presented in isolation, it remains an open question how attention mechanisms work in more complex scenes, where objects produce different images on the retina depending on where they are located in the scene. For example, two objects that produce an image of the same size on the retina can be of vastly different sizes in the real world if one is nearby and the other one is farther away. In a series of studies, we have investigated the interaction between top-down attentional set and contextual modulation of object appearance. These projects provided evidence for two mechanisms that allow for effective template-based attentional selection under naturalistic viewing conditions; one in which the representation of the visual input is updated according to scene context, and one in which the top-down attentional set is adjusted based on scene context. These and related results have been published in a series of peer-reviewed articles (e.g. Gayet & Peelen, 2019; Battistoni et al., 2020; Gayet & Peelen, 2022; Lerebourg et al., 2023; Gayet et al., in press)
Interactions between scene and object processing
In naturalistic vision, objects and scenes are processed interactively, with scene information informing object recognition and object information informing scene recognition (e.g. seeing a lake supports the inference that the distant object is a boat, and seeing a boat supports the inference that the foggy scene is a lake). In behavioral, fMRI, MEG, and TMS studies we have revealed interactions between scene and object processing during the viewing of static and dynamic natural scenes. For example, using fMRI and MEG, we tested how object perception supports scene recognition. Our results showed that the representation of scene layout was facilitated by contextual object cues. Together with other studies investigating the interaction between object and scene processing, we have gained a better understanding of the extensive interactions between object- and scene-selective pathways in visual cortex. These and related results have been published in a series of primary research articles (e.g. Brandman & Peelen, 2019; Wischnewski et al., 2021; Brandman & Peelen, 2023; Aldegheri et al., 2023) and have been summarized in a review article (Peelen et al., in press)
Inter-object grouping based on real-world regularities
We quickly recognize objects even when these objects are embedded in highly cluttered scenes. In a series of studies, we have shown that this is partially accounted for by perceptual adaptations to regularities in real-world environments, including the perceptual grouping of highly predictable constellations of objects (e.g. lamp above table). In one project, we used EEG to investigate how identity-based and positional regularities interact during visual object processing. We found that the degree to which the visual system integrates identity information carried by concurrent objects depended on their relative position. These results provided evidence that the spatial and identity relations between objects influence object processing in an interactive manner, with these regularities jointly facilitating object perceptibility itself. That presenting objects in their typical relative positions enhanced the representation of their contextual association under these conditions sheds new light on visual adaptations for simplifying scene analysis. These and related results have been published in a series of peer-reviewed articles (e.g. Quek & Peelen, 2020; Thorat et al., 2022; Yan et al., 2023)
The functional role and representational content of preparatory states
When searching for an object in our environment, we maintain a visual template of the object in memory, causing our visual system to favor template-matching visual input. In a series of studies, we have investigated the functional role and representational content of these preparatory templates. For example, in one fMRI study, we investigated the content of the template when participants oriented towards larger “anchor” objects (e.g. a table) to find small target objects (e.g. a pen) in natural scenes. Our results showed that preparatory activity in visual cortex represented the anchor object that guided spatial attention. These results indicate that preparatory activity supports naturalistic search by reflecting a self-generated biasing signal for attentional guidance, independent of external cues and target report. These findings have been presented at conferences and will soon be submitted for publication.
Attention in real-world scenes
Although attentional selection has been extensively studied in laboratory settings, where objects are presented in isolation, it remains an open question how attention mechanisms work in more complex scenes, where objects produce different images on the retina depending on where they are located in the scene. For example, two objects that produce an image of the same size on the retina can be of vastly different sizes in the real world if one is nearby and the other one is farther away. In a series of studies, we have investigated the interaction between top-down attentional set and contextual modulation of object appearance. These projects provided evidence for two mechanisms that allow for effective template-based attentional selection under naturalistic viewing conditions; one in which the representation of the visual input is updated according to scene context, and one in which the top-down attentional set is adjusted based on scene context. These and related results have been published in a series of peer-reviewed articles (e.g. Gayet & Peelen, 2019; Battistoni et al., 2020; Gayet & Peelen, 2022; Lerebourg et al., 2023; Gayet et al., in press)
Interactions between scene and object processing
In naturalistic vision, objects and scenes are processed interactively, with scene information informing object recognition and object information informing scene recognition (e.g. seeing a lake supports the inference that the distant object is a boat, and seeing a boat supports the inference that the foggy scene is a lake). In behavioral, fMRI, MEG, and TMS studies we have revealed interactions between scene and object processing during the viewing of static and dynamic natural scenes. For example, using fMRI and MEG, we tested how object perception supports scene recognition. Our results showed that the representation of scene layout was facilitated by contextual object cues. Together with other studies investigating the interaction between object and scene processing, we have gained a better understanding of the extensive interactions between object- and scene-selective pathways in visual cortex. These and related results have been published in a series of primary research articles (e.g. Brandman & Peelen, 2019; Wischnewski et al., 2021; Brandman & Peelen, 2023; Aldegheri et al., 2023) and have been summarized in a review article (Peelen et al., in press)
Inter-object grouping based on real-world regularities
We quickly recognize objects even when these objects are embedded in highly cluttered scenes. In a series of studies, we have shown that this is partially accounted for by perceptual adaptations to regularities in real-world environments, including the perceptual grouping of highly predictable constellations of objects (e.g. lamp above table). In one project, we used EEG to investigate how identity-based and positional regularities interact during visual object processing. We found that the degree to which the visual system integrates identity information carried by concurrent objects depended on their relative position. These results provided evidence that the spatial and identity relations between objects influence object processing in an interactive manner, with these regularities jointly facilitating object perceptibility itself. That presenting objects in their typical relative positions enhanced the representation of their contextual association under these conditions sheds new light on visual adaptations for simplifying scene analysis. These and related results have been published in a series of peer-reviewed articles (e.g. Quek & Peelen, 2020; Thorat et al., 2022; Yan et al., 2023)
The functional role and representational content of preparatory states
When searching for an object in our environment, we maintain a visual template of the object in memory, causing our visual system to favor template-matching visual input. In a series of studies, we have investigated the functional role and representational content of these preparatory templates. For example, in one fMRI study, we investigated the content of the template when participants oriented towards larger “anchor” objects (e.g. a table) to find small target objects (e.g. a pen) in natural scenes. Our results showed that preparatory activity in visual cortex represented the anchor object that guided spatial attention. These results indicate that preparatory activity supports naturalistic search by reflecting a self-generated biasing signal for attentional guidance, independent of external cues and target report. These findings have been presented at conferences and will soon be submitted for publication.
The studies conducted as part of this ERC project have revealed new insights into the neural mechanisms underlying the efficiency of naturalistic vision. Going beyond the state of the art, we have demonstrated how attention mechanisms in visual cortex support visual search in scenes, how scene- and object-selective neural pathways interact to facilitate perception, and how neural adaptations to natural scene structure enable the visual brain to optimally represent the large number of objects contained in real-world environments, thereby simplifying the neural code for scene analysis