Periodic Reporting for period 1 - PreVis (How perceptual history shapes vision: neural mechanisms and predictive properties)
Okres sprawozdawczy: 2023-10-01 do 2025-09-30
What we perceive at any given time is not only determined by the input reaching our sensory organs in the present, but also by what we perceived in the recent past, a few seconds ago. Past information (or “perceptual history”) plays a ubiquitous role in perception, systematically affecting how we perceive sensory stimuli in the present. For example, a vertical line could appear tilted if we saw a tilted line a few seconds before. Similarly, a sequence of events – like a flashing light – might appear as more or less numerous than it actually is, depending on the sequences that we saw before it. This phenomenon, also named “serial dependence,” effectively makes a present object to appear more similar to an object we saw before than it actually is. Recent research has found that perceptual history affects every domain of visual perception, starting from the most basic aspects of a scene, like orientation, colour, or shape, to more complex attributes like face identity or emotions. This suggests that this phenomenon reflects a fundamental brain processing mechanism supporting our perception. Many aspects of the functioning of this mechanism however remain unknown. One important question that remains unaswered is whether the influence of perceptual history purely reflects the information conveyed by past sensory stimuli, or whether it may work in a predictive fashion. That is, based on what the past information predicts, rather than past information per se. The brain indeed is not a passive receptor of information, but constantly predicts future inputs to speed up and optimise the processing of incoming information. Additionally, another interesting question is whether perceptual history effects work only within each sensory modality – that is, a past visual image affecting a present visual image, but not a sound – or whether this phenomenon may work across different modalities. Our perception is inherently multi-sensory, and indeed the brain possesses specific mechanisms to combine information from different sensory modalities. For example, we immediately know that an object fell to the floor based on the sound it makes, without the need to actually see the event to know what happened. In the PreVis project, we sought to address the predictive nature of perceptual history effects in the context on multi-sensory perception, and particularly in vision and audition. Considering the fundamental nature of the brain mechanisms involved with perceptual history, and the ubiquitous effect that they play in our perception, understanding them is something extremely important to understand how our brain generates perception. This phenomenon may have indeed important implications in several contexts that require accurate decisions, like a medical doctor classifying imaging results. We know that a past image can affect how we perceive an image in the present, but could a sound affect it as well? And why? Is it because the brain is trying to predict what comes next? Is the effect limited to information that is currently relevant for a task, or also irrelevant information from different senses could bias our judgments? With PreVis, we addressed these questions to gain new insights into how human perception works and the influences contributing to it. This is expected to have an important impact in the scientific sector, allowing us to refine our understanding of perception and devise new theories that can capture it more accurately.
In the core experimental work of PreVis, we used a combination of psychophysical (behavioural) and electrophysiological (EEG) techniques to measure perceptual history effects and uncover their predictive properties. Specifically, we devised psychophysical tasks in which volunteer participants were exposed to simple stimuli in different sensory modalities, like series of brief visual flashes or series of sounds, and were asked to judge their numerosity. To assess the impact of perceptual history on current perception, we measured the extent to which the perceived numerosity of a current sequence is affected by the sequences seen or heard previously during the task. To assess the predictive properties of perceptual history in this context, we first modulated the task performed by participants. Specifically, in two separate experimental conditions, we used (1) a task that required judging only visual stimuli (“uni-modal” visual task), or (2) a task that required comparing visual and auditory stimuli (“cross-modal” task). If perceptual history effects are driven by the past stimuli per se, then the effect should be identical irrespective of the task and the task-relevance of different sensory modalities. If the effect is instead predictive, it should work according to which information is currently more relevant. In particular, one key aspect of this design is that, at the moment of seeing or hearing a stimulus, the brain does not know what the next one will be (a visual or an auditory stimulus). Hence, we hypothesised that it may predict what the modality of the next stimulus might be according to the relevance of the sensory modalities driven by the task demands, and implement perceptual history accordingly. This experiment was then replicated adding the electroencephalography (EEG) technique, in order to measure when in the brain processing time-course perceptual history is combined with current sensory inputs to generate the effect. The EEG has indeed a very fine temporal resolution, and allows to track how the brain electrical responses evoked by a stimulus unfold over time with millisecond precision. Our hypothesis in this context concerned the timing of the influence of past stimuli on current brain processing. If perceptual history effectively influences how a stimulus appear in our conscious perception, then it should affect the early stages of stimulus processing. In a third experiment, we further addressed whether the combination of past and present information could be modulated by attention to individual components of a multisensory stimulus. That is, when presenting both auditory and visual sequences together, does the effect work according to both of them, or only according to the component that an individual is asked to attend? Participants in this case were instructed by a cue to attend and judge either the visual or auditory component of an audio-visual stimulus. Then, we assessed whether the judgement of the successive stimulus was affected by the attended component, or both.
Overall, the results of PreVis have provided several important contributions beyond the state of the art, allowing us to gain new insights into the mechanisms of perceptual history effects and thus deepening our understanding of how human perception works. First, our results demonstrate that perceptual history effects can work across different sensory modalities, and in particular vision and audition. This means that what we see in the present can be biased not only by images seen in the recent past, but also by previous sounds. This is important as it shows that the brain is very flexible when it comes to combining past and present information, and can also use information from different sensory modalities if deemed potentially relevant. Second, we have demonstated that the extent to which past and present information is integrated depends on the predicted relevance of the information. Namely, in a task in which visual and auditory stimuli are constantly alternated and need to be compared, the brain preferentially uses past information coming from a different modality rather than the same modality of the present stimulus. This could reflect an attempt to predict the next stimulus modality based on the most frequent dynamics of the stimuli and their relevance. Third, besides the overall task, we have demonstrated that also attention to individual components of a multi-sensory (audio-visual) stimulus can modulate the effect. Specifically, only the component of the previous stimulus that was explicitly attended is later combined with current sensory information. This reinforces the idea that the combination of past and present information is not a passive phenomenon, but an active process based on the relevance of sensory information. Finally, using electroencephalography, we have demonstrated that perceptual history affects the brain responses to a present stimulus very early after its onset, during the perceptual processing of the stimulus itself. This demonstrates that the influence of perceptual history effectively changes the appearance of an image or a sound in our conscious perception, and not merely how we judge the stimulus. Overall, these results paint a new picture of how perceptual history effects work, going beyond what was previously known about this phenomenon. In particular, the project results show that the brain systematically uses past information drawing from multiple sensory modalities, and it does so in an active fashion, guided by predictions about the relevance of the stimuli and attention directed to them.