Final Report Summary - THEPREDICTIVEBRAIN (Dynamic Predictions: It's all Rhythms)
Imagine you drive down a busy street, and you notice a traffic light that has just turned yellow. Deciding whether you should step on the break or speed up crucially depends on your prediction whether the traffic light will turn red or green, and when it will do that. If you speed up and the traffic light turns red before you have crossed it, you risk a fine or even an accident. Hitting the break if the traffic light is going to be green might cause a rear-end collision with the driver behind you who made a better prediction about the traffic light. Making a good prediction is thus a crucial skill in everyday life, and in the worst case, even crucial for survival. Predicting allows us to react faster, to anticipate potential danger, and to solve problems efficiently. There is growing scientific consensus that a fundamental principle of brain function is its ability to predict future events based on prior information and to constantly match expectations against signals from the. Despite this general consensus about critical role of predictions in perception and action, much less is known, and no agreement exists regarding how this is achieved, i.e. which is the neuronal mechanism of predictions. The broad goal of this proposal was to investigate the neuronal mechanism serving the human brain’s predictive capacities. More specifically, it was also investigated whether predictions about different stimulus attribute, e.g. time (“when”) or content (“what”), rely on the same neural mechanism. We have performed several studies to investigate predictions using invasive intracranial recordings, including laminar resolution recordings, magnetoencephalography, and computational modelling. This brought us closer to understanding where, when, and how predictions affect neural processing.
Our studies demonstrate that predictions optimize stimulus processing, such that for instance subjects react faster when they can predict which stimulus will occur, or when that stimulus will occur. In addition, we can show that predictions, in particular content-based predictions, increase perceptual sensitivity, i.e. subjects can see more when they know what stimulus to expect. This effect occurs for artificial stimuli, e.g. letters that are degraded due to the presence of noise, as well as for more naturalistic stimuli such as faces embedded in complex backgrounds. Together, our studies have shown that predictions exert a sizable effect on perception, either by increasing perceptual sensitivity (Aru,2015; Mayer, 2015), or by speeding up information accrual thereby optimizing stimulus processing.
We have also investigated at what stage of stimulus processing predictions take effect. In the literature, different accounts have been put forward: one account posits that predictions do not alter stimulus processing per se, but decision making, i.e. it is easier for subjects to decide among perceptual choices. Here, predictions are seen as affecting decisional or post-perceptual stages, and thus the effect of predictions is predicted to occur late during stimulus processing, in decision making areas. In contrast, the perceptual account posits that predictions affect neural processing at early, perceptual stages, thereby changing the quality of the internal representations. We conducted two magnetoencephalographic studies to determine when in time predictions affect neural processing. Both of our studies, employing different stimuli (faces and letters, respectively), and different populations of subjects reached a similar conclusion: when subjects can predict which stimulus was going to be presented, neural responses for predicted versus unpredicted stimuli started to differ early in time, about 100 ms after stimulus onset, suggesting that predictions alter perceptual and not decision making stages. Specifically, we found that perceptual benefits correlated with brain activity in an early time window (80-95 ms), and that this effect localized to occipital and posterior parietal brain regions. These results led us to conclude that predictions likely affect perceptual stages. Similarly, in a letter categorization task, we observed that the amplitude of early sensory responses was increased in the presence of predictions, suggesting a role in the selective amplification of predicted information. In the same study, we also investigated whether slow prestimulus alpha oscillations in task-relevant brain regions serve as carriers of sensory predictions, and whether they interact with stimulus processing to influence early categorization processes thereby boosting perception. We observed increased prestimulus alpha oscillations in a multisensory network representing grapheme/phoneme associations in the presence of sensory predictions. This prestimulus activity had a marked influence on stimulus-evoked activity, suggesting a role in the selective amplification of predicted information. Together, these results suggest that predictions exert early, sensory effects that boost perception, and that low-frequency alpha oscillations might serve as a mechanism to carry and test sensory predictions.
To investigate whether predictions concerning different stimulus attributes, in our case content and time, engage the same neural mechanism, we used experimental paradigms in which we manipulated content and time predictions while acquiring electrophysiological signal directly from the human brain in patients with medically refractory epilepsy. We combined these recordings with biophysically realistic computational modelling to better understand the underlying mechanisms. Our modelling results indicate that “what” and “when” predictability rely on complementary neural processes: “what” predictions appear to increase short-term plasticity in auditory areas, while “when” predictability appear to increase synaptic gain in motor areas. Thus, content and temporal predictions seem to involve complementary neural mechanisms in different cortical regions, suggesting domain-specific prediction signaling along the cortical hierarchy.
Finally, we have extended the test of the predictive brain hypothesis to more naturalistic scenarios, i.e. “visual active sensing” in tasks involving eye movements, and language. While collecting eye movement data and recording neural activity directly from the brains of patients with medically refractory epilepsy we demonstrated that during blinks and eye movements, high-level visual areas exhibited lower activity during blinks and eye movements, conditions in which the motor signal generates internal predictions. In contrast, low-level visual areas did not exhibited differences between conditions. In addition, we have worked on extensions to the language domain in which predictions at different temporal timescales co-occur. To that end, we have developed a paradigm in which different aspects of the language representation, such as words, syllables, syntax and semantics, which encompass different timescales, can be tracked in parallel while selectively addressing each of those components.
Altogether our studies have demonstrated that predictions are central for cognition and motor control, improving perception and leading to faster and more efficient information processing. In naturalistic contexts, predictions derived from the motor system might explain why we perceive the world as stable despite the constant blurring or interruptions of sensory information caused by eye movements and blinks. Predictions appear to exert their effect at early, perceptual stages of processing. Importantly, as not all predictions are equal, the mechanisms and cortical areas implementing predictions of different sensory attributes appear to differ. The observed degree of functional specialization might confer more efficient encoding of predictions regarding different, and possibly uncorrelated, aspects of the environment (e.g. “what”, “where”, and “when”) than to encode every possible combination of features of the incoming stimuli.
Our results lay the foundations for a better understanding for one of the brain’s fundamental computational capacities. Furthermore, in recent years, several proposals have been put forward that ascribe the symptomatology of neurodevelopmental diseases such as autism to deficits in the ability to predict the environment. By elucidating the neural mechanisms of predictions, our results may thus also ultimately contribute to a better understanding of these diseases, and thus inform the medical community as well as affected members of society.
Our studies demonstrate that predictions optimize stimulus processing, such that for instance subjects react faster when they can predict which stimulus will occur, or when that stimulus will occur. In addition, we can show that predictions, in particular content-based predictions, increase perceptual sensitivity, i.e. subjects can see more when they know what stimulus to expect. This effect occurs for artificial stimuli, e.g. letters that are degraded due to the presence of noise, as well as for more naturalistic stimuli such as faces embedded in complex backgrounds. Together, our studies have shown that predictions exert a sizable effect on perception, either by increasing perceptual sensitivity (Aru,2015; Mayer, 2015), or by speeding up information accrual thereby optimizing stimulus processing.
We have also investigated at what stage of stimulus processing predictions take effect. In the literature, different accounts have been put forward: one account posits that predictions do not alter stimulus processing per se, but decision making, i.e. it is easier for subjects to decide among perceptual choices. Here, predictions are seen as affecting decisional or post-perceptual stages, and thus the effect of predictions is predicted to occur late during stimulus processing, in decision making areas. In contrast, the perceptual account posits that predictions affect neural processing at early, perceptual stages, thereby changing the quality of the internal representations. We conducted two magnetoencephalographic studies to determine when in time predictions affect neural processing. Both of our studies, employing different stimuli (faces and letters, respectively), and different populations of subjects reached a similar conclusion: when subjects can predict which stimulus was going to be presented, neural responses for predicted versus unpredicted stimuli started to differ early in time, about 100 ms after stimulus onset, suggesting that predictions alter perceptual and not decision making stages. Specifically, we found that perceptual benefits correlated with brain activity in an early time window (80-95 ms), and that this effect localized to occipital and posterior parietal brain regions. These results led us to conclude that predictions likely affect perceptual stages. Similarly, in a letter categorization task, we observed that the amplitude of early sensory responses was increased in the presence of predictions, suggesting a role in the selective amplification of predicted information. In the same study, we also investigated whether slow prestimulus alpha oscillations in task-relevant brain regions serve as carriers of sensory predictions, and whether they interact with stimulus processing to influence early categorization processes thereby boosting perception. We observed increased prestimulus alpha oscillations in a multisensory network representing grapheme/phoneme associations in the presence of sensory predictions. This prestimulus activity had a marked influence on stimulus-evoked activity, suggesting a role in the selective amplification of predicted information. Together, these results suggest that predictions exert early, sensory effects that boost perception, and that low-frequency alpha oscillations might serve as a mechanism to carry and test sensory predictions.
To investigate whether predictions concerning different stimulus attributes, in our case content and time, engage the same neural mechanism, we used experimental paradigms in which we manipulated content and time predictions while acquiring electrophysiological signal directly from the human brain in patients with medically refractory epilepsy. We combined these recordings with biophysically realistic computational modelling to better understand the underlying mechanisms. Our modelling results indicate that “what” and “when” predictability rely on complementary neural processes: “what” predictions appear to increase short-term plasticity in auditory areas, while “when” predictability appear to increase synaptic gain in motor areas. Thus, content and temporal predictions seem to involve complementary neural mechanisms in different cortical regions, suggesting domain-specific prediction signaling along the cortical hierarchy.
Finally, we have extended the test of the predictive brain hypothesis to more naturalistic scenarios, i.e. “visual active sensing” in tasks involving eye movements, and language. While collecting eye movement data and recording neural activity directly from the brains of patients with medically refractory epilepsy we demonstrated that during blinks and eye movements, high-level visual areas exhibited lower activity during blinks and eye movements, conditions in which the motor signal generates internal predictions. In contrast, low-level visual areas did not exhibited differences between conditions. In addition, we have worked on extensions to the language domain in which predictions at different temporal timescales co-occur. To that end, we have developed a paradigm in which different aspects of the language representation, such as words, syllables, syntax and semantics, which encompass different timescales, can be tracked in parallel while selectively addressing each of those components.
Altogether our studies have demonstrated that predictions are central for cognition and motor control, improving perception and leading to faster and more efficient information processing. In naturalistic contexts, predictions derived from the motor system might explain why we perceive the world as stable despite the constant blurring or interruptions of sensory information caused by eye movements and blinks. Predictions appear to exert their effect at early, perceptual stages of processing. Importantly, as not all predictions are equal, the mechanisms and cortical areas implementing predictions of different sensory attributes appear to differ. The observed degree of functional specialization might confer more efficient encoding of predictions regarding different, and possibly uncorrelated, aspects of the environment (e.g. “what”, “where”, and “when”) than to encode every possible combination of features of the incoming stimuli.
Our results lay the foundations for a better understanding for one of the brain’s fundamental computational capacities. Furthermore, in recent years, several proposals have been put forward that ascribe the symptomatology of neurodevelopmental diseases such as autism to deficits in the ability to predict the environment. By elucidating the neural mechanisms of predictions, our results may thus also ultimately contribute to a better understanding of these diseases, and thus inform the medical community as well as affected members of society.