From spatial relationships to temporal correlations: New vistas on predictive coding

Project Information

SPATEMP

Grant agreement ID: 850861

DOI

10.3030/850861

Project closed

EC signature date 28 January 2020

Start date 1 February 2020

End date 31 January 2025

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 750 000,00

EU contribution

€ 1 750 000,00

Coordinated by

ERNST STRUNGMANN INSTITUTE GGMBH
Germany

Periodic Reporting for period 4 - SPATEMP (From spatial relationships to temporal correlations: New vistas on predictive coding)

Reporting period: 2024-08-01 to 2025-01-31

Summary of the context and overall objectives of the project

The idea that the brain generates predictions about the external world has been a leading neuroscience theory in the past two decades (Rao & Ballard, 1999, Friston, 2010). Predictive processing depends on the extensive recurrent connectivity within and between areas, which is a hallmark of the cerebral cortex. Recurrent interactions also give rise to neuronal synchronization and cortical rhythms, which are known to further modulate the functional and structural connectivity among neural populations. This raises the question: What role do brain rhythms and synchronization play in predictive processing? One of the most influential proposals of how predictive processing is implemented is hierarchical predictive-coding theory which posits that perceptual inference is implemented by the feedforward routing of surprising or unpredicted signals (i.e. prediction errors), and the distribution of sensory predictions down the hierarchy via feedback projections. One major theory that emerged from electrophysiological investigations is that FF (error) and FB (prediction) signaling use distinct frequency channels, namely gamma and alpha/beta rhythms, respectively. Yet an alternative theory that emerged in recent years (Vinck & Bosman, 2016) is that gamma rhythms emerge when sensory information is predictable by the spatiotemporal context, and play a role in efficient encoding of information. We test this hypothesis in humans and non-human primates by using deep neural networks to quantify predictability in natural scenes, and investigating the relation of predictability to neural signals in visual cortical areas. We further examine how this relation changes with learning, what the contribution of gamma rhythms is to perception and neural transmission, and how gamma rhythms emerge mechanistically from recurrent interactions within and between areas. This research sheds light on the way in which the cortex efficiently encodes information and how neural populations flexibly communicate with each other, which may have important implications for the design of energy-efficient neural networks. Furthermore, rhythmic activity is a fundamental property of processes in the body and the brain. Cortical rhythms are distorted in many brain diseases, e.g. Parkinson, Epilepsy, Alzheimer’s disease. Furthermore, brain stimulation at gamma frequencies might counter neural degeneration. Thus, the current findings can inform mechanisms and treatment of these diseases.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Our research aimed to understand how the brain forms predictions about the world. A key idea was that brain rhythms known as gamma oscillations (fast, regular electrical activity) help the brain process visual information more efficiently—especially when what we see matches what we expect. To test this, we used advanced artificial intelligence models that learned to predict patterns in natural images. We found that gamma activity in the brain’s visual area (V1) increased when images were more predictable, especially at the level of basic visual features like edges or textures. This principle—predictability—explained gamma activity better than any other image feature. When natural images did not match expectations, we observed broadband bursts of activity and increased firing of neurons, rather than gamma. This distinction led us to propose a new theory called feature-specific predictive processing, where different brain signals reflect different levels of prediction and error. Our work also challenges existing assumptions. Standard computer models of vision (convolutional neural networks, or CNNs) failed to explain gamma activity because they overlook the brain’s sensitivity to shapes and spatial relationships. Surprisingly, even randomly initialized CNNs predicted brain responses just as well as trained ones—suggesting that common models may be missing crucial principles of brain function. We found that gamma emerges when the brain is in a stable, low-dimensional state, representing predictable input. In contrast, surprise or complexity leads to more irregular activity. Gamma also occurs at the edge of the brain’s representational space—where visual stimuli are most distinct and easily remembered—offering a potential explanation for our finding that gamma correlates with memorability. We discovered that gamma influences specific types of brain cells in long-term communication (inhibitory interneurons), and developed a new theoretical framework called CTCOM (Coherence Through Communication). This explains how coherence across brain areas can emerge naturally from communication itself, helping resolve long-standing puzzles in neuroscience. Finally, we explored how gamma rhythms relate to learning by showing how predictive processing depends on plasticity mechanisms and interactions between excitatory and inhibitory neurons. Altogether, this work presents a new framework for how brain rhythms contribute to predictive processing—the idea that the brain continuously generates expectations and updates them when reality doesn’t match. Our findings provide new insight into how the brain processes sensory information, supports learning and memory, and what might go wrong in neurological disorders where these rhythms are disrupted.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project gave rise to several paradigm changes: 1) Our results indicate that gamma does not mediate the signaling of prediction errors, as assumed by many previous theories, but rather emerges when stimuli are predictable. 2) Our results indicate that gamma and firing rates encode distinct information, and do not carry redundant information. In particular our findings have suggested an update to standard predictive coding theories by suggesting predictive coding is highly feature specific, and that predictability of low- and high-level features is signaled by gamma and firing rates, respectively. 3) Our findings suggest major limitations of convolutional neural networks, standard models of neural activity, in explaining activity in the early visual cortex. 4) Our findings suggest that gamma is not a mechanism to increase communication to other areas, as previously assumed, but may rather lead to increased recruitment of inhibition in a downstream area. 5) Our findings suggest that that gamma coherence between areas may naturally arise as a consequence of communication, rather than being a mechanism for it, as assumed by many previous theories. 6) Our findings suggest a new theory, namely that gamma may allow for the stable, efficient encoding of information in natural images, but only when those natural images are low-dimensional. 7) Our findings suggest new perceptual consequences of gamma, explaining why images yielding gamma may be distinct from other images and are more memorable, in contrast to previous theories that linked gamma to perceptual binding or attention.

Gamma increases with predictability, but decreases with image salience