The spatio-temporal representational architecture of memory

Project Information

STREAM

Grant agreement ID: 715714

DOI

10.3030/715714

Project closed

EC signature date 24 February 2017

Start date 1 March 2017

End date 31 August 2023

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 493 400,00

EU contribution

€ 1 493 400,00

Coordinated by

UNIVERSITY OF GLASGOW
United Kingdom

Periodic Reporting for period 5 - STREAM (The spatio-temporal representational architecture of memory)

Reporting period: 2022-02-01 to 2023-08-31

Summary of the context and overall objectives of the project

Understanding human memory is one of the big challenges in neuroscience. People tend to believe that a good memory is one that preserves a maximally precise record or “snapshot” of the past. STREAM opposes this “snapshot” view, and is instead motivated by the premise that memory is a highly adaptive and reconstructive process. The major scientific goal of this project is to understand how the human brain reconstructs memories of past events, with a focus on what features of a memory are reconstructed when in time and where in the brain. Are memories systematically different from the original sensory experience they code, and how do they change over time and when we repeatedly recall the same event? The project addresses these questions using novel behavioural tasks together with cutting-edge brain imaging and machine learning (“decoding”) techniques. These methods allow us to decompose memories into their constituent elements (e.g. sensory details or the semantic gist of an event), and to track how the reconstruction of these elements unfolds in time when participants are cued to remember complex episodes. Methods include fMRI, MEG, simultaneous EEG-fMRI and intracranial EEG recordings directly from the human hippocampus. The experimental work in STREAM rests on a framework of reverse reconstruction during memory recall, with the central working hypothesis that the human brain tends to reconstruct memories in a backwards fashion, such that the information flow is reversed compared to the original sensory experience.

Over the duration of the project, we have found highly consistent evidence in support of such a reverse reconstruction hierarchy, including in brain activity patterns and in behaviour using feature-specific reaction times. While sensory information is primarily processed along a detailed-perceptual to abstract-conceptual gradient, we find that the elements contained in a memory are reconstructed along a reversed gradient that prioritizes abstract-conceptual over detailed-perceptual information. Also consistent with STREAM's original predictions, we show that memories become semanticised over time and with repeated recall, suggesting an important role for active memory reconstruction in shaping our long-term memories. Moreover, STREAM revealed that the neural reactivation of a memory during retrieval is clocked by brain oscillations in the theta rhythm, visible in neural activity and even in behaviour. Overall, STREAM succeeded in its original goal to decompose memories into constituent elements to address the question "what's in a memory, when in time?", allowing fundamental insights into the timeline of memory recall, and the features of visual memories that are preferentially reconstructed. Understanding how memories are reconstructed in the healthy human brain, step by step, constitutes a major leap in our knowledge and has important implications for clinical and other applied work, for example for understanding the over-generalization of memories in post-traumatic stress disorders, or common memory errors in eyewitness testimony.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

At the outset of STREAM, we developed a novel experimental paradigm to decompose visual memories into their fundamental features, to then track the re-emergence of the relevant memory components in neural time and space. Over the runtime of STREAM, we found robust empirical evidence for STREAM's hypothesis that memory reconstruction follows a reverse gradient compared to perception. A perception-memory flip in the representational feature hierarchy was found using EEG (Linde-Domingo et al., 2019), feature-specific reaction times (Linde-Domingo et al., 2019; Lifanov et al., 2021; Postzich et al., in progress), simultaneous EEG-fMRI recordings (Lifanov et al., 2022, bioRxiv), and parallel MEG and fMRI recordings (Postzich et al., in progress). The EEG and behavioural work revealed when in time, after presenting a memory cue, different features of a visual memory are reconstructed. This work consistently showed that high-level semantic features are reactivated significantly earlier than low-level perceptual features during recall, in reverse order of their initial perceptual processing. The simultaneous EEG-fMRI work additionally allowed us to map this timeline onto the brain areas processing different types of features during perception and recall.

Consistent with the original proposal, we also find that this memory reactivation is clocked by neural oscillations in the theta frequency range, and that these rhythmic fluctuations can be observed in the hippocampus (ter Wal et al., 2021; Kerren et al., 2018, 2022), and even in button press behaviours (Ter Wal et al., 2021). This includes work in epileptic patients using intracranial EEG (iEEG) to record signals directly from the human hippocampus (ter Wal et al., 2021), speaking directly as to the sources of these memory signatures. The methodological developments related to this objective also led to the publication of the “Brain Time Toolbox” (van Bree et al., 2022), a tool allowing researchers to link brain oscillations to the content that is decodable from EEG/MEG/iEEG recordings. We summarised our first milestone findings from STREAM in a highly-cited review paper about the timeline (“neural chronometry”) of memory recall (Staresina & Wimber, 2019). The data from several other experiments are still in the analysis or write-up stage, with first results having been presented at international conferences.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The results obtained in STREAM have moved the field a major step beyond what was previously known about the spatiotemporal architecture of memory reconstruction. The most important advance in knowledge comes from our findings which, for the first time, decompose memories into their constituent elements, and pinpoint exactly when in time distinct elements of a memory are reactivated in the brain and become consciously accessible. With very little time-resolved information available at the time, the project has provided fundamental new insights into the timeline of memory recall, showing a primarily reverse reconstruction gradient for memories and their clocking by theta oscillations. The project work also applied novel state-of-the-art decoding and other multivariate pattern analysis techniques to imaging and electrophysiological data, including a fusion of imaging modalities, the use of novel feature-specific reaction times, and the development of a new toolbox, moving the memory field from a methodological viewpoint. Finally, the project revealed major new insights into how the quality of memories changes over time and how such changes can be quantified in the lab.

publicationsummary-figure.png