Skip to main content

Decoding memory processing from experimental and spontaneous human brain activity using intracranial electrophysiological recordings and machine learning based methods.

Periodic Reporting for period 2 - DecoMP_ECoG (Decoding memory processing from experimental and spontaneous human brain activity using intracranial electrophysiological recordings and machine learning based methods.)

Reporting period: 2017-03-01 to 2018-02-28

My project's name is DecoMP-ECoG. It stands for Decoding Memory Processing using ElectroCorticoGraphy. The aim of this project is to understand how memories are formed in the human brain, based on electrical brain signals.

There are three main components in this work, based on the keywords in the title:

Our brain displays a specific pattern of brain activity when learning new information. There is considerable evidence that this pattern is then spontaneously replayed by the brain during rest and/or sleep. However, little is known about these spontaneous replays of brain activity. My project aims at identifying such replays and characterizing where and when they occur. To this end, I taught subjects a sequence of 12 images and had them rest afterwards (Figure 1).

In this project, decoding refers to the use of machine learning methods to model brain activity. Machine learning is a set of techniques that teach a computer the relationship between some measured values (called data points) and labels. An example of data point would be the brain activity of a subject when shown an image. The corresponding label could then be whether this person has 'learned' or 'forgotten' the corresponding image. In this work, I used such models to identify which features of brain activity predict when images are learned. I also aimed to pinpoint times when the brain spontaneously replays the learned images in rest or sleep.

To measure brain activity, I recorded the electrical activity on the surface of the brain (Figure 2) in three structures involved in image processing and memory, namely the occipital cortex, the temporal cortex and the hippocampus. This kind of recording is called 'ElectroCorticoGraphy' (a.k.a. 'ECoG') or intracranial electroencephalography (EEG).

Expected outcomes
This research can have impact both in terms of neuroscience and for developing new methods for analyzing brain signals.
• Neuroscience:
This project could provide insights on how our brain creates and stores a permanent record of information and accesses it as long as years later. In the long-term, a better understanding of human memory could guide investigations of memory dysfunctions involved in degenerative diseases such as age-related dementia. Preliminary results suggested that the encoding of information starts during short rest periods interspersed in the learning task. These results will however need to be confirmed in a larger and healthy population.
• Methods:
Machine learning modelling of anatomical and functional brain data is a promising research, computer-aided diagnosis and brain-computer interface tool. However, few works have been applied to electrical brain signals. Therefore, adapting these methods to such recordings is crucial to provide state-of-the-art research and clinical tools. In the current research, we have applied an algorithm that can automatically select the important features for the problem at hand. However, relating this feature selection to brain signals is not straightforward, so we simulated ECoG data to investigate this relationship.

I expect to make the results of my work fully accessible so that other groups can test the novel methods developed in this project. As part of this effort, I implemented the methods I developed in the open-source software PRoNTo. Version 2.1 of this software was recently released and version 3 (including ECoG modelling) will soon be available. The simulated data and corresponding code, as well as a preprocessing pipeline for ECoG data are also open source.

I have conducted this research in the USA for the first two years (Laboratory of Behavioral and Cognitive Neuroscience, Stanford University) under the supervision of Dr. Parvizi and then in the UK (Department of Computer Science, University College London) for one year under the supervision of Dr. Mourão-Miranda.
I have designed a memory task that patients with variable levels of cognition can perform. This experiment was recorded in 10 patients implanted with intracranial electrodes over the ventral temporal cortex (7) and in the hippocampus (3). A pre-processing pipeline has been developed and is accessible open-source on Github (used in Kucyi et al., 2018).

The main strategy to investigate memory processing in the human brain has been defined and implemented in PRoNTo (Schrouff et al., 2013, 2016, 2018, software in development).

In addition, another data set was used to investigate the spatial, temporal and frequency distribution of face perception in the human ventral temporal cortex (work submitted). Those results have been presented at international conferences.

The relationship between brain activity and how the machine learning model makes its decision was also investigated. While these experiments were not planned in the project proposal, their exploration was crucial to determine when we could confidently relate machine learning models to brain activity and when to avoid making such inference (Schrouff and Mourao-Miranda, 2018 and presented at a conference).
Preliminary results on three subjects yielded very interesting results, including an increasing complexity in the information encoded by a visual localizer, our encoding task and our retrieval task. It also seems that memory traces appear in rest periods during the encoding session, which has been shown in mice but not in humans. If confirmed, these results could provide new grounds for exploring human memory.

I have also adapted my open-source software PRoNTo to electrophysiological data and provided a multiple kernel learning (MKL) algorithm to perform machine learning modeling of such recordings. These results were published in Journal of Neuroscience Methods and used in other publications from Dr. Parvizi’s team (Daitch et al., 2016, PNAS). When the code is released (in a couple of months, only pending documentation), we expect more teams to use our tools and advance neuroscience research in various directions.

In addition, the use of machine learning tools to understand brain functioning is still an open question, mostly when investigating the relationship between a model performance and parameters and brain activity. In this project, we simulated data to investigate when to relate the model parameters with brain function, and when this relationship should be questioned (Schrouff and Mourao-Miranda, 2018 and oral presentation at PRNI workshop). Furthermore, clinical data sets often include confounding variables (e.g. system the data was recorded on) that can be difficult to disentangle from the variable of interest. We hence implemented a Multi-Task Learning approach that treats each categorical confound as a subgroup, with the different subgroup being related in terms of the variable of interest. This could have a large impact on how clinical datasets are modelled (publication in preparation).
Memory task the subjects need to perform
Intracranial EEG recordings