Final Report Summary - CODEC (Real-time decoding of conscious and non-conscious perceptual events in the human brain) The general aim of this project was to lay the groundwork for the use of non-invasive brain-computer interface (BCI) techniques in the study of movement initiation and for clinical use to detect signs of consciousness in non-communicative patients. In the process we sought to investigate the neural activity preceding the onset of voluntary movement and conscious sensory events. The research was divided into two parallel sub-projects: (1) study the neural antecedents of voluntary self-initiated movement, using combined magnetoencephalography (MEG) and electroencephalography (EEG) monitoring, with the goal of being able to detect their signatures in real time; and (2) develop the capability to reliably differentiate conscious from non-conscious EEG responses to auditory tone sequences, in both patients and healthy control subjects. Sub-project 2 was part of an ongoing initiative in collaboration with researchers at the Hôpital de la Salpetriére (Paris) and involved the supervision of a second-year Master student (Jean-Rémi King). Sub-project 1 was done in collaboration with prof. Robert Schapire (Princeton University), Dr Lauri Parkkonen (Elekta NeuroMag), and Dr Jacobo Sitt (INSERM, France). The end goal of sub-project 1 was highly ambitious and although we did not reach it, we completed most of the specific objectives, and we now have the necessary foundation to pursue this end goal - this work will continue in a subsequent project. In addition we also made an important discovery in the process that has led to a landmark publication (Schurger, Sitt, & Dehaene, PNAS, 2012), and also developed a technique for the derivation of spatial filters in the analysis of multi-sensor time-series data (Schurger et al, in review). The work products of sub-project 2 were completed and the goal was achieved in the aggregate over a set of data from each subject (King et al, in review), although performance on single trials is not yet sufficient for asynchronous real-time use (further research will be required). In addition to research work there were also several contributions under the rubric of 'knowledge transfer and organisation'. These include the founding and organising of a bi-weekly seminar on multivariate statistical learning techniques (which continues to the present), the organising and hosting of a full-day educational visit by a group of advanced high-school students, co-organising of the weekly seminar for the four teams in our research unit (INSERM U992), and the co-organising of a full-day interdisciplinary colloquium on 'causality in physics and neuroscience', with a focus on neural events in the brain as the cause of movement initiation. The lead researcher (Aaron Schurger) was embedded in the research team of Prof. Stanislas Dehaene (INSERM U992 / NeuroSpin / CEA-Saclay, FRANCE). Real-time decoding of human-brain activity requires significant high-quality methodological work, and this forms the main thrust of the project. However, unlike previous work in this area, these methodological advances were aimed at enabling future work in two specific areas of theoretical and empirical enquiry: (1) the voluntary self-initiation of movement, and its attendant subjective phenomena ('ownership' of the causes and consequences of the action), and (2) the onset of conscious sensory events. These phenomena are intimately linked through the notion of the conscious 'self' - that aspect of the psyche that is severely diminished or lost in comatose and minimally conscious states (MCS), and profoundly disturbed in schizophrenia and other psychoses. They are here linked methodologically through our proposed effort to detect, rather than discriminate, such events in real time, and theoretically through the presumed involvement of a threshold-crossing event in both phenomena. In both contexts, we planned to study the performance of the decoder across a range of latencies in order to gain insight into the build-up toward a conscious neural event (motor intention or sensory perception), within the framework of evidence-accumulation models of perception and decision-making. Because the methodological and empirical aspects of the proposed work are inter-dependent, both empirical and methodological advances were expected as products of the proposed work. Sub-project 1: Detecting the neural antecedents of self-initiated movement Summary: M/EEG data were recorded using the on-site Elekta NeuroMag equipment at the NeuroSpin research centre (CEA-Saclay, France). The researcher devised a novel experimental paradigm that can be used to isolate the neural activity patterns specifically associated with the onset of a self-initiated movement. In collaboration with Lauri Parkkonen (Elekta NeuroMag), Jacobo Sitt (INSERM), Bertand Monfort (CEA-Saclay), and Antoine Souloumiac (CEA-Saclay) we tackled the technical and computer programming issues related to real-time data processing using the Elekta NeuroMag system. This work contributed to the development and implementation by Elekta NeuroMag of a new 'real-time data server', which is now installed at NeuroSpin. The researcher also performed a detailed feasibility study for achieving very short latencies between BCI decisions and stimulus events. The researcher collaborated with Robert Schapire (Princeton University) and his graduate student Mehmet Basbug in the application of the AdaBoost (Schapire 2003) machine-learning algorithm to distinguishing movement-preceding from non-movement-preceding neural activity patterns in M/EEG data. As a final goal we planned to apply the asynchronous BCI in two experiments wherein external events are triggered in advance of the movement that would otherwise have triggered them. Although we did not reach this ambitious goal, we are now in an excellent position to carry this out in a subsequent project, thanks to the work that was carried out. Part of this work involved empirical research into the neural antecedents of self-initiated movement. This research resulted in a novel computational model and set of empirical results that together resulted in a landmark publication in PNAS (Schurger, Sitt et al. 2012). Finally, techniques that we developed in order to derive spatial filters that capture movement-preceding activity in multi-sensor (EEG and / or MEG) data turned out to be of more general use, and so this has been prepared and submitted as a methods paper, with an accompanying software toolbox (Schurger, Marti et al. in review). Each of these work products was also communicated in public lectures and / or conference posters. In addition, under the auspices of this project, the researcher founded and organised a bi-weekly seminar on the use of statistical learning techniques in cognitive neuroimaging research, co-organised a full-day colloquium on causality in physics and neuroscience, and hosted a group of advanced high-school students for a full day of learning and experimentation at the NeuroSpin research centre. Sub-project 2: detecting the occurrence of conscious sensory responses Summary: MEG and / or EEG data from healthy control subjects had been previously recorded as part of studies carried out by members of the Dehaene lab. EEG data from a large group (n = 158) of vegetative state (VS), MCS and conscious state (CS) patients had also been previously recorded by F. Faugeras of the Naccache research group at the Hôpital de la Salpétriere in Paris. Intra-cranial data were also available from a group of 9 patients who had been monitored for drug-resistant epilepsy (considered as control subjects from the standpoint of being fully conscious). All data sets data were recorded while subjects were presented with a series of brief 5-tone sequences (the 'local-global' paradigm (Beckenstein, Dehaene et al. 2009)). Deviant tones within a single 5-tone sequence ('local deviants') can provoke a neural response even in the absence of consciousness, whereas deviant sequences in the series of tone sequences ('global deviants') provoke a neural response that may be specific to conscious processing. The researcher supervised a first-year Master student, Jean-Rémi King, in the application of machine-learning techniques to these data. The student learned the fundamentals of machine learning and pattern classification under the researcher's guidance, and worked on applying those techniques to the two data sets provided. The student investigated the potential of multivariate pattern classifiers to extract subject-specific EEG patterns and predict single-trial local or global novelty responses. The approach was first validated on the EEG, MEG, and intracranial data from control subjects: responses to local deviants were found to be robust to distraction, whereas responses to global deviants required that the subject actively pay attention. The same techniques were then applied to the patient data. For the local response, the proportion of patients with significant decoding scores (M = 60 %) did not vary with the state of consciousness. By contrast, for the global response, only 14 % of VS patients present a significant effect, compared to 31 % of MCS and 52 % of CS patients (King et al, in review). Related documents Final Report - CODEC (Real-time decoding of conscious and non-conscious perceptual events in the human brain)
