In this project we developed a novel brain-computer interface (BCI) that allows human subjects to write letters of the alphabet simply by imagining how the letters look. The system, thus, works purely based on mental imagery, i.e. without speaking or hand-writing. This has been achieved using advanced, non-invasive, brain scanning and newly developed analysis methods.
Using ultra-high field functional magnetic resonance imaging (fMRI) at 7 Tesla, the developed software product reads-out and visualises letter shapes that are in the mind's eye of participants. New developed analysis methods allow to reconstruct letter shapes in real-time from ongoing activity patterns in early cortical visual areas. This allows to show participants the mental images in their mind’s eye during brain scanning. The real-time display of mental images is important because it provides feedback to the participant whether the displayed letter is indeed the one the participant imagined. The real-time reconstruction was only made possible by the development of novel analysis methods (Bhat et al. 2019) and the employment of neural learning techniques (auto-encoder networks) as part of the analysis. The neural network substantially increased the robustness of letter reconstruction by “denoising” the brain activity patterns obtained during letter imagery (Senden et al., 2019; Van Hoof et al. 2019).
The project also investigated whether it would be possible by participants to improve the quality of the letter reconstruction in case of errors. In the developed fMRI paradigm, participants receive a first visualisation of an imagined letter after a few seconds. Based on this visual feedback, participants then re-imagine the letter shape in case that it does not match the letter they intended to convey. This two-stage process allows to correct errors, e.g. in case that a part of a letter is missing from the presented reconstruction. This paradigm establishes a closed-loop system, which is an important achievement for building a brain-computer interface for patients.
The project achieved its goal to evaluate the feasibility of reading imagined letter shapes from the mind's eye, including dissemination of new neuroscientific knowledge and analysis methods. A major overarching motivation of the project is to develop a novel communication tool for the benefit of severely motor-impaired patients, especially patients suffering from the locked-in syndrome (LIS). These patients require means other than motor behaviour to be able to communicate. The implementation of brain-computer interfaces is based on the presence and volitional modulation of brain signals. While there are non-invasive electro-encephalography (EEG) and functional near-infrared spectroscopy (fNIRS) brain-computer interfaces available for locked-in patients, these BCIs often suffer from reliability issues in patient applications (EEG) or low expressive power (fNIRS) usually allowing only yes/no answers. For the potential benefit of patients, this project provides an innovative new possibility that could enable full communication of words and sentences based on imagined letter shape. The conducted work with healthy participants has, however, also revealed high inter-subject variability, i.e. the achievable quality of letter shape reconstructions differs substantially between participants. This finding requires further investigation to understand the sources of the observed variability across subjects before the system can be offered to patients. While further fMRI experiments are necessary to assess feasibility in patients, the developed Mind's Eye BCI software has been integrated in a commercial software product so that it is available for tests with locked-in patients in the future.
Publications
Bhat S, Lührs M, Goebel R & Senden M (2019). Real-Time pRF Mapping using Gradient Descent on Hashed-Gaussian Tiles. Conference of the Society for Neuroscience. Chicago, USA.
Senden, M. Emmerling TC, van Hoof R, Frost MA, Goebel R (2019). Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. Brain Structure and Function, 224, 1246-1262.
Van Hoof. R., Bhat, S., Lührs, M., Senden, M. & Goebel, R. (2019). Real-time reconstruction of letter shapes in the Mind’s Eye. Real-time Functional Imaging and Neurofeedback Conference. Aachen-Maastricht, Germany / The Netherlands.
