Periodic Reporting for period 4 - BrainCom (High-density cortical implants for cognitive neuroscience and rehabilitation of speech using brain-computer interfaces.)
Reporting period: 2020-12-01 to 2022-04-30
Over 5 million people suffer from aphasia worldwide, most often following a brain stroke, but also from neurodegenerative disorders affecting the motor production and articulation of speech, locked-in syndrome, or coma. While motor rehabilitation training can help people recover some of their speech ability in case of partial aphasia, new approaches remain to be explored to restore communication and, ultimately, speech in severe aphasic patients.
Very few data are available at the cellular and multicellular level, which is the required resolution to obtain sufficient decoding precision to predict continuous speech. One of the reasons is the lack of an available technology capable of recording neural signals with a high spatial and temporal resolutions over large cortical areas. With this technology at hand, we will eventually identify the areas to extract the most relevant signals and properly understand the meaning of cortical signals to optimize decoding protocols.
The overarching goal of BrainCom is to develop a new generation of very large-scale neuroprosthetic cortical devices based on novel materials and technologies that can provide a unique leap forward towards a better understanding of cortical speech networks and the advancement of rehabilitation solutions to restore speech and communication in disabled patients using innovative brain-computer paradigms. To target the broadly distributed neural system of the language network, BrainCom will use novel electronic technologies based on nanomaterials to fabricate ultra-flexible implants enabling recording over large cortical areas with unprecedented spatial and temporal resolution.
Very few data are available at the cellular and multicellular level, which is the required resolution to obtain sufficient decoding precision to predict continuous speech. One of the reasons is the lack of an available technology capable of recording neural signals with a high spatial and temporal resolutions over large cortical areas. With this technology at hand, we will eventually identify the areas to extract the most relevant signals and properly understand the meaning of cortical signals to optimize decoding protocols.
The overarching goal of BrainCom is to develop a new generation of very large-scale neuroprosthetic cortical devices based on novel materials and technologies that can provide a unique leap forward towards a better understanding of cortical speech networks and the advancement of rehabilitation solutions to restore speech and communication in disabled patients using innovative brain-computer paradigms. To target the broadly distributed neural system of the language network, BrainCom will use novel electronic technologies based on nanomaterials to fabricate ultra-flexible implants enabling recording over large cortical areas with unprecedented spatial and temporal resolution.
i) Develop electronic technologies for brain mapping to record from large number of active sites over large areas of the cortex. The g-SGFET technology has reached high maturity, achieving high performance and homogeneity, with yields close to 100%. We have validated the 2 different multiplexing strategies, time-domain and frequency-domain multiplexing (TDM and FDM), with arrays of 8x8 g-SGFET in acute in vivo experiments. The g-SGFETs technology has been used in chronically in vivo, demonstrating the potential of g-SGFET arrays for the study of wide frequency band LFP activity in freely moving animals. Regarding the scalability of the technology, arrays of 256 and 512 transistors have been already validated in saline. In addition, MoS2/Graphene flexible technology for multiplexing has been demonstrated by modelling, fabricating and testing monolithically integrated arrays.
Two 1024-channel compact ASICs have been designed, and fabricated in 0.18µm CMOS technology for the TDM and FDM multiplexing strategies of 32x32 GFET sensor arrays. Thanks to these novel strategies, the large-scale sensor arrays can be built from GFET-only devices with the corresponding simplification and reduction in costs of their manufacturability. The modular FPGA-based headstages incorporating the TDM and FDM ASICs have been also tested in vitro.
ii) Advance the fundamental understanding of the link between surface and intracortical signals and dynamics in cortical circuits. We have developed flexible cranial interface and performed first of a kind distributed recordings together with 3D behaviour, and a Green-function-based frequency domain procedure for LFP decomposition, and we have identified sparse and distributed anatomical basis for decoding of spatial representation from hippocampal LFP in theta frequency band. We have established joined depth and ECoG recordings in motor cortex combined with 3D limb analysis in head-fixed mice and identified contribution of theta dynamics to motor cortex activity; we have performed first wireless recordings of vocal and cortical ECoG activity in minipigs and found activation of motor cortex at single trial level.
iii) Gain new fundamental understanding of the distributed brain circuits of speech and their plastic flexibility before and after lesions. We characterized brain dynamics underlying overt and inner speech production in humans, both at the whole brain level using non-invasive, and more locally using invasive large-scale electrophysiology. ECoG datasets were collected in patients for word level speech production, characterizing the covert speech network. We have also developed original decoding methods, and characterization of best neural signals to be decoded. Intracortical activity was collected during covert and covert speech in Broca area, and found significant differences in the ensemble dynamics between both conditions, suggesting that this area of speech production is differently engaged between overt and covert production.
iv) Development of speech BCI proof of concept. We developed a software framework to customize real-time processing of data streams, incorporating real-time DNN-based articulatory-to-acoustic speech synthesis for closed-loop BCI applications. A first closed-loop speech BCI paradigm was tested in an epileptic patient. We developed a neuromorphic approach to perform online spike sorting that anticipates the advent of BrainCom technology for large scale neural recordings with low power. We developed deep-learning methods to decode neural activity underlying overt speech production. Intelligible speech reconstruction from cortical activity could be obtained for a number of speech items and in particular sequences of vowels. We have obtained preliminary decoding of covert speech production from cortical activity.
v) Developed an ethics framework to identify and explore issues linked to the use of brain implants. Collaborative research addressed questions generated by neural decoding techniques, including the prospect for involuntary speech and inadvertent ‘mind reading’. Further work addressed questions concerning user responsibility, shared control of devices, and the status of ‘brain data’. We addressed ethical policymaking for neurotechnologies with, e.g. contributions to think tanks, and to UK parliamentary understanding of neuroprosthetic technology – its opportunities and challenges. Wider audiences were reached via media interviews and debates, and via a substantial interactive ethics section of the BrainCom virtual reality app.
Two 1024-channel compact ASICs have been designed, and fabricated in 0.18µm CMOS technology for the TDM and FDM multiplexing strategies of 32x32 GFET sensor arrays. Thanks to these novel strategies, the large-scale sensor arrays can be built from GFET-only devices with the corresponding simplification and reduction in costs of their manufacturability. The modular FPGA-based headstages incorporating the TDM and FDM ASICs have been also tested in vitro.
ii) Advance the fundamental understanding of the link between surface and intracortical signals and dynamics in cortical circuits. We have developed flexible cranial interface and performed first of a kind distributed recordings together with 3D behaviour, and a Green-function-based frequency domain procedure for LFP decomposition, and we have identified sparse and distributed anatomical basis for decoding of spatial representation from hippocampal LFP in theta frequency band. We have established joined depth and ECoG recordings in motor cortex combined with 3D limb analysis in head-fixed mice and identified contribution of theta dynamics to motor cortex activity; we have performed first wireless recordings of vocal and cortical ECoG activity in minipigs and found activation of motor cortex at single trial level.
iii) Gain new fundamental understanding of the distributed brain circuits of speech and their plastic flexibility before and after lesions. We characterized brain dynamics underlying overt and inner speech production in humans, both at the whole brain level using non-invasive, and more locally using invasive large-scale electrophysiology. ECoG datasets were collected in patients for word level speech production, characterizing the covert speech network. We have also developed original decoding methods, and characterization of best neural signals to be decoded. Intracortical activity was collected during covert and covert speech in Broca area, and found significant differences in the ensemble dynamics between both conditions, suggesting that this area of speech production is differently engaged between overt and covert production.
iv) Development of speech BCI proof of concept. We developed a software framework to customize real-time processing of data streams, incorporating real-time DNN-based articulatory-to-acoustic speech synthesis for closed-loop BCI applications. A first closed-loop speech BCI paradigm was tested in an epileptic patient. We developed a neuromorphic approach to perform online spike sorting that anticipates the advent of BrainCom technology for large scale neural recordings with low power. We developed deep-learning methods to decode neural activity underlying overt speech production. Intelligible speech reconstruction from cortical activity could be obtained for a number of speech items and in particular sequences of vowels. We have obtained preliminary decoding of covert speech production from cortical activity.
v) Developed an ethics framework to identify and explore issues linked to the use of brain implants. Collaborative research addressed questions generated by neural decoding techniques, including the prospect for involuntary speech and inadvertent ‘mind reading’. Further work addressed questions concerning user responsibility, shared control of devices, and the status of ‘brain data’. We addressed ethical policymaking for neurotechnologies with, e.g. contributions to think tanks, and to UK parliamentary understanding of neuroprosthetic technology – its opportunities and challenges. Wider audiences were reached via media interviews and debates, and via a substantial interactive ethics section of the BrainCom virtual reality app.
Building up on BrainCom technology, we expect to lay foundation for breakthroughs in both fundamental neuroscience and clinical rehabilitation. The BrainCom unique high-density implant technology will shed new light on the fundamental understanding of surface cortical activity with respect to intracortical signals in small and large animal models, which will naturally lead to identifying key neural activity features to be used in BCI decoders. Further, the brain dynamics underlying overt and inner speech production will be characterized in humans which will provide further insights into speech production. BrainCom novel technologies will advance two clinical applications. First, high-density surface recordings will improve intraoperative mapping of functional areas to guide tissue removal during brain surgery. Second, BrainCom technology aims at developing BCI solutions to restore continuous speech in patients unable to speak, opening a new area in functional rehabilitation. Ultimately, the outcome of BrainCom is expected to open up new research directions towards a better fundamental understanding of the brain activity and towards a new generation of brain-machine interfaces for neurorehabilitation.