Periodic Reporting for period 1 - NEMO BMI (Auto-adaptive Neuromorphic Brain Machine Interface: toward fully embedded neuroprosthetics)
Période du rapport: 2022-10-01 au 2023-09-30
Nearly 746,000 people sustain a spinal cord injury every year, with dramatic human, societal, and economic costs, leading to impairment or complete loss of motor functions. Motor Brain-Machine Interfaces (BMIs) translate brain neural signals into commands to external effectors. BMIs raise hopes that limb mobility may be restored, providing patients with control over orthoses, prostheses, or over their own limbs using electrical stimulation. One of the main challlenges in the field of BMI is the need for regular and time-consuming calibration of the systems by a team of experts which prevents dissemination to a large patient population.
To address the need of the largest number of patients with SCI, the next generation of BMI needs to be compatible with daily, autonomous use.
BMIs aim at translating brain neural signals into commands to robotic [1] or implanted electrical stimulator [2] [3] effectors. The ongoing clinical trials carried out by EPFL, and CEA (STIMO-BSI - NCT04632290, UP2 - NCT05665998 and ‘BCI & Tetraplegia’ - NCT02550522) raise great hopes for SCI patients. They effectively assess the feasibility of chronic motor BMIs, based on the WIMAGINE® wireless Electrocorticogram (ECoG) recording implant, for long-term use in daily life.
Epidural spinal stimulation on the other hand aims to translate the commands into electrical stimulator effectors. ONWARD has developed ARCIM Therapy, an implantable medical grade neurostimulation platform with unique real-time control capabilities. This platform includes a implantable pulse generator (IPG) that enables real-time control of 16 stimulation channels.
In the NEMO-BMI project, we will develop new auto-adaptive algorithms for brain decoding and spinal cord stimulation that will significantly contribute to enhancing knowledge on brain adaptation mechanisms. We foresee novelty in the design and implementation of neuromorphic hardware to sustain fast, secure, miniaturized and low-power neuroprosthetics.
[1] for public access: English Portal - An unprecedented neuroprosthetic allows a tetraplegic patient fitted with an exoskeleton to move (cea.fr) or for the scientific article: Benabid et al, An exoskeleton controlled by an epidural wireless brain–machine interface in a tetraplegic patient: a proof-of-concept demonstration, The Lancet Neurology, 2019, https://doi.org/10.1016/S1474-4422(19)30321-7
[2] Wagner, F.B. Mignardot, JB., Le Goff-Mignardot, C.G. et al. Targeted neurotechnology restores walking in humans with spinal cord injury. Nature 563, 65–71 (2018). https://doi.org/10.1038/s41586-018-0649-2
[3] Henri Lorach, Guillaume Charvet, Jocelyne Bloch, Grégoire Courtine, Brain–spine interfaces to reverse paralysis , National Science Review, Volume 9, Issue 10, October 2022, nwac009, https://doi.org/10.1093/nsr/nwac009
To address the need of the largest number of patients with SCI, the next generation of BMI needs to be compatible with daily, autonomous use.
BMIs aim at translating brain neural signals into commands to robotic [1] or implanted electrical stimulator [2] [3] effectors. The ongoing clinical trials carried out by EPFL, and CEA (STIMO-BSI - NCT04632290, UP2 - NCT05665998 and ‘BCI & Tetraplegia’ - NCT02550522) raise great hopes for SCI patients. They effectively assess the feasibility of chronic motor BMIs, based on the WIMAGINE® wireless Electrocorticogram (ECoG) recording implant, for long-term use in daily life.
Epidural spinal stimulation on the other hand aims to translate the commands into electrical stimulator effectors. ONWARD has developed ARCIM Therapy, an implantable medical grade neurostimulation platform with unique real-time control capabilities. This platform includes a implantable pulse generator (IPG) that enables real-time control of 16 stimulation channels.
In the NEMO-BMI project, we will develop new auto-adaptive algorithms for brain decoding and spinal cord stimulation that will significantly contribute to enhancing knowledge on brain adaptation mechanisms. We foresee novelty in the design and implementation of neuromorphic hardware to sustain fast, secure, miniaturized and low-power neuroprosthetics.
[1] for public access: English Portal - An unprecedented neuroprosthetic allows a tetraplegic patient fitted with an exoskeleton to move (cea.fr) or for the scientific article: Benabid et al, An exoskeleton controlled by an epidural wireless brain–machine interface in a tetraplegic patient: a proof-of-concept demonstration, The Lancet Neurology, 2019, https://doi.org/10.1016/S1474-4422(19)30321-7
[2] Wagner, F.B. Mignardot, JB., Le Goff-Mignardot, C.G. et al. Targeted neurotechnology restores walking in humans with spinal cord injury. Nature 563, 65–71 (2018). https://doi.org/10.1038/s41586-018-0649-2
[3] Henri Lorach, Guillaume Charvet, Jocelyne Bloch, Grégoire Courtine, Brain–spine interfaces to reverse paralysis , National Science Review, Volume 9, Issue 10, October 2022, nwac009, https://doi.org/10.1093/nsr/nwac009
Two main axes were pursued during the first period: 1) auto-adaptive brain decoding algorithms and 2) auto-adaptive spinal cord stimulation.
1) During this first period, according to the plan, CEA and IICT worked on decoding algorithm software to improve the decoding performances in particular the stability of the decoding models.
One of our ultimate goals being to perform model updates without supervision during free use, CEA is currently developing an innovative Auto-adaptive BMI (A-BMI) approach adding a supplementary loop with the neuronal response decoder, which evaluates the level of coherence between user’s intended motions and effector actions. This neural response is also recorded by the WIMAGINE implant located on the duramater above the sensorimotor cortex. CEA has preliminary results with three patients showing that the classification results with the auto-adaptive strategy on 2 or 3 classes are closed to the one achieved with fully supervised decoder training.
Concerning the optimization of the decoding software, a thorough optimization of the algorithm for the inference was achieved and allowed us to run the decoder on a portable electronic board (Raspberry PI4) instead of a powerful laptop accounting for a 90% electrical consumption reduction. Meanwhile, CEA is also working on the specification for a silicon chip that may perform the brain motor intent decoding and reduce again the electrical power requirement.
Meanwhile a 3-dimensional Spiking neural network (3D-SNN) architecture adaptable to the individual brain template for features extraction from the recorded by ECoG brain signals was developed by IICT. A modular neuromorphic architecture for Motor Control Decoder (MCD) and Neural Response Decoder (NRD) was developed in Python. It incorporates 3D-SNN and a recurrent neural network - Echo state network (ESN) - for decoder output. An on-line training algorithm for MCD/NRD was developed. Optimization of model hyper-parameters as well as training and testing of the MCD was performed on a supercomputer platform using experimental data base provided by CEA. A good classification accuracy on recognition of types of movements from ECoG signals was achieved. Work on accuracy improvement with respect to desired trajectories prediction continues. Analyses and experimental study on the implementation of the algorithm on Intels' neuromorphic chip Loihi started and first tests of the 3D-SNN running speed on Intel's platform was performed.
2) Regarding the spinal cord stimulation, EPFL and ONWARD have implemented a safe and efficient search algorithm for stimulation parameters optimization. EPFL introduced a semi-automated algorithm allowing rapid personalization of Targeted Epidural Electrical Stimulation (TEES) of the spinal cord towards a specific motor function. EPFL demonstrated the capacity to optimize three different functional movements for the lower limb: hip flexion, knee extension and whole leg flexion, fundamental building blocks for locomotion. EPFL and ONWARD established a reward measure by characterizing the motor output through wireless electromyography and kinematics sensors. ONWARD evaluated the software resources and the optimal sensors to achieve product-grade implementation and integration with the ARC-IM system. The team compared to the human-driven approach, and demonstrated a significant reduction of the time to achieve personalized and optimized stimulation comparable or outperforming those optimized by highly trained experts. A proof of concept experiment demonstrating the capacity to extract stimulation satisfaction from cortical signals.
In the meantime several database were prepared and shared with the consortium for developing and testing the algorithms.
1) During this first period, according to the plan, CEA and IICT worked on decoding algorithm software to improve the decoding performances in particular the stability of the decoding models.
One of our ultimate goals being to perform model updates without supervision during free use, CEA is currently developing an innovative Auto-adaptive BMI (A-BMI) approach adding a supplementary loop with the neuronal response decoder, which evaluates the level of coherence between user’s intended motions and effector actions. This neural response is also recorded by the WIMAGINE implant located on the duramater above the sensorimotor cortex. CEA has preliminary results with three patients showing that the classification results with the auto-adaptive strategy on 2 or 3 classes are closed to the one achieved with fully supervised decoder training.
Concerning the optimization of the decoding software, a thorough optimization of the algorithm for the inference was achieved and allowed us to run the decoder on a portable electronic board (Raspberry PI4) instead of a powerful laptop accounting for a 90% electrical consumption reduction. Meanwhile, CEA is also working on the specification for a silicon chip that may perform the brain motor intent decoding and reduce again the electrical power requirement.
Meanwhile a 3-dimensional Spiking neural network (3D-SNN) architecture adaptable to the individual brain template for features extraction from the recorded by ECoG brain signals was developed by IICT. A modular neuromorphic architecture for Motor Control Decoder (MCD) and Neural Response Decoder (NRD) was developed in Python. It incorporates 3D-SNN and a recurrent neural network - Echo state network (ESN) - for decoder output. An on-line training algorithm for MCD/NRD was developed. Optimization of model hyper-parameters as well as training and testing of the MCD was performed on a supercomputer platform using experimental data base provided by CEA. A good classification accuracy on recognition of types of movements from ECoG signals was achieved. Work on accuracy improvement with respect to desired trajectories prediction continues. Analyses and experimental study on the implementation of the algorithm on Intels' neuromorphic chip Loihi started and first tests of the 3D-SNN running speed on Intel's platform was performed.
2) Regarding the spinal cord stimulation, EPFL and ONWARD have implemented a safe and efficient search algorithm for stimulation parameters optimization. EPFL introduced a semi-automated algorithm allowing rapid personalization of Targeted Epidural Electrical Stimulation (TEES) of the spinal cord towards a specific motor function. EPFL demonstrated the capacity to optimize three different functional movements for the lower limb: hip flexion, knee extension and whole leg flexion, fundamental building blocks for locomotion. EPFL and ONWARD established a reward measure by characterizing the motor output through wireless electromyography and kinematics sensors. ONWARD evaluated the software resources and the optimal sensors to achieve product-grade implementation and integration with the ARC-IM system. The team compared to the human-driven approach, and demonstrated a significant reduction of the time to achieve personalized and optimized stimulation comparable or outperforming those optimized by highly trained experts. A proof of concept experiment demonstrating the capacity to extract stimulation satisfaction from cortical signals.
In the meantime several database were prepared and shared with the consortium for developing and testing the algorithms.