Periodic Reporting for period 1 - INTERSPINE (From Neurons to Robots: Non-Invasive, General-Purpose Interfacing With Human Spinal Motor Neurons)
Reporting period: 2017-07-01 to 2018-12-31
Summary of the context and overall objectives of the project
1. Introduction
During the ERC Advanced grant DEMOVE (“Decoding the Neural Code of Human Movements for a New Generation of Man-machine Interfaces”), Prof. Farina and his team had developed electrodes and computational methods that allow the understanding of the mechanisms underlying the generation of human movements, from the cellular level to the function. These methods not only contributed to revising established principles in our fundamental knowledge on human movement (as well as establishing new ones), but they also helped develop novel methods for controlling robotic bionic limbs. With the ERC PoC INTERSPINE, we have translated these latter methods, and specifically those based on non-implanted electrodes, into the development of a human-machine interface system that extracts neural information from the human spinal cord and transforms it into accurate commands to external devices, such as prostheses, teleoperated robots, and computer games.
During DEMOVE, it was demonstrated (in laboratory conditions) that neural information sent from the spinal cord to the human muscles through the nerves can be identified and fully decoded from electrical signals detected using electrodes placed on the skin overlying the innervated muscles. In other words, it is possible to establish a direct non-invasive neural interface by recording surface electromyogram (sEMG) signals and extract the timing of each activation of neural cells in the spinal cord, known as motor neurons. This requires tens to hundreds of electrodes (i.e. high-density sEMG) combined with wearable devices for recording, as well as real-time processing methods to extract the neural information buried in the body-surface electrical potentials, i.e. sEMG decomposition.
INTERSPINE have extended the approaches established in DEMOVE by developing a system with real-time processing capabilities, which were not contemplated in DEMOVE. The real-time operation crucially opens the pathway to true neural interfacing between humans and external devices since it allows natural and online control. Moreover, the non-invasiveness of the proposed approach poses this effort in a unique perspective globally since it proposes the first neural interface based on wearable devices. We tested and confirmed the accuracy of the system, and we validated its real-time performance, with both synthetic and experimental high-density sEMG (i.e. HD-sEMG) signals. The resulting system achieves full neural decoding from surface recordings with a computational time within a few ms per processing interval. Finally, in INTERSPINE we also fully demonstrated the viability of this real-time neural interfacing system in the control of an upper-limb prosthesis.
2. Brief System description
The proposed system consists of a module for the recording of electrical signals from muscles in multiple locations (>100 locations), and an algorithm embedded in customized electronics for the real-time decoding of the muscle electrical activity into the activity of the neural cells in the spinal cord that control the target muscles. The algorithm was developed during the Advanced ERC grant DEMOVE while its real-time implementation was the focus of INTERSPINE.
The core of the decoding procedure is the identification of a separation matrix that is applied to the multi-channel EMG recordings to identify the neural spiking activity from individual spinal neural cells. This separation matrix is identified during a brief training phase of the algorithm and then applied in real-time during the algorithm execution. The result is the full decoding of the neural code underlying motor tasks that, because of the real-time implementation, can then be used to drive any external devices, in applications such as prostheses, gaming, computer interaction.
3. Validation & Results
The proposed system has been extensively validated through synthetic sEMG signal datasets (with known ground truth) and experimental sEMG recordings. Finally, the system was also demonstrated via a multi-degree of freedom prosthetic control paradigm, including tests on patients with upper limb loss.
Synthetic sEMG datasets were generated using a model previously developed by the research group of Prof. Farina and specifically customized for INTERSPINE. The simulations included realistic EMG signals as recorded by ~200 electrodes, similar to the experimental conditions. The >1000 simulated signals varied in terms of simulated anatomy, location of the sources, additive noise, and other generation model parameters. The tested real-time decomposition achieved accuracy similar (within 0.5%) to state of the art offline systems, with a computational time 20x faster. The system was further tested in experimental recordings from the Tibialis Anterior (TA) and Flexor Digitorum Superficialis (FDS) muscles, as representative muscles of the lower and upper limb, in >10 individuals and >50 experimental sessions. Again, the experimental results showed that the performance of the developed real-time system was comparable with the offline state of the art.
Finally, the system was demonstrated in a prosthetic hand control (Michelangelo Hand, Ottobock) paradigm where the movements along two degrees-of-freedom (DoF) (hand open/close and hand pronate/supinate) were controlled in real-time by the decomposed neural activity. The control was achieved by applying machine learning to the output of the real-time decoding system. A graphical user interface (GUI), that displays decomposed neural activity as feedback for training with the neural interface, was also developed. The resulting control system allowed mapping of four hand/wrist movements in real-time control with accuracy >90%. This is the first demonstration of real-time neural interfacing with a wearable system.
Part of these results were published and reported in “Barsakcioglu DY, Farina D, 2018, A real-time surface EMG decomposition system for non-invasive human-machine interfaces.” A patent application for the system was also filed on August 23rd 2018 (Application No. 1812951.0) and we are currently in discussion with relevant representatives of industry for future licencing arrangements.
4. On-going work
The objective of INTERSPINE was not only to develop the first man-machine interface based on decoded neural cell activity in a completely non-invasive way, but also to demonstrate a robust association between the identified neural activity and external commands to devices. This has been demonstrated during the project, as planned, within a 2-DoF classifier-based prosthetic hand control scenario. We are currently extending this paradigm by conducting experimental studies, on healthy individuals and amputee patients, to demonstrate our system under a 3-DoF simultaneous and proportional control scenario. To date, there exists no demonstration for intuitive, robust and accurate 3-DoF simultaneous and proportional control in upper limb prosthetics, and the system developed within INTERSPINE will pioneer this research gap. The preliminary findings in this respect are very promising.
In parallel, the system is being fully implemented and tested in an FPGA-based real-time sEMG decomposition hardware, which allows a fully wearable implementation. A wearable real-time decomposition device will be finalised in the following months using an off-the-shelf commercial analogue-front-end device, FPGA-based real-time sEMG decomposition module, and an off-the-shelf wireless communication module.
5. Socioeconomic Impact
We have managed to provide a proof-of-concept demonstration of the first interface with spinal neural cells completely realized with non-invasive, wearable systems. Robust, accurate, and effective human-machine interfaces are at the core of every assistive/rehabilitation sciences and related clinical/industrial systems. Compared to the state-of-the-art in prosthesis control, for example, the system developed in INTERSPINE offers a more accurate, robust and enriched set of commands. This will have a direct impact in the upper limb prosthetic field, considering that current man-machine interfacing systems in this field are rejected by approximately half of the patients (Farina & Aszmann. Sci Transl Med. 6(257):257ps12 2014; Jiang et al. IEEE Signal Process. Mag 29(5), 152-150, 2012). Considering 1.5 million with upper or lower limb loss in the U.S. only and 230,000 new cases occurring each year (similar statistics, in relative terms, are valid for Europe), the expected economic impact for health services, governments, people directly affected (amputees and their families) and medical devices industry is very large.
The developed system is not limited to assistive devices only. Being a universal interface for accurate control, the technology developed in INTERSPINE will also open new market opportunities for robotic rehabilitation in therapeutic applications (e.g. stroke patients and other motor impairments), robots for telesurgery, electrical stimulation systems for suppression of tremor, as well as in a number of non-medical areas. The industry of computer games and smart portable devices, for example, would be an additional target (and we are indeed in negotiation with a company targeting these areas for licensing our technology).
6. Conclusions
We have achieved the goal of proving the proof of concept for a new neurotechnology that targets large-scale applications. We have done so by leveraging the basic science results and algorithms developed in the ERC Advanced Grant DEMOVE. We have furthered these findings with the main aim of real-time neural control, which has been successfully achieved and extensively tested and validated. We believe the market of rehabilitation/assistive devices as well as consumer electronics could be revolutionized with the introduction of the developed non-invasive, accurate neural interface with the spinal cord circuitries. Having developed, verified and demonstrated the system, we are certain that in the very near future, we will see it utilised in several application domains.
During the ERC Advanced grant DEMOVE (“Decoding the Neural Code of Human Movements for a New Generation of Man-machine Interfaces”), Prof. Farina and his team had developed electrodes and computational methods that allow the understanding of the mechanisms underlying the generation of human movements, from the cellular level to the function. These methods not only contributed to revising established principles in our fundamental knowledge on human movement (as well as establishing new ones), but they also helped develop novel methods for controlling robotic bionic limbs. With the ERC PoC INTERSPINE, we have translated these latter methods, and specifically those based on non-implanted electrodes, into the development of a human-machine interface system that extracts neural information from the human spinal cord and transforms it into accurate commands to external devices, such as prostheses, teleoperated robots, and computer games.
During DEMOVE, it was demonstrated (in laboratory conditions) that neural information sent from the spinal cord to the human muscles through the nerves can be identified and fully decoded from electrical signals detected using electrodes placed on the skin overlying the innervated muscles. In other words, it is possible to establish a direct non-invasive neural interface by recording surface electromyogram (sEMG) signals and extract the timing of each activation of neural cells in the spinal cord, known as motor neurons. This requires tens to hundreds of electrodes (i.e. high-density sEMG) combined with wearable devices for recording, as well as real-time processing methods to extract the neural information buried in the body-surface electrical potentials, i.e. sEMG decomposition.
INTERSPINE have extended the approaches established in DEMOVE by developing a system with real-time processing capabilities, which were not contemplated in DEMOVE. The real-time operation crucially opens the pathway to true neural interfacing between humans and external devices since it allows natural and online control. Moreover, the non-invasiveness of the proposed approach poses this effort in a unique perspective globally since it proposes the first neural interface based on wearable devices. We tested and confirmed the accuracy of the system, and we validated its real-time performance, with both synthetic and experimental high-density sEMG (i.e. HD-sEMG) signals. The resulting system achieves full neural decoding from surface recordings with a computational time within a few ms per processing interval. Finally, in INTERSPINE we also fully demonstrated the viability of this real-time neural interfacing system in the control of an upper-limb prosthesis.
2. Brief System description
The proposed system consists of a module for the recording of electrical signals from muscles in multiple locations (>100 locations), and an algorithm embedded in customized electronics for the real-time decoding of the muscle electrical activity into the activity of the neural cells in the spinal cord that control the target muscles. The algorithm was developed during the Advanced ERC grant DEMOVE while its real-time implementation was the focus of INTERSPINE.
The core of the decoding procedure is the identification of a separation matrix that is applied to the multi-channel EMG recordings to identify the neural spiking activity from individual spinal neural cells. This separation matrix is identified during a brief training phase of the algorithm and then applied in real-time during the algorithm execution. The result is the full decoding of the neural code underlying motor tasks that, because of the real-time implementation, can then be used to drive any external devices, in applications such as prostheses, gaming, computer interaction.
3. Validation & Results
The proposed system has been extensively validated through synthetic sEMG signal datasets (with known ground truth) and experimental sEMG recordings. Finally, the system was also demonstrated via a multi-degree of freedom prosthetic control paradigm, including tests on patients with upper limb loss.
Synthetic sEMG datasets were generated using a model previously developed by the research group of Prof. Farina and specifically customized for INTERSPINE. The simulations included realistic EMG signals as recorded by ~200 electrodes, similar to the experimental conditions. The >1000 simulated signals varied in terms of simulated anatomy, location of the sources, additive noise, and other generation model parameters. The tested real-time decomposition achieved accuracy similar (within 0.5%) to state of the art offline systems, with a computational time 20x faster. The system was further tested in experimental recordings from the Tibialis Anterior (TA) and Flexor Digitorum Superficialis (FDS) muscles, as representative muscles of the lower and upper limb, in >10 individuals and >50 experimental sessions. Again, the experimental results showed that the performance of the developed real-time system was comparable with the offline state of the art.
Finally, the system was demonstrated in a prosthetic hand control (Michelangelo Hand, Ottobock) paradigm where the movements along two degrees-of-freedom (DoF) (hand open/close and hand pronate/supinate) were controlled in real-time by the decomposed neural activity. The control was achieved by applying machine learning to the output of the real-time decoding system. A graphical user interface (GUI), that displays decomposed neural activity as feedback for training with the neural interface, was also developed. The resulting control system allowed mapping of four hand/wrist movements in real-time control with accuracy >90%. This is the first demonstration of real-time neural interfacing with a wearable system.
Part of these results were published and reported in “Barsakcioglu DY, Farina D, 2018, A real-time surface EMG decomposition system for non-invasive human-machine interfaces.” A patent application for the system was also filed on August 23rd 2018 (Application No. 1812951.0) and we are currently in discussion with relevant representatives of industry for future licencing arrangements.
4. On-going work
The objective of INTERSPINE was not only to develop the first man-machine interface based on decoded neural cell activity in a completely non-invasive way, but also to demonstrate a robust association between the identified neural activity and external commands to devices. This has been demonstrated during the project, as planned, within a 2-DoF classifier-based prosthetic hand control scenario. We are currently extending this paradigm by conducting experimental studies, on healthy individuals and amputee patients, to demonstrate our system under a 3-DoF simultaneous and proportional control scenario. To date, there exists no demonstration for intuitive, robust and accurate 3-DoF simultaneous and proportional control in upper limb prosthetics, and the system developed within INTERSPINE will pioneer this research gap. The preliminary findings in this respect are very promising.
In parallel, the system is being fully implemented and tested in an FPGA-based real-time sEMG decomposition hardware, which allows a fully wearable implementation. A wearable real-time decomposition device will be finalised in the following months using an off-the-shelf commercial analogue-front-end device, FPGA-based real-time sEMG decomposition module, and an off-the-shelf wireless communication module.
5. Socioeconomic Impact
We have managed to provide a proof-of-concept demonstration of the first interface with spinal neural cells completely realized with non-invasive, wearable systems. Robust, accurate, and effective human-machine interfaces are at the core of every assistive/rehabilitation sciences and related clinical/industrial systems. Compared to the state-of-the-art in prosthesis control, for example, the system developed in INTERSPINE offers a more accurate, robust and enriched set of commands. This will have a direct impact in the upper limb prosthetic field, considering that current man-machine interfacing systems in this field are rejected by approximately half of the patients (Farina & Aszmann. Sci Transl Med. 6(257):257ps12 2014; Jiang et al. IEEE Signal Process. Mag 29(5), 152-150, 2012). Considering 1.5 million with upper or lower limb loss in the U.S. only and 230,000 new cases occurring each year (similar statistics, in relative terms, are valid for Europe), the expected economic impact for health services, governments, people directly affected (amputees and their families) and medical devices industry is very large.
The developed system is not limited to assistive devices only. Being a universal interface for accurate control, the technology developed in INTERSPINE will also open new market opportunities for robotic rehabilitation in therapeutic applications (e.g. stroke patients and other motor impairments), robots for telesurgery, electrical stimulation systems for suppression of tremor, as well as in a number of non-medical areas. The industry of computer games and smart portable devices, for example, would be an additional target (and we are indeed in negotiation with a company targeting these areas for licensing our technology).
6. Conclusions
We have achieved the goal of proving the proof of concept for a new neurotechnology that targets large-scale applications. We have done so by leveraging the basic science results and algorithms developed in the ERC Advanced Grant DEMOVE. We have furthered these findings with the main aim of real-time neural control, which has been successfully achieved and extensively tested and validated. We believe the market of rehabilitation/assistive devices as well as consumer electronics could be revolutionized with the introduction of the developed non-invasive, accurate neural interface with the spinal cord circuitries. Having developed, verified and demonstrated the system, we are certain that in the very near future, we will see it utilised in several application domains.