Periodic Reporting for period 3 - NeuraViPeR (Neural Active Visual Prosthetics for Restoring Function)
Período documentado: 2023-09-01 hasta 2025-02-28
Over 300 million people world-wide are visually impaired and around 50% cannot benefit from existing and prospective treatments with gene and stem therapies, pharmaceuticals, and retinal prosthesis. Cortical neuroprostheses capable of evoking visual percepts through direct electrical stimulation of the occipital cortex are a promising approach for these patients. Current microelectrode arrays have too few electrodes and suffer from insufficient longevity for long-term stimulation and recording of neuronal activity. Furthermore, these devices are only able to restore a limited vision, with very low spatial resolution. NeuraViPeR aims to develop an advanced technology for a functional long-life high-channel count brain stimulation and recording implant, and new algorithms that can optimize the restoration of visual experience and that can be run on a low-power portable platform. It also aims to lay ground-breaking foundation for a radically new paradigm which consists not only of constructing a neuroprosthesis with thousands of electrodes but also the creation of adaptive machine learning algorithms for a new brain-computer interfacing technology, which will remain safe and effective for decades.
In the project, we developed innovative approaches for stimulation with high-electrode-count interfacing with the visual cortex. The work includes the creation of thin flexible probes that cause minimal tissue damage; new electrode coatings that will be stable even with long-term repeated electrical stimulation; and novel microchip methods for combining online channeling of the stimulation currents to many thousands of electrodes. It also combines stimulation with the monitoring of neuronal activity in higher cortical areas. We also developed new deep learning algorithms that transform the camera footage into stimulation patterns for the cortex and that use feedback on recorded brain states and eye tracking to improve perception in a closed-loop approach. The algorithms were mapped onto a low-latency, power-efficient hardware platform, to create a future neuroprosthesis system that is robust, and portable.
In the project, we developed innovative approaches for stimulation with high-electrode-count interfacing with the visual cortex. The work includes the creation of thin flexible probes that cause minimal tissue damage; new electrode coatings that will be stable even with long-term repeated electrical stimulation; and novel microchip methods for combining online channeling of the stimulation currents to many thousands of electrodes. It also combines stimulation with the monitoring of neuronal activity in higher cortical areas. We also developed new deep learning algorithms that transform the camera footage into stimulation patterns for the cortex and that use feedback on recorded brain states and eye tracking to improve perception in a closed-loop approach. The algorithms were mapped onto a low-latency, power-efficient hardware platform, to create a future neuroprosthesis system that is robust, and portable.
In view of technology, we realized flexible, ultra-thin high-channel count polyimide-based neural probes to be interfaced either with silicon chips or CMOS ASIC chips capable of recording neural activity and eliciting phosphenes by functional electrical stimulation. Different variants of electrode arrays were characterized in detail in view of electrode impedance and charge injection capacity. A CMOS-compatible assembly process and a respective implantation strategy using silicon-based insertion shuttles was established and successfully applied for in vivo recordings. For in vivo applications in non-human primates we designed a MR compatible linear probe version for implantation. We further developed the concept of a pedestal enabling the modular integration of 64-channel neural probes into a 1024 channel implant for non-human primates.
We designed, simulated and fabricated a CMOS ASIC for neural recording and stimulation, featuring 8 stimulation units, 128 output stages and 64 recording channels. The stimulation units offer flexible programming of pulse amplitude, frequency, duration and polarity. The recording channels provide low noise, low power amplification, filtering and digitization of multi-unit neural activity. To support in-vivo experiments with the flexible probes, we developed a modular data acquisition system with stackable, lightweight headstages and a base station unit (Viper Box), which can support the stimulation of more than 1000 channels simultaneously.
We completed the design of a neural network FPGA accelerator, VPDNN, that supports the required features of the phosphene stimulation network from SKU. Weight pruning and bit quantization of the network parameters were carried out to match the bit precision supported by the hardware. The system works with inputs from a camera and the network output was transmitted successfully to the recording and stimulation CMOS ASIC. Additional features were added to support the closed-loop setup with the NeuraViPeR probes in an animal system.
For validating the NeuraViPeR probe technology, mice were implanted with flexible polyimide probes in the primary visual cortex (area V1). Microstimulation through the electrodes successfully evoked a behavioral response in mice trained on a go/no-go stimulation detection task. We were able to monitor the stimulation threshold over up to 13 months revealing a stable perceptual threshold. To test direct efficacy of microstimulation on the neuronal recruitment we implanted mice with the multi-shank NeuraViper probe and combined microstimulation with 2-photon imaging.
A Reinforcement Learning (RL)-based framework was developed to optimize phosphene generation for navigation tasks while maintaining performance under dynamic perturbations. An in-silico RL-based prosthesis controller was designed for neural system restoration using recurrent neural networks. A biologically plausible phosphene simulator optimized stimulation parameters for naturalistic vision, modeling perceptual characteristics. A high-dimensional Bayesian optimization framework enabled patient-specific multi-electrode stimulation based on direct feedback. Gaze-contingent processing and eye movement integration were explored for neuroprosthetic vision in VR-based tasks. Two biologically inspired RL algorithms improved training efficiency using predictive processing and input decorrelation.
We have conducted simultaneous recordings and stimulation experiments over a 6-month period in the primary visual cortex of three blind subjects. We tested advanced stimulation protocols and refined methods to improve phosphene perception, adaptability, and usability. By incorporating real-time adjustments based on brain responses and patient feedback, we optimised stimulation settings for clearer, more stable perceptions. To ensure practical use, we tailored tasks to each subject’s interests and integrated eye-tracking technology to enhance usability.
We designed, simulated and fabricated a CMOS ASIC for neural recording and stimulation, featuring 8 stimulation units, 128 output stages and 64 recording channels. The stimulation units offer flexible programming of pulse amplitude, frequency, duration and polarity. The recording channels provide low noise, low power amplification, filtering and digitization of multi-unit neural activity. To support in-vivo experiments with the flexible probes, we developed a modular data acquisition system with stackable, lightweight headstages and a base station unit (Viper Box), which can support the stimulation of more than 1000 channels simultaneously.
We completed the design of a neural network FPGA accelerator, VPDNN, that supports the required features of the phosphene stimulation network from SKU. Weight pruning and bit quantization of the network parameters were carried out to match the bit precision supported by the hardware. The system works with inputs from a camera and the network output was transmitted successfully to the recording and stimulation CMOS ASIC. Additional features were added to support the closed-loop setup with the NeuraViPeR probes in an animal system.
For validating the NeuraViPeR probe technology, mice were implanted with flexible polyimide probes in the primary visual cortex (area V1). Microstimulation through the electrodes successfully evoked a behavioral response in mice trained on a go/no-go stimulation detection task. We were able to monitor the stimulation threshold over up to 13 months revealing a stable perceptual threshold. To test direct efficacy of microstimulation on the neuronal recruitment we implanted mice with the multi-shank NeuraViper probe and combined microstimulation with 2-photon imaging.
A Reinforcement Learning (RL)-based framework was developed to optimize phosphene generation for navigation tasks while maintaining performance under dynamic perturbations. An in-silico RL-based prosthesis controller was designed for neural system restoration using recurrent neural networks. A biologically plausible phosphene simulator optimized stimulation parameters for naturalistic vision, modeling perceptual characteristics. A high-dimensional Bayesian optimization framework enabled patient-specific multi-electrode stimulation based on direct feedback. Gaze-contingent processing and eye movement integration were explored for neuroprosthetic vision in VR-based tasks. Two biologically inspired RL algorithms improved training efficiency using predictive processing and input decorrelation.
We have conducted simultaneous recordings and stimulation experiments over a 6-month period in the primary visual cortex of three blind subjects. We tested advanced stimulation protocols and refined methods to improve phosphene perception, adaptability, and usability. By incorporating real-time adjustments based on brain responses and patient feedback, we optimised stimulation settings for clearer, more stable perceptions. To ensure practical use, we tailored tasks to each subject’s interests and integrated eye-tracking technology to enhance usability.
Our project results demonstrate the potential of intracortical microstimulation to restore functional vision in the blind. We systematically monitored stimulation parameters and how they depend on the electrode configurations to understand their impact on perception threshold, subjective appearance, and location in the visual field. Furthermore, we explored the potential of closed-loop strategies to enhance the effectiveness, safety, and long-term stability of visual cortex stimulation. We demonstrated that simultaneous stimulation via multiple electrodes can evoke discriminable phosphene percepts. Moreover, we assessed algorithms that transform camera images into phosphene patterns and the usability of phosphene perception in mobility and orientation tasks, laying the groundwork for future advancements in visual prosthetic technology. These results have a significant scientific impact in the field and support the case for using penetrating microelectrodes. They also highlight a number of important unanswered questions that have to be solved before a cortical visual neuroprosthesis can be considered a viable clinical therapy or option. The results of this project have helped to set the path for advancing the technology development that can provide greater confidence for individuals with profound blindness to navigate in many environments.