Periodic Reporting for period 2 - NeuraViPeR (Neural Active Visual Prosthetics for Restoring Function) Periodo di rendicontazione: 2022-03-01 al 2023-08-31 Sintesi del contesto e degli obiettivi generali del progetto Over 300 million people world-wide are visually impaired and around 50% cannot benefit from existing and prospective treatments with gene and stem therapies, pharmaceuticals, and retinal prosthesis. Cortical neuroprostheses capable of evoking visual percepts through direct electrical stimulation of the occipital cortex are a promising approach for these patients. Current microelectrode arrays have too few electrodes and suffer from insufficient longevity for long-term stimulation and recording of neuronal activity. Furthermore, these devices are only able to restore a limited vision, with very low spatial resolution. NeuraViPeR aims to develop an advanced technology for a functional long-life high-channel count brain stimulation and recording implant, and new algorithms that can optimize the restoration of visual experience and that can be run on a low-power portable platform. It also aims to lay ground-breaking foundation for a radically new paradigm which consists not only of constructing a neuroprosthesis with thousands of electrodes but also the creation of adaptive machine learning algorithms for a new brain-computer interfacing technology, which will remain safe and effective for decades. In the project, we are developing innovative approaches for stimulation with high-electrode-count interfacing with the visual cortex. The work includes the creation of thin flexible probes that cause minimal tissue damage; new electrode coatings that will be stable even with long-term repeated electrical stimulation; and novel microchip methods for combining online channeling of the stimulation currents to many thousands of electrodes. It also combines stimulation with the monitoring of neuronal activity in higher cortical areas. We are also developing new deep learning algorithms that transform the camera footage into stimulation patterns for the cortex and that use feedback on recorded brain states and eye tracking to improve perception in a closed-loop approach. The software algorithms will be translated onto low-latency, power-efficient neuromorphic deep learning hardware, to create a neuroprosthesis system that is robust, and portable. Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti In view of technology, we realized flexible, ultra-thin high-channel count polyimide-based neural probes to be interfaced either with silicon chips or CMOS ASIC chips capable of recording neural activity and eliciting phosphenes by functional electrical stimulation. Different variants of electrode arrays were characterized in detail in view of electrode impedance and charge injection capacity. A CMOS-compatible assembly process and a respective implantation strategy using silicon-based insertion shuttles was established and successfully applied for in vivo recordings. For in vivo applications in non-human primates we designed a MR compatible linear probe version for implantation. We further developed the concept of a pedestal enabling the modular integration of 64-channel neural probes into a 1024 channel implant for non-human primates. We have designed, simulated and fabricated the first version of the recording and stimulation CMOS ASIC comprised of 8 stimulation units, 128 output stages and 64 recording channels. The stimulation units can flexibility be programmed in terms of pulse amplitude, frequency, duration and polarity. The recording channels provide low noise and low power amplification, filtering and digitization of multi-unit neural activity. We have completed the design of the convolutional neural network hardware accelerator, VPDNN, that supports the required features of the phosphene stimulation neural network. Its performance (latency, power) was compared to implementation on other embedded devices. We also applied weight pruning and bit quantization of parameters to the stimulation network corresponding to the bit precision supported by VPDNN. The system works with inputs from a camera and the network output was transmitted successfully to recording and stimulation CMOS ASIC. For validating the NeuraViPeR probe technology, mice were implanted with a flexible polyimide probes in the primary visual cortex (area V1). Microstimulation through the electrodes successfully evoked a behavioral response in mice trained on a go/no-go stimulation detection task. We were able to monitor the stimulation threshold over up to 13 months revealing a stable perceptual threshold. Currently we are focusing on the ex-vivo evaluation of probe biocompatibility. On the software side we developed a Reinforcement Learning (RL) based network to optimize phosphene generation taking into account task-specific constraints. This framework is used to train novel deep networks for stimulus generation in an end-to-end fashion to optimize phosphene patterns and solve multi-stage navigation-based tasks in virtual environments. Algorithms were developed which maintain performance in the presence of perturbations in dynamical systems. An in-silico framework to train a prosthesis controller via RL was designed to restore function in an impaired neural system using dynamical systems and recurrent neural networks (RNN) to formulate the components of a brain-machine interaction. An end-to-end framework including a biologically plausible phosphene simulator was used to learn safe stimulation parameters (amplitude, pulse width and frequency) for phosphene generation optimized for naturalistic vision settings via real-life navigation videos. We have conducted simultaneous recordings and stimulation experiments over a 6-month period in the primary visual cortex of three blind subjects. Our efforts have led to significant advancements in our protocols for selecting candidates for a cortical visual prosthesis. Throughout this process, we meticulously monitored the evolution of electrode impedance measurements, ensuring the stability and reliability of our electrodes over time. With each blind volunteer, we implemented protocols to evaluate both technical and functional aspects of using intracortical electrical stimulation as an interface for a visual neuroprosthesis. Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora) Our project results so far demonstrate the potential of intracortical microstimulation to restore functional vision in the blind. On the technical front, we systematically monitored stimulation parameters and experimented with different electrode configurations to understand their impact on perception threshold, subjective appearance, and location in the visual field. Furthermore, we explored the potential of closed-loop strategies to enhance the effectiveness, safety, and long-term stability of visual cortex stimulation. We have demonstrated that simultaneous stimulation via multiple electrodes can evoke discriminable phosphene percepts. Moreover, we assessed the usability of phosphene perception in mobility and orientation tasks laying the groundwork for future advancements in visual prosthetic technology.These results have a significant scientific impact in the field and support the case for using penetrating microelectrodes. They also highlight a number of important unanswered questions that have to be solved before a cortical visual neuroprosthesis can be considered a viable clinical therapy or option. We expect that our results until the end of the project will allow us to advance the development of a technology that can enhance safety in navigation and provide greater confidence for individuals with profound blindness in many environments. Concept of NeuraViPeR
