Periodic Reporting for period 2 - NeuTouch (Understanding neural coding of touch as enabling technology for prosthetics and robotics)
Período documentado: 2021-03-01 hasta 2023-08-31
Touch is a gateway to explore the world and establish contact for interaction. It is crucial for robots to learn how to use objects, and enable cooperation, where robots and people act together to accomplish a task. It is crucial in the deployment of prostheses, to restore the sensation of contact.
Despite continuous advances, our knowledge is not yet sufficient to seamlessly integrate artificial limbs, nor to build agents that incorporate touch to interact intelligently with humans.
To progress, we educated interdisciplinary young investigators who will carry forward the research independently and propagate bio-inspired artificial touch to even younger investigators. NeuTouch aimed to build new technologies and deeper understanding of touch, while forming the next generation of researchers in this field.
Biological sensory systems capture the properties of surrounding objects and environment with remarkable efficiency. Grasping an object, our nervous system automatically adjusts hand force depending on the object’s size, weight, slipperiness, softness, using every available bit of information, optimising power consumption. Artificial systems acquire less information and consume more power; thus, technology has much to learn from biology.
NeuTouch aimed at improving artificial tactile systems through training and research. The training approach aimed to tutor young investigators to connect multiple disciplines relevant to the study of touch. NeuTouch assembled 8 research teams across Europe (IIT, EPFL, SISSA, Univ. of Bielefeld, Sheffield, Glasgow, Groningen, Goteborg, PAL robotics) and trained 15 PhD students from different countries (Germany, Italy, USA, Portugal, Spain, Iran, Turkey). They unravelled the principles of how biological tactile systems work – shedding light on the mechanics and neurobiology of touch perception – in synergy with the development of smarter and efficient technologies. The NeuTouch team built on this progress to prototype devices and circuits for artificial tactile sensors that will be integrated into the next generation of prostheses and robots. The team advanced robotic tactile exploration and manipulation and the technology for connecting to the nerves of prosthetic users. In parallel, the team developed skills towards best practices in science and communication and learnt the strength of cross-fertilisation across disciplines, culture, and peers to progress science to the benefit of society.
Despite continuous advances, our knowledge is not yet sufficient to seamlessly integrate artificial limbs, nor to build agents that incorporate touch to interact intelligently with humans.
To progress, we educated interdisciplinary young investigators who will carry forward the research independently and propagate bio-inspired artificial touch to even younger investigators. NeuTouch aimed to build new technologies and deeper understanding of touch, while forming the next generation of researchers in this field.
Biological sensory systems capture the properties of surrounding objects and environment with remarkable efficiency. Grasping an object, our nervous system automatically adjusts hand force depending on the object’s size, weight, slipperiness, softness, using every available bit of information, optimising power consumption. Artificial systems acquire less information and consume more power; thus, technology has much to learn from biology.
NeuTouch aimed at improving artificial tactile systems through training and research. The training approach aimed to tutor young investigators to connect multiple disciplines relevant to the study of touch. NeuTouch assembled 8 research teams across Europe (IIT, EPFL, SISSA, Univ. of Bielefeld, Sheffield, Glasgow, Groningen, Goteborg, PAL robotics) and trained 15 PhD students from different countries (Germany, Italy, USA, Portugal, Spain, Iran, Turkey). They unravelled the principles of how biological tactile systems work – shedding light on the mechanics and neurobiology of touch perception – in synergy with the development of smarter and efficient technologies. The NeuTouch team built on this progress to prototype devices and circuits for artificial tactile sensors that will be integrated into the next generation of prostheses and robots. The team advanced robotic tactile exploration and manipulation and the technology for connecting to the nerves of prosthetic users. In parallel, the team developed skills towards best practices in science and communication and learnt the strength of cross-fertilisation across disciplines, culture, and peers to progress science to the benefit of society.
NeuTouch fellows studied biological touch, with a focus on contributions relevant to design better artificial touch and improving information throughput. G. Corniani quantified how tactile sensors are distributed on the human body, and the mechanical response of skin during touch. V. Lang, M. Casal and G. Corniani improved upon the traditional understanding of tactile encoding, by showing that information is not segregated into different neuron categories. M. Casal developed an open-source software to facilitate analyses of large neural populations. Z. Darani showed that perception of vibration intensity adapts to changing statistics of the world, demonstrating a cognitive aspect of tactile perception.
For prostheses (WP1), work focused on technological developments that allow for the necessary information throughput identified in the biological results. O. Akoussi improved long term nerve stimulation and recordings by developing a model to simulate the interaction between nerve and electrodes that led to new electrodes that minimize the stresses applied on the nerve. M. Ozkan studied materials that stimulate neural growth for improving the connection between electrodes and nerves. G. Dominijanni developed a platform to study sensory-motor learning and how the brain represents embodiment and tactile information, paving the way towards the development of a new generation of wearable robots to allow three-manual tasks.
For robots (WP2), work focused on high-throughput and low-power algorithms for solving practical tasks. S. Müller-Cleve studied tactile exploration strategies by measuring interaction forces between human subjects’ hands and objects during blind object recognition. P. Villacis modelled tactile information integration over time and applied it to robotic tactile exploration. Robots and simulators were equipped with tactile sensing: S. Müller-Cleve implemented a skin-plugin for the iCub hand in the Gazebo and MuJoCo simulators and a spiking neural network for the classification of Braille letters that consumes 1000 times less power. J. Cmiral simulated force profiles observed when operating switches, knobs, and sliders. L. Lach developed a grasp and place controller with tactile feedback that reduces object movements and prevents damage.
For artificial tactile sensors (WP3), J. Neto and L. De Pamphilis improved techniques to print nanowires and nanotransistors for flexible substrates. E. Janotte and M. Mastella developed low-power circuits for the readout of piezoelectric and capacitive tactile sensors and for edge pre-processing using spiking neural networks for orientation and texture selectivity.
Fellows worked collaboratively within and across topics, supported by regular meetings (online or in-person) and 14 secondments, facilitating knowledge exchange and joint publications.
For prostheses (WP1), work focused on technological developments that allow for the necessary information throughput identified in the biological results. O. Akoussi improved long term nerve stimulation and recordings by developing a model to simulate the interaction between nerve and electrodes that led to new electrodes that minimize the stresses applied on the nerve. M. Ozkan studied materials that stimulate neural growth for improving the connection between electrodes and nerves. G. Dominijanni developed a platform to study sensory-motor learning and how the brain represents embodiment and tactile information, paving the way towards the development of a new generation of wearable robots to allow three-manual tasks.
For robots (WP2), work focused on high-throughput and low-power algorithms for solving practical tasks. S. Müller-Cleve studied tactile exploration strategies by measuring interaction forces between human subjects’ hands and objects during blind object recognition. P. Villacis modelled tactile information integration over time and applied it to robotic tactile exploration. Robots and simulators were equipped with tactile sensing: S. Müller-Cleve implemented a skin-plugin for the iCub hand in the Gazebo and MuJoCo simulators and a spiking neural network for the classification of Braille letters that consumes 1000 times less power. J. Cmiral simulated force profiles observed when operating switches, knobs, and sliders. L. Lach developed a grasp and place controller with tactile feedback that reduces object movements and prevents damage.
For artificial tactile sensors (WP3), J. Neto and L. De Pamphilis improved techniques to print nanowires and nanotransistors for flexible substrates. E. Janotte and M. Mastella developed low-power circuits for the readout of piezoelectric and capacitive tactile sensors and for edge pre-processing using spiking neural networks for orientation and texture selectivity.
Fellows worked collaboratively within and across topics, supported by regular meetings (online or in-person) and 14 secondments, facilitating knowledge exchange and joint publications.
M. Casal, Z. Darani, G. Corniani, V. Lang, and G. Dominijanni deepened our understanding of how organisms extract information from tactile stimuli, and on how the perception of tactile stimuli depends on the past, on how tactile and somatosensory information is represented in the brain, and on how this information influences behaviour. This understanding is fundamental to deliver better tactile perception in artificial devices, improving their capabilities in supporting humans both by assistive and collaborative work from robots, and for better usability of prosthetic devices. To this aim, O. Akoussi and M. Ozkan developed electrodes for improving the connection with nerves towards their direct stimulation for restoring tactile sensation; S. Müller-Cleve, P. Villacis, L. Lach and J. Cmiral have progressed the technology for robotic tactile exploration and manipulation, improving the grasping and placement of objects.
M. Mastella, E. Janotte, L. De Pamphilis and J. Neto made substantial progress in the development of devices and circuits for smart, efficient, and miniaturised artificial tactile sensors.
NeuTouch fellows provided open-source tools to analyse neural encoding, test tactile-based grasping and manipulation in robots, test tactile feedback in virtual environments, study the interaction of electrodes with neural tissue for improving implants for sensory feedback. These tools support a better understanding of the biological sense of touch and the improvement of the use of touch in robots and prosthetics.
End Result: 6 ESRs obtained their doctorates and are now employed either in industry or in academia as postdoctoral fellows, 8 are about to graduate. NeuTouch offered 3 schools and 3 workshops. It delivered 25 publications in top tier journals (Nature Machine Intelligence, Scientific Reports, Frontiers, Applied Electronic Materials, PLOS Comp. Biology, J. of Neural Engineering, J. of Neurophysiology) and conferences (ISCAS, IROS, ICRA, ICONS, EMBC) and 1 patent application. These are significant underestimates as NeuTouch-based research will continue to be seen in the literature and in the technology domain for several years.
M. Mastella, E. Janotte, L. De Pamphilis and J. Neto made substantial progress in the development of devices and circuits for smart, efficient, and miniaturised artificial tactile sensors.
NeuTouch fellows provided open-source tools to analyse neural encoding, test tactile-based grasping and manipulation in robots, test tactile feedback in virtual environments, study the interaction of electrodes with neural tissue for improving implants for sensory feedback. These tools support a better understanding of the biological sense of touch and the improvement of the use of touch in robots and prosthetics.
End Result: 6 ESRs obtained their doctorates and are now employed either in industry or in academia as postdoctoral fellows, 8 are about to graduate. NeuTouch offered 3 schools and 3 workshops. It delivered 25 publications in top tier journals (Nature Machine Intelligence, Scientific Reports, Frontiers, Applied Electronic Materials, PLOS Comp. Biology, J. of Neural Engineering, J. of Neurophysiology) and conferences (ISCAS, IROS, ICRA, ICONS, EMBC) and 1 patent application. These are significant underestimates as NeuTouch-based research will continue to be seen in the literature and in the technology domain for several years.