Periodic Reporting for period 1 - NIMA (NIMA: Non-invasive Interface for Movement Augmentation)
Reporting period: 2020-10-01 to 2021-09-30
Humans interact with the environment in a dexterous and highly efficient way that is unmatched by any other living organism. However, using additional artificial limbs in conjunction with natural limbs (e.g. a third arm or hand instead of only two) would allow for a seemingly infinite number of new possibilities to interact with the environment. In the research project NIMA we work towards making this vision a reality. The overall aim is to design, build and test interfaces that allow persons to control artificial limbs (including ‘virtual limbs’ or applications on a computer or mobile device) in coordination with their natural limbs.
If successful, NIMA will take human motor capacities to a new level and allow humans to perform tasks that are impossible to perform with the natural limbs alone. The reach of such movement augmentation is far, with applications ranging from work in industrial environments (e.g. overhead work in aircraft assembly), and surgical environments, to everyday applications where subjects can control their smartphone apps while simultaneously using their hands. For instance, laparoscopic surgeons could become able to perform surgical procedures with three hands allowing them to carry out tasks that require a skilled synchronisation of the manipulation of three surgical instruments that currently cannot be realised with minimally invasive access.
As an interdisciplinary team of experts in neuroscience, neurotechnology, human-machine interfaces, robotics, and ethics, we collaborate to accomplish the following objectives: (i) Push the borders of technology by creating a wearable supernumerary robotic limb (SRL), and non-invasive interfaces with multimodal sensory feedback to control multiple limbs or objects; (ii) Understand the cognitive and neural mechanisms underlying movement augmentation; (iii) Apply movement augmentation to extend a surgeon capabilities and autonomy, develop manipulation with a wearable supernumerary robotic arm and the two hands as well as a 3-hands computer interface; (iv) Evaluate the ethical and safety aspects of movement augmentation.
If successful, NIMA will take human motor capacities to a new level and allow humans to perform tasks that are impossible to perform with the natural limbs alone. The reach of such movement augmentation is far, with applications ranging from work in industrial environments (e.g. overhead work in aircraft assembly), and surgical environments, to everyday applications where subjects can control their smartphone apps while simultaneously using their hands. For instance, laparoscopic surgeons could become able to perform surgical procedures with three hands allowing them to carry out tasks that require a skilled synchronisation of the manipulation of three surgical instruments that currently cannot be realised with minimally invasive access.
As an interdisciplinary team of experts in neuroscience, neurotechnology, human-machine interfaces, robotics, and ethics, we collaborate to accomplish the following objectives: (i) Push the borders of technology by creating a wearable supernumerary robotic limb (SRL), and non-invasive interfaces with multimodal sensory feedback to control multiple limbs or objects; (ii) Understand the cognitive and neural mechanisms underlying movement augmentation; (iii) Apply movement augmentation to extend a surgeon capabilities and autonomy, develop manipulation with a wearable supernumerary robotic arm and the two hands as well as a 3-hands computer interface; (iv) Evaluate the ethical and safety aspects of movement augmentation.
Controlling SRLs requires an appropriate interface. One primary focus of NIMA is therefore to investigate three different kinds of non-invasive interfaces for movement augmentation: (1) a body interface (BI) using measured movement or force of the body part it is affixed to as input signals, (2) a muscle interface (MI) that picks up muscle activity and (3) a neural interface (NI) that extracts the signals of motor units (corresponding to the activity of motor neurons in the spinal cord).
Substantial progress has been made in developing and evaluating body interfaces for movement augmentation where our results demonstrate that subjects can control three virtual hands. For the neural interface we investigated whether subjects can voluntarily control individual motor units from a single muscle independently as a basis to control multiple limbs simultaneously. Our findings suggest that motor units could not be controlled independently during initial recruitment but once motor units were active flexibility of control may increase. For the muscle interface, a pilot study on the wrist was carried out with preliminary results on muscle activation patterns that subjects can generate independent of wrist force.
A wearable sensory feedback suit able to provide vibrotactile and electrotactile stimulation with high versatility and large bandwidth has been developed and validated. We could confirm the feasibility of proprioceptive-guided real-time tracking of an SRL through our system.
A wearable SRL was developed based on a light commercial manipulator and tested. As a step towards application of movement augmentation to robot-assisted surgery, an ergonomic foot interface with four degrees-of-freedom has been used to control a soft endoscope.
The challenge of human motor augmentation raises numerous ethical and safety questions, both at the micro scale of individual (safety of users sharing their workspace with a SRL or being impacted by movement augmentation abilities) or at the macro scale of society and humanity (ethics of movement augmentation). Regarding workspace safety; a risk analysis was conducted with the NIMA SRL platforms and a set of hardware and software measures were proposed to ensure user safety. In parallel, ethical reflections on issues were developed and an online survey was administered to relevant stakeholders for gathering information on potential real-life interactions enabled by NIMA.
Substantial progress has been made in developing and evaluating body interfaces for movement augmentation where our results demonstrate that subjects can control three virtual hands. For the neural interface we investigated whether subjects can voluntarily control individual motor units from a single muscle independently as a basis to control multiple limbs simultaneously. Our findings suggest that motor units could not be controlled independently during initial recruitment but once motor units were active flexibility of control may increase. For the muscle interface, a pilot study on the wrist was carried out with preliminary results on muscle activation patterns that subjects can generate independent of wrist force.
A wearable sensory feedback suit able to provide vibrotactile and electrotactile stimulation with high versatility and large bandwidth has been developed and validated. We could confirm the feasibility of proprioceptive-guided real-time tracking of an SRL through our system.
A wearable SRL was developed based on a light commercial manipulator and tested. As a step towards application of movement augmentation to robot-assisted surgery, an ergonomic foot interface with four degrees-of-freedom has been used to control a soft endoscope.
The challenge of human motor augmentation raises numerous ethical and safety questions, both at the micro scale of individual (safety of users sharing their workspace with a SRL or being impacted by movement augmentation abilities) or at the macro scale of society and humanity (ethics of movement augmentation). Regarding workspace safety; a risk analysis was conducted with the NIMA SRL platforms and a set of hardware and software measures were proposed to ensure user safety. In parallel, ethical reflections on issues were developed and an online survey was administered to relevant stakeholders for gathering information on potential real-life interactions enabled by NIMA.
The progress in the first year has developed and assessed efficient body interfaces for trimanual manipulation and as a potential interface to assist and extend surgical manipulation, with superior performance to current control of third arm robot in surgery using switching of activities. Moreover, the developed feedback suit offers, for the first time, an integrated, flexible and powerful tool to provide users with different types of haptic feedback, able to convey information that can be used for real-time tracking of SRLs. A wearable Supernumerary Robotic Limb platform was designed and implemented which can be used for development and testing of SRL control strategies. Additionally, a generic framework to analyse workspace safety of SRLs was proposed. These works and achievements, together with the pioneering work on muscle and neural interfaces, give the consortium an outstanding international position in the emerging field of movement augmentation.
Already at this early stage of the project, the consortium has published several publications at high-ranked journals and conferences including joint publications from multiple partners such as a comprehensive review on movement augmentation and a submitted manuscript on ethics of augmentation. Furthermore, several members of the consortium have launched a lecture series and organised a symposium on movement augmentation at the IEEE conference on Neural Engineering 2021 increasing visibility of work carried out in NIMA. In summary, with its foundational work, we expect NIMA to provide building blocks of knowledge, skills, methods and technologies as well as safety data and ethical considerations that future work in movement augmentation will be able to build upon.
Already at this early stage of the project, the consortium has published several publications at high-ranked journals and conferences including joint publications from multiple partners such as a comprehensive review on movement augmentation and a submitted manuscript on ethics of augmentation. Furthermore, several members of the consortium have launched a lecture series and organised a symposium on movement augmentation at the IEEE conference on Neural Engineering 2021 increasing visibility of work carried out in NIMA. In summary, with its foundational work, we expect NIMA to provide building blocks of knowledge, skills, methods and technologies as well as safety data and ethical considerations that future work in movement augmentation will be able to build upon.