Periodic Reporting for period 2 - NIMA (NIMA: Non-invasive Interface for Movement Augmentation)
Periodo di rendicontazione: 2021-10-01 al 2024-03-31
Humans interact with the environment in a dexterous and highly efficient way that is unmatched by any other living organism. However, using additional artificial limbs in conjunction with natural limbs (e.g. a third arm or hand instead of only two) would allow for a seemingly infinite number of new possibilities to interact with the environment. In the research project NIMA we work towards making this vision a reality. The overall aim is to design, build and test interfaces that allow persons to control artificial limbs (including ‘virtual limbs’ or applications on a computer or mobile device) in coordination with their natural limbs.
If successful, NIMA will take human motor capacities to a new level and allow humans to perform tasks that are impossible to perform with the natural limbs alone. The reach of such movement augmentation is far, with applications ranging from work in industrial environments (e.g. overhead work in aircraft assembly), and surgical environments, to everyday applications where subjects can control their smartphone apps while simultaneously using their hands. For instance, laparoscopic surgeons could become able to perform surgical procedures with three hands allowing them to carry out tasks that require a skilled synchronisation of the manipulation of three surgical instruments that currently cannot be realised with minimally invasive access.
As an interdisciplinary team of experts in neuroscience, neurotechnology, human-machine interfaces, robotics, and ethics, we collaborate to accomplish the following objectives: (i) Push the borders of technology by creating a wearable supernumerary robotic limb (SRL), and non-invasive interfaces with multimodal sensory feedback to control multiple limbs or objects; (ii) Understand the cognitive and neural mechanisms underlying movement augmentation; (iii) Apply movement augmentation to extend a surgeon capabilities and autonomy, develop manipulation with a wearable supernumerary robotic arm and the two hands as well as a 3-hands computer interface; (iv) Evaluate the ethical and safety aspects of movement augmentation.
If successful, NIMA will take human motor capacities to a new level and allow humans to perform tasks that are impossible to perform with the natural limbs alone. The reach of such movement augmentation is far, with applications ranging from work in industrial environments (e.g. overhead work in aircraft assembly), and surgical environments, to everyday applications where subjects can control their smartphone apps while simultaneously using their hands. For instance, laparoscopic surgeons could become able to perform surgical procedures with three hands allowing them to carry out tasks that require a skilled synchronisation of the manipulation of three surgical instruments that currently cannot be realised with minimally invasive access.
As an interdisciplinary team of experts in neuroscience, neurotechnology, human-machine interfaces, robotics, and ethics, we collaborate to accomplish the following objectives: (i) Push the borders of technology by creating a wearable supernumerary robotic limb (SRL), and non-invasive interfaces with multimodal sensory feedback to control multiple limbs or objects; (ii) Understand the cognitive and neural mechanisms underlying movement augmentation; (iii) Apply movement augmentation to extend a surgeon capabilities and autonomy, develop manipulation with a wearable supernumerary robotic arm and the two hands as well as a 3-hands computer interface; (iv) Evaluate the ethical and safety aspects of movement augmentation.
Various body interfaces for movement augmentation were developed and evaluated. Our studies show that controlling three end-effectors simultaneously using body interfaces is intuitive, requires little training and results in excellent performance which can be comparable or better than when working with a partner. A wearable SRL based on a light commercial manipulator was developed, tested and its control with body interfaces for tri-manipulation evaluated. Furthermore, the wearable SRL was adapted to be fixed on a wheelchair or on a mobile industrial stool while following the body’s movement. Towards applications in robot-assisted surgery, an ergonomic foot interface with four degrees-of-freedom (dof) was evaluated with regard to the control of a soft endoscope and additional grasping with surgical tools. A finger interface with haptic feedback for control of mobile phones, computers and SRLs was developed and tested.
To provide supplementary sensory feedback from a supernumerary effector (SE), a wearable sensory feedback suit able to provide vibro- and electrotactile stimulation with high versatility and large bandwidth was developed and validated. Supplementary sensory feedback improved control in a trimanual task, was more effective (in the short term) when it carried information concerning end-effector Cartesian position than joint angle and produced better control performance than the inherent contact feedback of a wearable SRL alone. An experimental protocol to induce and assess embodiment of SRLs was designed and we collected preliminary data on the modulation of peripersonal space after SRL use. Finally, we tested a new feedback strategy with a closed-loop approach, based on electrotactile stimulation to allow participants to control an SRL in the absence of vision.
For the neural interface we investigated whether subjects can independently control individual motor units from a single muscle as a basis to control additional dof. In our studies, participants did not achieve continuous independent modulation of individual motor units’ firing rates or flexible recruitment of units. As an alternative control signal, we investigated high-frequency activity of motor units. Our findings indicate that high-frequency activity can be modulated by mental tasks and correlates with EEG oscillations suggesting that it reflects (at least partially) cortical oscillations. Regarding the muscle interface, we found limited potential for using the wrist’s musculoskeletal system for controlling SEs.
The development of technologies for human movement augmentation raises ethical and safety questions, both at the scale of individuals (safety of users sharing their workspace with an SRL or being impacted by movement augmentation abilities) and at the scale of society and humanity (ethics of movement augmentation). Regarding workspace safety, a risk analysis was conducted with the NIMA SRL platforms, a set of hard- and software measures were proposed to ensure user safety and a framework to analyse workspace safety of SRLs was proposed and validated in controlled laboratory experiments. In parallel, ethical reflections on issues were developed, information on potential real-life interactions enabled by NIMA was collected by an online survey and ethical guidelines on movement augmentation technology were proposed.
Findings of NIMA were published in scientific journals and conference proceedings, including many publications on collaborations involving multiple partners of the consortium. The consortium organized several symposia on movement augmentation at major international conferences, increasing the visibility of the project and connecting with major international players and projects in augmentation. Results were also disseminated to the general public via the NIMA website as well as Tecnalia’s website and LinkedIn profile. Results were exploited in training of students and young researchers, by contributing to the EU’s TechEthos cluster on ethical challenges of emerging technologies, through the EU’s Innovation Radar and by a one-day exploitation workshop. With the low technology readiness level of this FET Open project, most products and service creation are foreseen for later exploitation activities.
To provide supplementary sensory feedback from a supernumerary effector (SE), a wearable sensory feedback suit able to provide vibro- and electrotactile stimulation with high versatility and large bandwidth was developed and validated. Supplementary sensory feedback improved control in a trimanual task, was more effective (in the short term) when it carried information concerning end-effector Cartesian position than joint angle and produced better control performance than the inherent contact feedback of a wearable SRL alone. An experimental protocol to induce and assess embodiment of SRLs was designed and we collected preliminary data on the modulation of peripersonal space after SRL use. Finally, we tested a new feedback strategy with a closed-loop approach, based on electrotactile stimulation to allow participants to control an SRL in the absence of vision.
For the neural interface we investigated whether subjects can independently control individual motor units from a single muscle as a basis to control additional dof. In our studies, participants did not achieve continuous independent modulation of individual motor units’ firing rates or flexible recruitment of units. As an alternative control signal, we investigated high-frequency activity of motor units. Our findings indicate that high-frequency activity can be modulated by mental tasks and correlates with EEG oscillations suggesting that it reflects (at least partially) cortical oscillations. Regarding the muscle interface, we found limited potential for using the wrist’s musculoskeletal system for controlling SEs.
The development of technologies for human movement augmentation raises ethical and safety questions, both at the scale of individuals (safety of users sharing their workspace with an SRL or being impacted by movement augmentation abilities) and at the scale of society and humanity (ethics of movement augmentation). Regarding workspace safety, a risk analysis was conducted with the NIMA SRL platforms, a set of hard- and software measures were proposed to ensure user safety and a framework to analyse workspace safety of SRLs was proposed and validated in controlled laboratory experiments. In parallel, ethical reflections on issues were developed, information on potential real-life interactions enabled by NIMA was collected by an online survey and ethical guidelines on movement augmentation technology were proposed.
Findings of NIMA were published in scientific journals and conference proceedings, including many publications on collaborations involving multiple partners of the consortium. The consortium organized several symposia on movement augmentation at major international conferences, increasing the visibility of the project and connecting with major international players and projects in augmentation. Results were also disseminated to the general public via the NIMA website as well as Tecnalia’s website and LinkedIn profile. Results were exploited in training of students and young researchers, by contributing to the EU’s TechEthos cluster on ethical challenges of emerging technologies, through the EU’s Innovation Radar and by a one-day exploitation workshop. With the low technology readiness level of this FET Open project, most products and service creation are foreseen for later exploitation activities.
NIMA has made significant progress in the development and evaluation of efficient body interfaces for trimanual manipulation and as a potential interface to assist and extend surgical manipulation. The developed feedback suit offers a novel integrated, flexible and powerful tool to provide different types of supplementary feedback from SEs, able to convey information that can be used for real-time tracking of SEs. This allowed to systematically test and compare different strategies for providing feedback and determine their specific usefulness. A wearable SRL platform was designed, implemented and tested. In addition, an adaptable framework to analyse workspace safety of SRLs and ethical guidelines on movement augmentation were proposed. These advancements, together with the pioneering work on muscle and neural interfaces, are highly innovative, as also underlined by the selection of NIMA’s sensory feedback suit as an innovation by the EU’s innovation radar. The NIMA consortium has thus secured an excellent international position in the emerging field of movement augmentation.