Periodic Reporting for period 1 - TRIMANUAL (TRIMANUAL coordination assistance for hemiplegic stroke individuals)
Reporting period: 2019-07-01 to 2021-06-30
Artificial supernumerary limbs (SL) that assist their user with minimal cognitive effort and can be seamlessly controlled both independently and in combination with their natural limbs could revolutionise industry and surgery by allowing individuals to achieve actions that are otherwise impossible on their own. This could also have a profound impact on the life of disabled people by providing a mechanism capable of both replacing/restoring the functionality of the impaired limb without the need for amputation. The study of robotic limbs is well established, however, to date the capability of human users to use such devices for SLS has not been studied, nor has there been an interface developed that can be used intuitively.
The development of an intuitive and easy to control interface for SLs would allow the benefits of supernumerary limbs to go from the lab to reality. They would allow a single individual to control more DoFs than they currently can so that they could perform two-person assembly tasks or surgical tasks without the need for a second individual and the communication issues that that can commonly lead to. However, the ultimate benefits possible for such an approach is limited by the overall capability of human users to augment their capability and control additional degrees of freedom. Such knowledge is therefore necessary for further developments within this field.
Based on this, the project possesses the following overall objective:
O1 To develop a non-invasive human-machine interface enabling rich command and feedback behaviours for everyday SL
O2 To determine a control framework capable of executing independent and trimanual operations.
O3 To implement a SL enabling people with upper-limb disabilities to execute common ADLs.
At its conclusion the project has
- Shown that a 4-DoF foot interface can be used for the real-time control of supernumerary limbs
- Developed a novel 3-DoF haptic feedback scheme with which to communicate the task space position of the supernumerary limb to a user
- Shown that human users can execute independent and trimanual operations using direct control
- Identified that human users are more suited to performing trimanual actions when there is coupling rather than independence, and that with limited learning that their performance can exceed that of two users performing the same task
However, because of the ongoing COVID-19 pandemic the project was not able to yet test an integrated version of this scheme onto individuals with upper-limb disabilities, however, this will be performed as possible in the near future.
The development of an intuitive and easy to control interface for SLs would allow the benefits of supernumerary limbs to go from the lab to reality. They would allow a single individual to control more DoFs than they currently can so that they could perform two-person assembly tasks or surgical tasks without the need for a second individual and the communication issues that that can commonly lead to. However, the ultimate benefits possible for such an approach is limited by the overall capability of human users to augment their capability and control additional degrees of freedom. Such knowledge is therefore necessary for further developments within this field.
Based on this, the project possesses the following overall objective:
O1 To develop a non-invasive human-machine interface enabling rich command and feedback behaviours for everyday SL
O2 To determine a control framework capable of executing independent and trimanual operations.
O3 To implement a SL enabling people with upper-limb disabilities to execute common ADLs.
At its conclusion the project has
- Shown that a 4-DoF foot interface can be used for the real-time control of supernumerary limbs
- Developed a novel 3-DoF haptic feedback scheme with which to communicate the task space position of the supernumerary limb to a user
- Shown that human users can execute independent and trimanual operations using direct control
- Identified that human users are more suited to performing trimanual actions when there is coupling rather than independence, and that with limited learning that their performance can exceed that of two users performing the same task
However, because of the ongoing COVID-19 pandemic the project was not able to yet test an integrated version of this scheme onto individuals with upper-limb disabilities, however, this will be performed as possible in the near future.
The following work (and associated results) has been performed throughout the project
- Interface development
o 4 DoF foot-based control interface was shown to be suited for trimanual applications. This work was published as
Yanpei Huang, Jonathan Eden, Lin Cao, Etienne Burdet, and Soo Jay Phee. "Tri-manipulation: An evaluation of human performance in 3-handed teleoperation." IEEE Transactions on Medical Robotics and Bionics 2, no. 4 (2020): 545-548.
o Motor unit recruitment order was investigated as a mechanism for SL command
Initial results suggest that human subjects lack sufficient voluntary control for real-time use. This work is intended to be submitted in eLife and currently available as
Mario Bräcklein, Jaime Ibáñez, Deren Yusuf Barsakcioglu, Jonathan Eden, Etienne Burdet, Carsten Mehring and Dario Farina. ." The control and training of single motor units in isometric tasks are constrained by a common synaptic input signal." https://www.biorxiv.org/content/early/2021/08/04/2021.08.03.454908
o Proprioceptive feedback encoding system developed and shown to be intuitive for human users
Evaluation of variance in frequency and amplitude for electrotactile feedback showed that subjects could best understand simultaneous modification of both factors. This work has been accepted to be presented in the 2021 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).
The scheme has since been expanded to 3 DoF with a fully modular vibrotactile system.
- Understanding of human capability
o Ability of human subjects to adapt to delay studied
Was found that human subjects can adapt to smaller delays and recognise delay even for small delay values. This work was presented in the 2021 WorldHaptics conference and published as
Ivanova, Ekaterina, Jonathan Eden, Silei Zhu, Gerolamo Carboni, Aaron Yurkewich, and Etienne Burdet. "Short time delay does not hinder haptic communication benefits." IEEE Transactions on Haptics (2021).
o Impact of trimanual control versus bimanual control evaluated
Almost no loss of performance to extend from bimanual to trimanual when the actions of the hands are independent.
When hands are coupled there is a loss in performance, however, characteristics of hand movement such as smoothness remain unchanged
This was published as
Yanpei Huang, Jonathan Eden, Lin Cao, Etienne Burdet, and Soo Jay Phee. "Tri-manipulation: An evaluation of human performance in 3-handed teleoperation." IEEE Transactions on Medical Robotics and Bionics 2, no. 4 (2020): 545-548.
o Comparison of trimanual and dyadic performance across different kinds of tasks
Dyad performance is preferred and outperforms trimanual performance in all tasks when subjects have not trained in trimanual behaviours. This work was published as
Alessia Noccaro, Jonathan Eden, Giovani Di Pino, Domenico Formica, and Etienne Burdet. "Human performance in three-hands tasks." Scientific Reports 11, no. 1 (2021): 1-8.
After training, trimanual performance exceeds initial dyad and trimanual performance. Subjects still have a preference for working in a dyad. This work is accepted to be presented in the 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC).
- Interface development
o 4 DoF foot-based control interface was shown to be suited for trimanual applications. This work was published as
Yanpei Huang, Jonathan Eden, Lin Cao, Etienne Burdet, and Soo Jay Phee. "Tri-manipulation: An evaluation of human performance in 3-handed teleoperation." IEEE Transactions on Medical Robotics and Bionics 2, no. 4 (2020): 545-548.
o Motor unit recruitment order was investigated as a mechanism for SL command
Initial results suggest that human subjects lack sufficient voluntary control for real-time use. This work is intended to be submitted in eLife and currently available as
Mario Bräcklein, Jaime Ibáñez, Deren Yusuf Barsakcioglu, Jonathan Eden, Etienne Burdet, Carsten Mehring and Dario Farina. ." The control and training of single motor units in isometric tasks are constrained by a common synaptic input signal." https://www.biorxiv.org/content/early/2021/08/04/2021.08.03.454908
o Proprioceptive feedback encoding system developed and shown to be intuitive for human users
Evaluation of variance in frequency and amplitude for electrotactile feedback showed that subjects could best understand simultaneous modification of both factors. This work has been accepted to be presented in the 2021 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).
The scheme has since been expanded to 3 DoF with a fully modular vibrotactile system.
- Understanding of human capability
o Ability of human subjects to adapt to delay studied
Was found that human subjects can adapt to smaller delays and recognise delay even for small delay values. This work was presented in the 2021 WorldHaptics conference and published as
Ivanova, Ekaterina, Jonathan Eden, Silei Zhu, Gerolamo Carboni, Aaron Yurkewich, and Etienne Burdet. "Short time delay does not hinder haptic communication benefits." IEEE Transactions on Haptics (2021).
o Impact of trimanual control versus bimanual control evaluated
Almost no loss of performance to extend from bimanual to trimanual when the actions of the hands are independent.
When hands are coupled there is a loss in performance, however, characteristics of hand movement such as smoothness remain unchanged
This was published as
Yanpei Huang, Jonathan Eden, Lin Cao, Etienne Burdet, and Soo Jay Phee. "Tri-manipulation: An evaluation of human performance in 3-handed teleoperation." IEEE Transactions on Medical Robotics and Bionics 2, no. 4 (2020): 545-548.
o Comparison of trimanual and dyadic performance across different kinds of tasks
Dyad performance is preferred and outperforms trimanual performance in all tasks when subjects have not trained in trimanual behaviours. This work was published as
Alessia Noccaro, Jonathan Eden, Giovani Di Pino, Domenico Formica, and Etienne Burdet. "Human performance in three-hands tasks." Scientific Reports 11, no. 1 (2021): 1-8.
After training, trimanual performance exceeds initial dyad and trimanual performance. Subjects still have a preference for working in a dyad. This work is accepted to be presented in the 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC).
Progress beyond the state of the art:
- The first quantification of human capability to perform trimanual activities compared to both bimanual and dyadic equivalent tasks
- The development of a 4 DoF foot-based interface real-time control
- The development of a 3 DoF haptic feedback mechanism
Impact:
The development of supernumerary limbs can enable human subjects to perform tasks that they would not otherwise be able to do. However, while supernumerary limbs offer new potential degrees of freedom, their control also possesses the potential for degradation of performance due to its overall complexity. This project has in particular aided the potential impact of supernumerary limbs by developing a taxonomy of trimanipulation and from this determining the forms of tasks at which human users are best capable of exploiting SLs. Based on the results future research will have greater impact by focussing particularly on tasks which exploit either physical coupling or dependency of the limb function such as would be the case during manual assembly or surgical cutting.
The project has also developed new systems to better interface with supernumerary limbs. To go from the lab to household applications, supernumerary limbs will need more intuitive easy to learn interfaces. Accordingly, the development of intuitive haptic feedback mechanisms to provide artificial proprioception of where the device is as well as the development of 4 DoF command interfaces will greatly aid in the future exploitation of SLs by the broader public.
- The first quantification of human capability to perform trimanual activities compared to both bimanual and dyadic equivalent tasks
- The development of a 4 DoF foot-based interface real-time control
- The development of a 3 DoF haptic feedback mechanism
Impact:
The development of supernumerary limbs can enable human subjects to perform tasks that they would not otherwise be able to do. However, while supernumerary limbs offer new potential degrees of freedom, their control also possesses the potential for degradation of performance due to its overall complexity. This project has in particular aided the potential impact of supernumerary limbs by developing a taxonomy of trimanipulation and from this determining the forms of tasks at which human users are best capable of exploiting SLs. Based on the results future research will have greater impact by focussing particularly on tasks which exploit either physical coupling or dependency of the limb function such as would be the case during manual assembly or surgical cutting.
The project has also developed new systems to better interface with supernumerary limbs. To go from the lab to household applications, supernumerary limbs will need more intuitive easy to learn interfaces. Accordingly, the development of intuitive haptic feedback mechanisms to provide artificial proprioception of where the device is as well as the development of 4 DoF command interfaces will greatly aid in the future exploitation of SLs by the broader public.