Periodic Reporting for period 2 - TACTILITY (TACTIle feedback enriched virtual interaction through virtual realITY and beyond)
Okres sprawozdawczy: 2021-01-01 do 2022-09-30
TACTILITY aims to substantially increase the quality of immersive VR experience used locally or remotely, based on transcutaneous electro-tactile stimulation delivered through electrical pulses with high resolution spatio-temporal distribution. Significant development of technologies for stimulation, textile-based multi-pad electrodes and tactile sensation e-skin, coupled with ground-breaking research of perception of elicited tactile sensations in VR, were needed.
The key novelty is in the combination of:
1) ground-breaking research of electrotactile stimuli that evoke tactile sensations,
2) advanced hardware, that integrates the novel electrotactile stimulation system and artificial electronic patches with smart textile technologies and VR control devices in a wearable mobile system,
3) novel software, handling real-time encoding and transmission of tactile information from virtual objects in VR, as well as from the distant tactile sensors placed on robotic or human hands.
Proposed research and innovation action has resulted in the development of all the physical components (electrotactile glove with stimulator and kinematic glove) and the software components (haptic effects library, electrotactile calibration methods and VR haptic effects authoring and rendering pipeline) which form the core of the new TACTILITY technology to provide electrotactile feedback in VR and increase immersiveness and embodiement. This technology will lead to the next generation of interactive systems with higher quality experience for both local and tele-manipulation applications. TACTILITY will enable high fidelity experience through low-cost, user friendly, wearable and mobile technology.
The key novelty is in the combination of:
1) ground-breaking research of electrotactile stimuli that evoke tactile sensations,
2) advanced hardware, that integrates the novel electrotactile stimulation system and artificial electronic patches with smart textile technologies and VR control devices in a wearable mobile system,
3) novel software, handling real-time encoding and transmission of tactile information from virtual objects in VR, as well as from the distant tactile sensors placed on robotic or human hands.
Proposed research and innovation action has resulted in the development of all the physical components (electrotactile glove with stimulator and kinematic glove) and the software components (haptic effects library, electrotactile calibration methods and VR haptic effects authoring and rendering pipeline) which form the core of the new TACTILITY technology to provide electrotactile feedback in VR and increase immersiveness and embodiement. This technology will lead to the next generation of interactive systems with higher quality experience for both local and tele-manipulation applications. TACTILITY will enable high fidelity experience through low-cost, user friendly, wearable and mobile technology.
The main result achieved during TACTILITY project are the Gamma demonstrators. This is the result of the technical efforts dedicated mainly in WP2 to 7 to the development of the different components of the system and to their validation and integration into the demonstrators. Several versions of the stimulator, electrodes and gloves have been manufactured and distributed between partners. Also, a new version of the kinematic glove for hand tracking has been designed. Similarly, several efforts have been devoted to the development of the SW modules for WP3 and WP4. The final system including the Gamma electrotactile feedback and kinematic glove and the SW modules necessary (calibration, haptic effects authoring and rendering of electrotactile effects) and scenarization in the VR environment have been developed and integrated into the Gamma demonstrator.
Alpha to Gamma prototypes allowed starting the research on electro-tactile feedback in WP3, where results regarding psychometric assessment, contact combinations, moving and dynamic patterns to mimic different tactile perceptions and tactile icons have been obtained. This allowed conveying intuitive tactile feedback with high resolution and communicating multiple variables simultaneously. All these results were integrated in the library of high-fidelity tactile sensations or haptic library, whose development was completed for the Gamma prototype.
Alpha to Gamma prototypes also allowed the integration with the VR environment in WP4. The final version of the Unity stimulator control, with the electrotactile rendering pipeline has been finished and along with the Manus Core SDK allowed the realization of the final prototype integrating kinematic information and electrotactile feedback. Research activities were focused mainly on enabling the interaction with virtual objects using the kinematic information provided by the glove and conducting experiments in virtual reality using electrotactile feedback. The results showed good acceptability of the TACTILITY technology and a remarkable performance of electrotactile feedback for contact rendering.
The work on WP5 was focused on one hand on the technical work to enhance virtual objects with tactile feedback in order to increase the realism of interactions, and on the other hand on the preparation of the experiments, methods and protocols to study the psychological and subjective variables that have an impact on the virtual touch and embodiment experience. Feedback after testing from participants was positive about the immersiveness and psychologists considered it a useful environment for clinical purposes and willing to introduce it in their regular practice.
The work on WP6 related to the telemanipulation scenario allowed implementing the final electronic skin prototype with 64 sensors and implementing passive and active touch experiments in order to provide tactile feedback to the human user. Research allowed studying the relation between the applied force on the skin patch and the measured sensor charge, and with that information process tactile data and their encoding into stimulation parameters for electrotactile feedback. The Gamma electrotactile glove placed on the user hand allowed the remote user receiving that feedback. The work related to the communication of that information to a remote receiver allowed assuring low-latency perception for an optimal closed-loop control.
Alpha to Gamma prototypes and have been used to show to selected partners in the industrial and scientific fields and the general public the capabilities of the TACTILITY system in a VR environment on the 2 specific workshops organized and in one fair, with over 400 people seeing or testing the system. Other dissemination actions have been launched through mass media and social networks to the general public, and through the preparation of webinars and/or workshops to the scientific and academic stakeholders, in a remote manner during the COVID-19 pandemic. The consortium has generated 11 publications in journals and 16 in conferences, with over 10 more in preparation, also a patent has been filled.
Alpha to Gamma prototypes allowed starting the research on electro-tactile feedback in WP3, where results regarding psychometric assessment, contact combinations, moving and dynamic patterns to mimic different tactile perceptions and tactile icons have been obtained. This allowed conveying intuitive tactile feedback with high resolution and communicating multiple variables simultaneously. All these results were integrated in the library of high-fidelity tactile sensations or haptic library, whose development was completed for the Gamma prototype.
Alpha to Gamma prototypes also allowed the integration with the VR environment in WP4. The final version of the Unity stimulator control, with the electrotactile rendering pipeline has been finished and along with the Manus Core SDK allowed the realization of the final prototype integrating kinematic information and electrotactile feedback. Research activities were focused mainly on enabling the interaction with virtual objects using the kinematic information provided by the glove and conducting experiments in virtual reality using electrotactile feedback. The results showed good acceptability of the TACTILITY technology and a remarkable performance of electrotactile feedback for contact rendering.
The work on WP5 was focused on one hand on the technical work to enhance virtual objects with tactile feedback in order to increase the realism of interactions, and on the other hand on the preparation of the experiments, methods and protocols to study the psychological and subjective variables that have an impact on the virtual touch and embodiment experience. Feedback after testing from participants was positive about the immersiveness and psychologists considered it a useful environment for clinical purposes and willing to introduce it in their regular practice.
The work on WP6 related to the telemanipulation scenario allowed implementing the final electronic skin prototype with 64 sensors and implementing passive and active touch experiments in order to provide tactile feedback to the human user. Research allowed studying the relation between the applied force on the skin patch and the measured sensor charge, and with that information process tactile data and their encoding into stimulation parameters for electrotactile feedback. The Gamma electrotactile glove placed on the user hand allowed the remote user receiving that feedback. The work related to the communication of that information to a remote receiver allowed assuring low-latency perception for an optimal closed-loop control.
Alpha to Gamma prototypes and have been used to show to selected partners in the industrial and scientific fields and the general public the capabilities of the TACTILITY system in a VR environment on the 2 specific workshops organized and in one fair, with over 400 people seeing or testing the system. Other dissemination actions have been launched through mass media and social networks to the general public, and through the preparation of webinars and/or workshops to the scientific and academic stakeholders, in a remote manner during the COVID-19 pandemic. The consortium has generated 11 publications in journals and 16 in conferences, with over 10 more in preparation, also a patent has been filled.
Tactility has achieved several goals. We have demonstrated the feasibility of high-resolution stimulation using matrix electrodes, showing that subjects can identify many stimulation points activated sequentially (single contact) and simultaneously (multi-contact), recognize changes in contact size, and perceive moving sensations very reliably. We have investigated and developed approaches that can be used for faster calibration of the system, a critical step toward successful practical application. The guidelines were also proposed on how to map the object properties into stimulation profiles (e.g. modulating frequency to indicate texture) and it has been demonstrated that high-fidelity electrotactile stimulation can improve the feeling of immersion when interacting with objects in VR.
We also have addressed two critical aspects of telemanipulation enhanced with tactile sense. We have experimentally investigated how communication disturbances on remote closed-loop control affect the subjects’ ability to exploit tactile feedback when controlling a remote dynamic system, this will inform the design of more effective systems for telemanipulation with tactile feedback. We have also developed a complete pipeline for telemanipulation in which a remote robotic hand conveys high-density tactile feedback to the user. The e-skin recognizes contact events, which are then conveyed to the teleoperator by delivering electrotactile stimulation through a high-density Tactility glove. Tests shown that the subjects could successfully utilize the feedback to recognize different mechanical stimuli delivered to the remote hand.
We also have addressed two critical aspects of telemanipulation enhanced with tactile sense. We have experimentally investigated how communication disturbances on remote closed-loop control affect the subjects’ ability to exploit tactile feedback when controlling a remote dynamic system, this will inform the design of more effective systems for telemanipulation with tactile feedback. We have also developed a complete pipeline for telemanipulation in which a remote robotic hand conveys high-density tactile feedback to the user. The e-skin recognizes contact events, which are then conveyed to the teleoperator by delivering electrotactile stimulation through a high-density Tactility glove. Tests shown that the subjects could successfully utilize the feedback to recognize different mechanical stimuli delivered to the remote hand.