Computational Design of Multimodal Tactile Feedback within Immersive Environments

Project Information

ADVHANDTURE

Grant agreement ID: 101088708

DOI

10.3030/101088708

EC signature date 5 April 2023

Start date 1 October 2023

End date 30 September 2028

Funded under

• European Research Council (ERC)

Total cost

€ 1 999 750,00

EU contribution

€ 1 999 750,00

Coordinated by

INSTITUT NATIONAL DES SCIENCES APPLIQUEES DE RENNES
France

Periodic Reporting for period 1 - ADVHANDTURE (Computational Design of Multimodal Tactile Feedback within Immersive Environments)

Reporting period: 2023-10-01 to 2026-03-31

Summary of the context and overall objectives of the project

These last years have seen many progresses in haptic technologies for providing various tactile stimuli through innovative haptic interfaces that users could wear across all their body. However, despite these advances, virtual reality systems are today still facing a fundamental lack of rich tactile feedback able to convey compelling sensations. ADVHANDTURE aims at introducing an original computational approach for handling multimodal tactile feedback within immersive environments through the design of computational models dedicated to immersive tactile haptics. The hand -by its fundamental role in the sense of touch- is at the heart of the adventure across the tactile exploration of immersive environments. Three main scientific objectives are targeted in the project: 1) the design of rendering models for combining in a spatially-coherent manner tactile modalities within immersive environments, 2) the development of efficient algorithms for the temporal integration of various tactile stimuli within immersive environments, 3) the design of 3D interaction paradigms for combining multimodal tactile stimuli with multi-sensory integration.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Given the scientific objectives of the project, different research directions have been initiated, following the path of the 3D interaction loop relating the user to the virtual environment he is working with.

Design of innovative haptic interfaces. As our main goal is to understand how to combine tactile stimuli on the hand, a significant part of our research activities in this first period was dedicated to the
design of haptic interfaces in order to generate various types of haptic stimuli. We explored several types of stimuli, from vibrations to ultrasounds, combined with various shapes of interfaces.

Design of perception studies. The main objective of our project is to improve our understanding of the human perception about the combination of haptic stimuli applied on the hand. We first started to
study how we could combine the feedback of a real object with a haptic stimulus. For that purpose, we conducted different experiments where we evaluated how real textures could be enhanced with
vibrotactile stimuli and if we could observe differences in terms of perception between Augmented Reality and Virtual Reality environments. Regarding the combination of haptic stimuli and their effects on the hand, we also explored how we could use vibrotactile stimuli on the arm and their effects on the hand perception.

Design of 3D rendering techniques. Given our innovative interfaces and the perception studies previously conducted, we proposed several 3D rendering techniques for 3D interaction within
virtual environments. For ultrasound mid-air haptics, we designed a specific rendering for bimanual interactions, providing a wider space for 3D manipulation in Virtual Reality. We also combined our

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Fundamentally, ADVHANDTURE will significantly contribute to pave the way for computational haptic models able to render a wide variety of tactile sensations within immersive environments, a still under-explored challenge of computer graphics, virtual reality and haptics fields. The ground-breaking nature of the proposal relies on its multidisciplinary aspect, gathering knowledge of different scientific domains to leverage synergistic outcomes that would be beneficial to the different research communities. It will also enrich our understanding of rendering haptic sensations through the use of combined tactile modalities, as well as the role of haptic feedback during 3D interaction with virtual worlds.