Computational Modelling of Tangible Objects on Multisensory Interfaces

Current interactive systems provide realistic audio-visual feedback while haptic feedback is limited even though haptic interactions with devices are essential and touch plays a large role to convey immersive sensations. Thus, usable high-fidelity haptic rendering has the potential to improve interactions with interactive interfaces and future metaverses. Knowledge has started emerging on haptic rendering and multisensory perception of tangible objects. Still, combining multiple techniques to stimulate the full bandwidth of tactile sensitivity remains a hard challenge and little is known about the perceptual integration of tactile cues such as friction, temperature, or vibrations. Thus, there is the intricate twofold challenge of developing new haptic techniques based on models of cognitive multisensory integration whose investigation requires advanced stimulation techniques. Thus, the originality of TANGO is to address the problem of multisensory rendering of tangible objects from a holistic view by considering simultaneously multi-dimensional haptic techniques, tactile acuity across its psychophysical dimensions, and mechanisms of multisensory integration. Specifically, TANGO aims to 1/ characterize typical gestures and contact mechanics during interaction with real objects. 2/ understand how gestures and the dimension of haptic feedback influence perception of simulated tactile objects 3/ create a computational model of multisensory integration that outputs perceived realism during interaction with objects. Studies will also consider two main population factors: age and sensory impairment. Finally, the outcome from 1-3 will serve to design sensorily rich tangible 3D elements on displays, in virtual reality, and on maps for visually impaired. By a strongly interdisciplinary approach combining haptics, computer science, and neuroscience, TANGO will optimize haptic rendering in multisensory interactive interfaces to enhance its use in human-computer interactions.

Research activities have studied tactile perception during tactile perception with real objects, especially the richness of elicited sensations (e.g. rough, warm, soft). Two types of interactions were tested: musical gestures on strings that exhibit subtle differences, and free manipulation of everyday objects. To test perception, participants in the experiments performed either a task in which they had to spot the discrepant object and one in which they freely described their tactile sensations. The respective goals of the tasks were to understand which tactile dimensions matter when differences are tiny and which ones are more often spontaneously reported after interaction with everyday objects.
We also aim to render rich tactile cues with actuators and understand their integration. To that end, we have produced a combination of vibrotactile and compliant haptic cues. These studies highlighted interesting phenomena of sensory dominance. For example, alterations in low vibrotactile frequencies are most perceived, added vibration can change the roughness of soft surfaces.
Importantly, we have tested perception of rich haptic cues alongside visual cues in discrimination and matching tasks. Experiments were performed either with the visuals on a screen (non-colocated) or in virtual reality (collocated). Results show that multisensory integration is dependent on the task and the type of haptic interaction. These studies highlighted phenomena that provide evidence of sensory mechanisms going beyond the classic assessment of the stimulus reliability. To understand further how the sensory weighting occurs, we developed a new method for performing EEG experiments in conditions of active touch, which enables an insight into the brain response to missing or discrepant tactile information.
We are also testing new methods to investigate sensory integration of distinct haptic cues. One of them is the use of the temporal binding paradigm, which shows how cue integration alters human perception of time interval. We are also developing a second novel paradigm, cue extraction, which stems from vision research and allows to understand how difficult it is for humans to spot combinations of features in a set of items. We believe that this paradigm will enable to achieve improved understanding of which tactile cues are integrated and which one a processed in isolation.

TANGO has already produced several results that are pushing forward the related scientific questions. We have shown that expertise improves tactile perception through the spontaneous development of more complex gestures that are more efficient at extracting meaningful tactile information. Moreover, the main tactile dimension experienced during the task depended on expertise due to prior knowledge and more expert interaction.

Several studies within TANGO studied roughness, and two major outcomes have emerged. First, we show that roughness is most present in subjective reports of haptic sensation elicited by everyday objects even when the objects do not exhibit specificities in this dimension. Second, in a visio-haptic task, incongruence would be more easily detected when the visual sample is rougher than the haptic one. This phenomenon is intriguing and suggest that the visual and haptic stimuli are unevenly weighted during sensory integration. Still, this phenomenon is not yet fully understood and requires further investigation.

A major result of TANGO is that ambidextrous participants are more prone to visio-haptic illusion than right-handed ones. This finding might suggest a more multisensory wiring in ambidextrous people, which might alter the sensory weight that they put on the distinct sensory modalities compared to more lateralized people. This finding requires further exploration potentially with brain imaging to unravel the brain mechanisms at play.

A methodological outcome is a novel method for EEG imaging during active haptic exploration. This method was awarded the “Best Paper” award at the conference World Haptics 2025, which shows a keen interest in the haptic community for this new tool. We hope to use it in order to explore deeper the mechanisms underlying the most intriguing psychophysical results of TANGO.

Periodic Reporting for period 1 - TANGO (Computational Modelling of Tangible Objects on Multisensory Interfaces)

Descargar Descargar el contenido de la página