Periodic Reporting for period 1 - FEEL-XR (Feel-through Haptic Feedback for Augmented and Virtual Reality)
Período documentado: 2023-06-01 hasta 2024-11-30
Virtual and Augmented Reality (VR and AR), often summarized as Extended Reality (XR), is moving from a niche product to a mainstream market. Thanks to recent advances in wearable sensing and display technologies, powerful VR and AR headsets are commercially available, even for the general public, and promise to become even more miniaturized and cheaper in the coming years. A myriad of important application cases, ranging from gaming, video-conferencing or shopping to immersive design and engineering or to tele-surgery and virtual training, demonstrate the large innovation potential of XR. However, today’s technologies and applications for XR predominantly focus on the visual channel alone. The visual appearance of virtual elements can be rendered at a stunning fidelity, and real-world objects can be visually augmented in powerful and very realistic ways. In contrast, rendering the haptic feel of objects in XR is still mostly unaddressed, although it is common knowledge that haptic perception is an essential part of how humans experience the world, hence important for realism and immersion.
Some first commercial solutions include haptic actuators in handheld VR controllers or wearable VR gloves. However, the rigid mechanical actuators with rather large form factors come with challenges regarding (1) deployment of rigid technolgoies in the curved and soft human body, (2) inhibiting the natural tactile sensing of the skin, (3) limited spatial resolution, (4) power consumption, and (5) customization for individuals. Adding computer-generated, real-time haptic feedback to objects and surfaces through an ergonomic wearable device is an important gap that needs to be addressed to move Extended Reality to a new level of immersion.
This project set out to mature the crucial components needed for tactile feedback in Augmented and Virtual Reality that can dynamically augment and alter the haptic feel of real-world objects, surfaces, and the human body. We contribute a micron-thin wearable interface that can be ergonomically worn on diverse (curved, soft) body sites, for which it can be easily customized in size and shape. It allows the user to naturally feel physical objects or surfaces across the interface while at the same time delivering computer-generated tactile output at a high resolution, with high resolution and low power consumption. The technical feasibility and user experience of the technology has already been demonstrated and quantified in the PI’s ERC grant InteractiveSkin. The crucial activity around which FEEL-XR revolves is exploring and validating innovative application cases. This requires two essential technical activities to be performed: (1) development and validation of a refined prototype with integrated sensing/action loop and (2) fine-tuning of closed-loop haptic stimulation parameters.
Some first commercial solutions include haptic actuators in handheld VR controllers or wearable VR gloves. However, the rigid mechanical actuators with rather large form factors come with challenges regarding (1) deployment of rigid technolgoies in the curved and soft human body, (2) inhibiting the natural tactile sensing of the skin, (3) limited spatial resolution, (4) power consumption, and (5) customization for individuals. Adding computer-generated, real-time haptic feedback to objects and surfaces through an ergonomic wearable device is an important gap that needs to be addressed to move Extended Reality to a new level of immersion.
This project set out to mature the crucial components needed for tactile feedback in Augmented and Virtual Reality that can dynamically augment and alter the haptic feel of real-world objects, surfaces, and the human body. We contribute a micron-thin wearable interface that can be ergonomically worn on diverse (curved, soft) body sites, for which it can be easily customized in size and shape. It allows the user to naturally feel physical objects or surfaces across the interface while at the same time delivering computer-generated tactile output at a high resolution, with high resolution and low power consumption. The technical feasibility and user experience of the technology has already been demonstrated and quantified in the PI’s ERC grant InteractiveSkin. The crucial activity around which FEEL-XR revolves is exploring and validating innovative application cases. This requires two essential technical activities to be performed: (1) development and validation of a refined prototype with integrated sensing/action loop and (2) fine-tuning of closed-loop haptic stimulation parameters.
While it is easy to dynamically change the visual experience of an object in AR or VR, it remains challenging to change its felt material experience. To demonstrate the innovation potential for interactive applications, the already existing tactile output interface was augmented with sensors that capture user activity. We have realized an embedded system with custom-built firmware that realizes a real-time sensing/action loop, where a user’s input is closely coupled to corresponding haptic feedback. This was a necessary prerequisite for providing much richer haptic experiences that are closely synchronized to the user’s tactile activity, such as pressing on a surface. Furthermore, we have extended the platform to feature two paired output units, to be compatible with the demands of grasping.
This technical advancement has allowed us to realize two main technical achievements:
We have presented an electro-tactile illusion of softness. By rendering electrotactile grains that are closely synchronized to the amount of pressure a user is applying on a surface, our system can modify the perceived softness of the surface. By controlling software parameters, several levels of softness can be achieved using the same hardware, in real time. Furthermore, due to the high resolution of tactile output, softness can be rendered in different shapes.
Our second main technical achievement involves tactile rendering of virtual objects that are grasped in-between two fingers in VR. Two stimulation units, one on the thumb and another on the index finger, are employed to render time-synchronized tactile cues. This approach opens up new opportunities for rendering of tactile spatial cues during grasping, such as objects that revolve in-between fingers or elements that pop out and pop in.
Taken together, these techniques enable new immersive methods using high-resolution tactile rendering (both in time and in space) in order to add to the realism of interaction with virtual and real objects in XR. In controlled experiments with users, we have characterized a range of tactile experiences that can be rendered with the platform thanks to closed-loop control.
This technical advancement has allowed us to realize two main technical achievements:
We have presented an electro-tactile illusion of softness. By rendering electrotactile grains that are closely synchronized to the amount of pressure a user is applying on a surface, our system can modify the perceived softness of the surface. By controlling software parameters, several levels of softness can be achieved using the same hardware, in real time. Furthermore, due to the high resolution of tactile output, softness can be rendered in different shapes.
Our second main technical achievement involves tactile rendering of virtual objects that are grasped in-between two fingers in VR. Two stimulation units, one on the thumb and another on the index finger, are employed to render time-synchronized tactile cues. This approach opens up new opportunities for rendering of tactile spatial cues during grasping, such as objects that revolve in-between fingers or elements that pop out and pop in.
Taken together, these techniques enable new immersive methods using high-resolution tactile rendering (both in time and in space) in order to add to the realism of interaction with virtual and real objects in XR. In controlled experiments with users, we have characterized a range of tactile experiences that can be rendered with the platform thanks to closed-loop control.
The results of this project have contributed new concepts for rendering high-resolution tactile output in very thin and ergonomic wearable devices, and specifically for rendering rich material experiences during object contact: virtual softness when pressing on a surface and virtual object geometry while grasping in XR. Through implementation (software, firmware, hardware), interactive applications and results of user studies, we have validated the platform’s ability to add to the realism of object interaction in XR environments.
In addition, the project has prepared the commercial exploitation of the innovation, by realizing technical demonstrators for several use cases, by securing IP, by developing a market analysis, by determining viable business models, and by exploring avenues to liaise with commercial partners. These represent the early steps in the commercialization process and have advanced our Technology Readiness Level (TRL) from 3 to 5.
In addition, the project has prepared the commercial exploitation of the innovation, by realizing technical demonstrators for several use cases, by securing IP, by developing a market analysis, by determining viable business models, and by exploring avenues to liaise with commercial partners. These represent the early steps in the commercialization process and have advanced our Technology Readiness Level (TRL) from 3 to 5.