Periodic Reporting for period 1 - SUN (Social and hUman ceNtered XR)
Periodo di rendicontazione: 2022-12-01 al 2024-05-31
Extended reality (XR) is an emerging technology with promising potential in many fields including health, communication and safety. However, overcoming XR’s limitations in providing authentic interactive social environments is key to further developing its applications.
In the SUN (Social and hUman ceNtered XR) project (https://www.sun-xr-project.eu(si apre in una nuova finestra)) we have identified the current limitations, to make XR real and easily usable, and we are addressing them. The SUN project aims to advance the social interactivity of XR technology by establishing scalable models with sustained and convincing virtual environments. The project will also improve the data-processing capability of wearable devices and develop wearable sensors and tactile interfaces to improve user experience.
SUN will investigate new 3D acquisition techniques which will go beyond capabilities of current laser scanning or photogrammetry solutions. We will leverage on Al to optimize and automatize 3D acquisition and we will use solutions for massive avatar creation.
SUN will define artificial intelligence-based techniques to dynamically and incrementally create links between physical objects and their digital twins.
SUN will explore the design of haptic devices that provide users with a realistic feeling when interacting with virtual objects. We will explore miniaturized mechanical and thermal actuators, EMG decoding solutions, and skin stretching actuators. We will also explore and define techniques based on multimodal interaction integrating gaze and gesture interaction. Interface between humans and the integrated physical and virtual world shall be as much natural as possible.
SUN will explore advanced streaming solutions that allow high quality, in terms of resolution, photo realism, and speed, even in presence of low resource and computational power, by optimizing client- server interaction and balancing computation between the two.
SUN will validate these models in three real-life situations related to three social and human related scenarios: rehabilitation therapy, improved safety and social interaction among workers, and facilitate interaction for disabled users.
In the SUN (Social and hUman ceNtered XR) project (https://www.sun-xr-project.eu(si apre in una nuova finestra)) we have identified the current limitations, to make XR real and easily usable, and we are addressing them. The SUN project aims to advance the social interactivity of XR technology by establishing scalable models with sustained and convincing virtual environments. The project will also improve the data-processing capability of wearable devices and develop wearable sensors and tactile interfaces to improve user experience.
SUN will investigate new 3D acquisition techniques which will go beyond capabilities of current laser scanning or photogrammetry solutions. We will leverage on Al to optimize and automatize 3D acquisition and we will use solutions for massive avatar creation.
SUN will define artificial intelligence-based techniques to dynamically and incrementally create links between physical objects and their digital twins.
SUN will explore the design of haptic devices that provide users with a realistic feeling when interacting with virtual objects. We will explore miniaturized mechanical and thermal actuators, EMG decoding solutions, and skin stretching actuators. We will also explore and define techniques based on multimodal interaction integrating gaze and gesture interaction. Interface between humans and the integrated physical and virtual world shall be as much natural as possible.
SUN will explore advanced streaming solutions that allow high quality, in terms of resolution, photo realism, and speed, even in presence of low resource and computational power, by optimizing client- server interaction and balancing computation between the two.
SUN will validate these models in three real-life situations related to three social and human related scenarios: rehabilitation therapy, improved safety and social interaction among workers, and facilitate interaction for disabled users.
Work performed and main achievements include:
- Assessment of the state of the art in XR technologies: A literature review was conducted regarding advancements on the fields that are aligned with the project’s main objectives.
- specifications and preliminary prototypes of various components of the SUN XR platform
- Initial definition scenarios to be used for the three Pilots. This was achieved with on-site meeting at the pilot's sites, and a number of on-line meeting with the involved parties. The scenario collected so far for the three pilots are the following:
-- Pilot 1: XR for Rehabilitation
--- Scenario P1C1: Rehabilitation of the Upper Limb
--- Scenario P1C2: Rehabilitation of the Lower Limb
-- Pilot 2: XR in Industry
--- Scenario P2C1: Training & Practice on using Personal Protective Equipements
--- Scenario P2C2: Shopfloor safety & Object Tracking
-- Pilot 3: XR for people with motor and communication problems
--- Scenario P3C1: Persons who suffered a stroke
--- Scenario P3C2: Persons affected by apaty after an accident
-Initial collection of the User requirements: the requirement collection has been produced by the development of user personas, user stories.
- Identification of the technical components to be developed and production of their specifications. Development of some preliminary prototypes.
- Analysis of the Ethics and GDPR issues: This activity focused on benchmark research of the legal and ethical requirements pertaining to fundamental rights for data protection and privacy.
- Assessment of the state of the art in XR technologies: A literature review was conducted regarding advancements on the fields that are aligned with the project’s main objectives.
- specifications and preliminary prototypes of various components of the SUN XR platform
- Initial definition scenarios to be used for the three Pilots. This was achieved with on-site meeting at the pilot's sites, and a number of on-line meeting with the involved parties. The scenario collected so far for the three pilots are the following:
-- Pilot 1: XR for Rehabilitation
--- Scenario P1C1: Rehabilitation of the Upper Limb
--- Scenario P1C2: Rehabilitation of the Lower Limb
-- Pilot 2: XR in Industry
--- Scenario P2C1: Training & Practice on using Personal Protective Equipements
--- Scenario P2C2: Shopfloor safety & Object Tracking
-- Pilot 3: XR for people with motor and communication problems
--- Scenario P3C1: Persons who suffered a stroke
--- Scenario P3C2: Persons affected by apaty after an accident
-Initial collection of the User requirements: the requirement collection has been produced by the development of user personas, user stories.
- Identification of the technical components to be developed and production of their specifications. Development of some preliminary prototypes.
- Analysis of the Ethics and GDPR issues: This activity focused on benchmark research of the legal and ethical requirements pertaining to fundamental rights for data protection and privacy.
Among the results currently obtained in the project and the ongoing activities, we mention:
- development of a sensor-motor hand prothesis with integrated thermal feedback, and solutions for restoration of natural thermal sensation in upper-limb amputees
--- this provides upper limb amputees with the possibility of feeling thermal stimuli, for instance when handling a virtual object. This, jointly with the use of VR/AR headset to provide people with interesting opportunities for making more realistic handling virtual objects.
- solutions for retrieving human motion data, using natural language, and solutions based on Unsupervised Representation Learning for Human Skeleton Data Retrieval
--- this allows representing human motion data in a way that can be easily retrieved using simple natural language queries. This can be used, for instance, to retrieve motions or exercises to be executed in a rehabilitation settings.
- solutions for image-based localization
--- techniques based on these solutions allow to localize the position of a person in a building analyzing the scene seen from his/her point of view. This allows precise localization, in settings where no other precise localization solutions are available.
- solutions and prototype to reconstruct and understand 3D scenes from egocentric images captured by an headsets
--- This allows to generate 3D scenes at an affordable cost by walking in the physical environment to acquire
- solutions of Texture Inpainting for Photogrammetric Models
--- this allows removing texture artifacts generally occurring in 3D models representing real-world objects, acquired with the use of photogrammetric techniques. This is very important to produce high quality 3D assets with reduced effort and cost.
- solution for shape optimization and form-finding of free-form shells
--- this allows a designing solutions that comply with structural constraints, including statics, space requirements, lighting, and cost.
- development of a Lightweight Haptic Feedback Glove Employing Normal Indentation, Lateral Skin Stretch and both Softness and Hardness Rendering
--- Current haptic devices often provide tactile feedback via only vibrations and kinesthetic feedback based on heavy and cumbersome exoskeletons hindering users’ motion. This technique offers an innovative, lightweight, flexible and easy-to-wear haptic glove providing realistic tactile feedback through normal indentation and lateral skin stretch in addition to vibrations, as well as high-fidelity kinesthetic feedback through strings pulled by servo motors.
- A Miniature Direct-Drive Hydraulic Actuator for Wearable Haptic Devices based on Ferrofluid Magnetohydrodynamic Levitation
--- This solution to haptic feedback is based on a miniature hydraulic actuator based on a linear electromagnetic motor with an embedded ferrofluid sealing. The solution has two main advantages: it shows no static friction due to the magnetohydrodynamic levitation effect of the ferrofluid, and the output force can be scaled (by varying the radius of the actuator) without introducing noise and friction of mechanical reduction mechanisms.
- development of a sensor-motor hand prothesis with integrated thermal feedback, and solutions for restoration of natural thermal sensation in upper-limb amputees
--- this provides upper limb amputees with the possibility of feeling thermal stimuli, for instance when handling a virtual object. This, jointly with the use of VR/AR headset to provide people with interesting opportunities for making more realistic handling virtual objects.
- solutions for retrieving human motion data, using natural language, and solutions based on Unsupervised Representation Learning for Human Skeleton Data Retrieval
--- this allows representing human motion data in a way that can be easily retrieved using simple natural language queries. This can be used, for instance, to retrieve motions or exercises to be executed in a rehabilitation settings.
- solutions for image-based localization
--- techniques based on these solutions allow to localize the position of a person in a building analyzing the scene seen from his/her point of view. This allows precise localization, in settings where no other precise localization solutions are available.
- solutions and prototype to reconstruct and understand 3D scenes from egocentric images captured by an headsets
--- This allows to generate 3D scenes at an affordable cost by walking in the physical environment to acquire
- solutions of Texture Inpainting for Photogrammetric Models
--- this allows removing texture artifacts generally occurring in 3D models representing real-world objects, acquired with the use of photogrammetric techniques. This is very important to produce high quality 3D assets with reduced effort and cost.
- solution for shape optimization and form-finding of free-form shells
--- this allows a designing solutions that comply with structural constraints, including statics, space requirements, lighting, and cost.
- development of a Lightweight Haptic Feedback Glove Employing Normal Indentation, Lateral Skin Stretch and both Softness and Hardness Rendering
--- Current haptic devices often provide tactile feedback via only vibrations and kinesthetic feedback based on heavy and cumbersome exoskeletons hindering users’ motion. This technique offers an innovative, lightweight, flexible and easy-to-wear haptic glove providing realistic tactile feedback through normal indentation and lateral skin stretch in addition to vibrations, as well as high-fidelity kinesthetic feedback through strings pulled by servo motors.
- A Miniature Direct-Drive Hydraulic Actuator for Wearable Haptic Devices based on Ferrofluid Magnetohydrodynamic Levitation
--- This solution to haptic feedback is based on a miniature hydraulic actuator based on a linear electromagnetic motor with an embedded ferrofluid sealing. The solution has two main advantages: it shows no static friction due to the magnetohydrodynamic levitation effect of the ferrofluid, and the output force can be scaled (by varying the radius of the actuator) without introducing noise and friction of mechanical reduction mechanisms.