Periodic Reporting for period 2 - SUN (Social and hUman ceNtered XR)
Okres sprawozdawczy: 2024-06-01 do 2025-11-30
Extended reality (XR) is an emerging technology with promising potential in many fields including health, communication and safety. However, overcoming XR’s limitations in providing authentic interactive social environments is key to further developing its applications.
In the SUN (Social and hUman ceNtered XR) project (https://www.sun-xr-project.eu(odnośnik otworzy się w nowym oknie)) we have identified the current limitations, to make XR real and easily usable, and we addressed them. The SUN project aimed at advancing the social interactivity of XR technology by establishing scalable models with sustained and convincing virtual environments. The project also improved the data-processing capability of wearable devices and developed wearable sensors and tactile interfaces to improve user experience.
SUN overall objectives are:
• Investigating new 3D acquisition techniques which go beyond capabilities of current laser scanning or photogrammetry solutions. We leveraged on Al to optimize and automatize 3D acquisition and we used solutions for massive avatar creation.
• Defining artificial intelligence-based techniques to dynamically and incrementally semantically enrich physical and 3D scenes.
• Designing haptic devices that provide users with a realistic feeling when interacting with virtual objects. We developed miniaturized mechanical and thermal actuators, EMG decoding solutions, and skin stretching actuators. We also explored and defined techniques based on multimodal interaction integrating gaze and gesture interaction. The aim is to make interface between humans and the integrated physical and virtual world as natural as possible.
• Advancing streaming solutions that allow high quality, in terms of resolution, photo realism, and speed, even in presence of low resource and computational power, by optimizing client- server interaction and balancing computation between the two.
• Validating these models in three real-life situations related to three social and human related scenarios: rehabilitation therapy, improved safety and social interaction among workers, and facilitate interaction for disabled users.
In the SUN (Social and hUman ceNtered XR) project (https://www.sun-xr-project.eu(odnośnik otworzy się w nowym oknie)) we have identified the current limitations, to make XR real and easily usable, and we addressed them. The SUN project aimed at advancing the social interactivity of XR technology by establishing scalable models with sustained and convincing virtual environments. The project also improved the data-processing capability of wearable devices and developed wearable sensors and tactile interfaces to improve user experience.
SUN overall objectives are:
• Investigating new 3D acquisition techniques which go beyond capabilities of current laser scanning or photogrammetry solutions. We leveraged on Al to optimize and automatize 3D acquisition and we used solutions for massive avatar creation.
• Defining artificial intelligence-based techniques to dynamically and incrementally semantically enrich physical and 3D scenes.
• Designing haptic devices that provide users with a realistic feeling when interacting with virtual objects. We developed miniaturized mechanical and thermal actuators, EMG decoding solutions, and skin stretching actuators. We also explored and defined techniques based on multimodal interaction integrating gaze and gesture interaction. The aim is to make interface between humans and the integrated physical and virtual world as natural as possible.
• Advancing streaming solutions that allow high quality, in terms of resolution, photo realism, and speed, even in presence of low resource and computational power, by optimizing client- server interaction and balancing computation between the two.
• Validating these models in three real-life situations related to three social and human related scenarios: rehabilitation therapy, improved safety and social interaction among workers, and facilitate interaction for disabled users.
Work performed and main achievements include:
• SUN explored and developed innovative approaches to reconstructing, streaming, and enriching visual content for XR. These advancements include artificial intelligence-based scalable methods for generating and improving 3D environments, objects, and avatars; high-performance solutions for interactive streaming in XR; and the application of artificial intelligence to analyse and enrich 3D assets.
• SUN investigated advanced solutions for collaborating and interacting in XR. The project introduced novel techniques for delivering thermal feedback to users, alongside innovative wearable haptic devices that enable more natural manipulation of virtual objects. Additionally, the project developed systems that support collaborative interaction among multiple users within a shared XR session.
• SUN also explored and experimented with novel paradigms for capturing user input and feedback. Specifically, it proposed and demonstrated the use of electromyography to infer user movement intentions, alongside sensor and computer vision-based techniques for postural assessment and pose estimation. These efforts were complemented by solutions for hand gesture recognition and user emotion detection.
• The components and solutions developed within SUN were integrated into the SUN Integrated Platform, an innovative technological framework designed to support the sustainable and secure development of XR applications. The platform incorporates solutions based on non-fungible tokens for managing XR assets, along with advanced techniques for detecting cyber threats in XR applications.
• SUN also addressed the social and human perspective of XR solutions by examining the legal, ethical, and societal dimensions of XR. This includes exploring the implications of processing personal and sensitive data collected through wearable devices, 3D acquisition technologies, and human-machine interaction systems.
Over the course of its three-year duration, the SUN project validated its technologies through real-world case studies focused on key application areas: XR for rehabilitation, XR for workplace safety and social interaction, and XR solutions designed to support individuals with mobility and verbal communication impairments. Validation activities were conducted at three distinct sites: the Rehabilitation Unit of Versilia Hospital in Lido di Camaiore (Italy), the shop floor of FACTOR in Valencia (Spain), and the Clinique Romande de Réadaptation of SUVA in Sion (Switzerland). In each case, the validation process actively involved real end-users, ensuring that the solutions were tested in authentic, practical contexts.
We believe that SUN has not only advanced the technological frontier of extended reality but has also demonstrated its tangible value in real-world contexts, paving the way for sustainable, ethical, and human-centered XR solutions. By integrating multidisciplinary research with practical validation, SUN leaves behind a robust foundation for future innovation, collaboration, and adoption of XR technologies across Europe and beyond.
• SUN explored and developed innovative approaches to reconstructing, streaming, and enriching visual content for XR. These advancements include artificial intelligence-based scalable methods for generating and improving 3D environments, objects, and avatars; high-performance solutions for interactive streaming in XR; and the application of artificial intelligence to analyse and enrich 3D assets.
• SUN investigated advanced solutions for collaborating and interacting in XR. The project introduced novel techniques for delivering thermal feedback to users, alongside innovative wearable haptic devices that enable more natural manipulation of virtual objects. Additionally, the project developed systems that support collaborative interaction among multiple users within a shared XR session.
• SUN also explored and experimented with novel paradigms for capturing user input and feedback. Specifically, it proposed and demonstrated the use of electromyography to infer user movement intentions, alongside sensor and computer vision-based techniques for postural assessment and pose estimation. These efforts were complemented by solutions for hand gesture recognition and user emotion detection.
• The components and solutions developed within SUN were integrated into the SUN Integrated Platform, an innovative technological framework designed to support the sustainable and secure development of XR applications. The platform incorporates solutions based on non-fungible tokens for managing XR assets, along with advanced techniques for detecting cyber threats in XR applications.
• SUN also addressed the social and human perspective of XR solutions by examining the legal, ethical, and societal dimensions of XR. This includes exploring the implications of processing personal and sensitive data collected through wearable devices, 3D acquisition technologies, and human-machine interaction systems.
Over the course of its three-year duration, the SUN project validated its technologies through real-world case studies focused on key application areas: XR for rehabilitation, XR for workplace safety and social interaction, and XR solutions designed to support individuals with mobility and verbal communication impairments. Validation activities were conducted at three distinct sites: the Rehabilitation Unit of Versilia Hospital in Lido di Camaiore (Italy), the shop floor of FACTOR in Valencia (Spain), and the Clinique Romande de Réadaptation of SUVA in Sion (Switzerland). In each case, the validation process actively involved real end-users, ensuring that the solutions were tested in authentic, practical contexts.
We believe that SUN has not only advanced the technological frontier of extended reality but has also demonstrated its tangible value in real-world contexts, paving the way for sustainable, ethical, and human-centered XR solutions. By integrating multidisciplinary research with practical validation, SUN leaves behind a robust foundation for future innovation, collaboration, and adoption of XR technologies across Europe and beyond.
We identified the following 23 Key Innovation Results (KIR), that is results that present some relevant degree of innovation.
• Automated_Posture_Assessment
• Automatic Avatar Production Pipeline
• Tokenized Platform
• Cyber threat detection for XR
• SUN XR integrated platform
• Bi-directional human computer interface for stroke patients
• Rehabilitation in the metaverse
• Computer-vision based 3D pose estimation
• Computer-vision based face emotion recognition
• Compliant Wearable Haptic Devices
• Multimodal fusion
• XR Collaborative Interaction Techniques
• Semantic Scene Understanding
• Semantic-Aware AI for 3D Environmental Exploration
• Advanced AI-Driven 3D Reconstruction and Enhancement System
• Logistic processes optimization
• A wearable Medium Density EMG device for real-time muscle decoding
• Wearable thermal feedback device for somatosensory-impaired users
• XR Glanceable Interfaces
• Flexible wearable haptics
• Hololight Stream
• OmniBridge
• Transcranial temporal interference stimulation (tTIS)
We also identified the following 16 Key Exploitable Results (KER), that is results that might be realistically exploited in the future:
• SUN Integrated Platform
• Automatic Avatar Production Pipeline
• A medium-density EMG and IMU interface for gait rehabilitation and movement analysis
• Tokenized Platform
• XR Threat Detection
• Hololight Stream
• Wearable thermal feedback device for somatosensory-impaired users
• Automated Posture Assessment
• Finger Haptic Devices
• OmniBridge
• Dense Vision-Language Models for Open-Vocabulary Understanding
• Advanced AI-driven 3D design, reconstruction, processing and optimization
• Policy Paper on Ethics
• Multimodal emotion recognition
• Emotion recognition dataset
• Task optimization component
• Automated_Posture_Assessment
• Automatic Avatar Production Pipeline
• Tokenized Platform
• Cyber threat detection for XR
• SUN XR integrated platform
• Bi-directional human computer interface for stroke patients
• Rehabilitation in the metaverse
• Computer-vision based 3D pose estimation
• Computer-vision based face emotion recognition
• Compliant Wearable Haptic Devices
• Multimodal fusion
• XR Collaborative Interaction Techniques
• Semantic Scene Understanding
• Semantic-Aware AI for 3D Environmental Exploration
• Advanced AI-Driven 3D Reconstruction and Enhancement System
• Logistic processes optimization
• A wearable Medium Density EMG device for real-time muscle decoding
• Wearable thermal feedback device for somatosensory-impaired users
• XR Glanceable Interfaces
• Flexible wearable haptics
• Hololight Stream
• OmniBridge
• Transcranial temporal interference stimulation (tTIS)
We also identified the following 16 Key Exploitable Results (KER), that is results that might be realistically exploited in the future:
• SUN Integrated Platform
• Automatic Avatar Production Pipeline
• A medium-density EMG and IMU interface for gait rehabilitation and movement analysis
• Tokenized Platform
• XR Threat Detection
• Hololight Stream
• Wearable thermal feedback device for somatosensory-impaired users
• Automated Posture Assessment
• Finger Haptic Devices
• OmniBridge
• Dense Vision-Language Models for Open-Vocabulary Understanding
• Advanced AI-driven 3D design, reconstruction, processing and optimization
• Policy Paper on Ethics
• Multimodal emotion recognition
• Emotion recognition dataset
• Task optimization component