Embodied Online Dancing and Partying with Digital Characters
With internet experience nowadays mostly being passive, disconnected and disembodied, many individuals are isolated instead of brought together. The EU-funded CAROUSEL project aims to overcome the isolation of online users by allowing them to dance and party together even if they are physically separated. Through imaginative combinations of artificial intelligence and immersive interaction technologies, including the creation of digital characters capable of interactive dancing, CAROUSEL enables internet users to feel each other’s presence, touch, and movement, from dancing together. These developments help overcome isolation and loneliness and bring improvements to human health, work, and wellbeing.
The Overall Objective
The uniqueness of the CAROUSEL value proposition resides in its capacity to empower meaningful visceral group social interaction to tangibly share imagination and physical movement in virtual and hybrid reality settings beyond the existing digital communication technology such as massive online game environments, peer-to-peer connections and social media networks. CAROUSEL paves the way for the emergence of social-physical behaviour of digital characters by better understanding human body-language and being capable of interacting autonomously with a group of people.
These goals were successfully achieved by setting and accomplishing the following strategic objectives:
-Design use cases as well as technical requirements for novel social interaction formats with AI/XR technologies and use validation results for an innovation transfer plan.
-Form the foundations for practical experimentation and deployment of collaborative XR solutions with broad application. Low-level remote sensing and actuation requires predictive latency aware schemes for connected devices, with 5G bandwidth conditions and beyond. More than 48 broadcast rich audio, haptic, motion and appearance signals per one server has been achieved by progress and delivery of DanceGraph, including DeFT-Net and NeFT-Net AI rhythmic prediction transformer models, and auditory occlusion effect of body presence in the direct sound path.
- Movement intelligence framework with the purpose of driving high-quality animated content, underpinning seamless and engaging visual experiences in the context of dancing. The framework interprets the signals from multiple people to animate user avatars in real time and control AI driven virtual characters that can perform dance motions in synchronization with music as well as in collaboration with the users.
- Large numbers of remote avatars reality-mixing together seamlessly rely on advancing augmented rendering ‘In the wild’ context of home interiors, dealing with spatially varying near field illumination and occlusions, and new schemes to affect the tele-presence of large bodies of people with the aggregation of spatial voices and proxemic touch and heat.
-Validation of the AI/XR-mediated experiences and interactions.
- Linking with relevant stakeholders and dissemination through academic publications, conference and trade-shows such as Medica 2023 and GamesCom2024.
