Periodic Reporting for period 3 - CAROUSEL (CAROUSEL+)
Berichtszeitraum: 2024-01-01 bis 2024-12-31
Embodied Online Dancing and Partying with Digital Characters
With internet experience nowadays mostly being passive, disconnected and disembodied, many individuals are isolated instead of brought together. The EU-funded CAROUSEL project aims to overcome the isolation of online users by allowing them to dance and party together even if they are physically separated. Through imaginative combinations of artificial intelligence and immersive interaction technologies, including the creation of digital characters capable of interactive dancing, CAROUSEL enables internet users to feel each other’s presence, touch, and movement, from dancing together. These developments help overcome isolation and loneliness and bring improvements to human health, work, and wellbeing.
The Overall Objective
The uniqueness of the CAROUSEL value proposition resides in its capacity to empower meaningful visceral group social interaction to tangibly share imagination and physical movement in virtual and hybrid reality settings beyond the existing digital communication technology such as massive online game environments, peer-to-peer connections and social media networks. CAROUSEL paves the way for the emergence of social-physical behaviour of digital characters by better understanding human body-language and being capable of interacting autonomously with a group of people.
These goals were successfully achieved by setting and accomplishing the following strategic objectives:
-Design use cases as well as technical requirements for novel social interaction formats with AI/XR technologies and use validation results for an innovation transfer plan.
-Form the foundations for practical experimentation and deployment of collaborative XR solutions with broad application. Low-level remote sensing and actuation requires predictive latency aware schemes for connected devices, with 5G bandwidth conditions and beyond. More than 48 broadcast rich audio, haptic, motion and appearance signals per one server has been achieved by progress and delivery of DanceGraph, including DeFT-Net and NeFT-Net AI rhythmic prediction transformer models, and auditory occlusion effect of body presence in the direct sound path.
- Movement intelligence framework with the purpose of driving high-quality animated content, underpinning seamless and engaging visual experiences in the context of dancing. The framework interprets the signals from multiple people to animate user avatars in real time and control AI driven virtual characters that can perform dance motions in synchronization with music as well as in collaboration with the users.
- Large numbers of remote avatars reality-mixing together seamlessly rely on advancing augmented rendering ‘In the wild’ context of home interiors, dealing with spatially varying near field illumination and occlusions, and new schemes to affect the tele-presence of large bodies of people with the aggregation of spatial voices and proxemic touch and heat.
-Validation of the AI/XR-mediated experiences and interactions.
- Linking with relevant stakeholders and dissemination through academic publications, conference and trade-shows such as Medica 2023 and GamesCom2024.
With internet experience nowadays mostly being passive, disconnected and disembodied, many individuals are isolated instead of brought together. The EU-funded CAROUSEL project aims to overcome the isolation of online users by allowing them to dance and party together even if they are physically separated. Through imaginative combinations of artificial intelligence and immersive interaction technologies, including the creation of digital characters capable of interactive dancing, CAROUSEL enables internet users to feel each other’s presence, touch, and movement, from dancing together. These developments help overcome isolation and loneliness and bring improvements to human health, work, and wellbeing.
The Overall Objective
The uniqueness of the CAROUSEL value proposition resides in its capacity to empower meaningful visceral group social interaction to tangibly share imagination and physical movement in virtual and hybrid reality settings beyond the existing digital communication technology such as massive online game environments, peer-to-peer connections and social media networks. CAROUSEL paves the way for the emergence of social-physical behaviour of digital characters by better understanding human body-language and being capable of interacting autonomously with a group of people.
These goals were successfully achieved by setting and accomplishing the following strategic objectives:
-Design use cases as well as technical requirements for novel social interaction formats with AI/XR technologies and use validation results for an innovation transfer plan.
-Form the foundations for practical experimentation and deployment of collaborative XR solutions with broad application. Low-level remote sensing and actuation requires predictive latency aware schemes for connected devices, with 5G bandwidth conditions and beyond. More than 48 broadcast rich audio, haptic, motion and appearance signals per one server has been achieved by progress and delivery of DanceGraph, including DeFT-Net and NeFT-Net AI rhythmic prediction transformer models, and auditory occlusion effect of body presence in the direct sound path.
- Movement intelligence framework with the purpose of driving high-quality animated content, underpinning seamless and engaging visual experiences in the context of dancing. The framework interprets the signals from multiple people to animate user avatars in real time and control AI driven virtual characters that can perform dance motions in synchronization with music as well as in collaboration with the users.
- Large numbers of remote avatars reality-mixing together seamlessly rely on advancing augmented rendering ‘In the wild’ context of home interiors, dealing with spatially varying near field illumination and occlusions, and new schemes to affect the tele-presence of large bodies of people with the aggregation of spatial voices and proxemic touch and heat.
-Validation of the AI/XR-mediated experiences and interactions.
- Linking with relevant stakeholders and dissemination through academic publications, conference and trade-shows such as Medica 2023 and GamesCom2024.
The results achieved include:
-GAMES COM 2024 with +1000 visitors and participants
-Medica 2023 and other trade fairs (GDC, MWC, SIGGRAPH, CES, etc.)
-19 workshops on scientific subjects
-+20 presentations in scientific conferences and workshops (4 keynote speeches on scientific conferences)
-36 scientific publications and 16 press articles
- +650 citations of our publications at the time of reporting (2 months after the project end)
-2 Patents owned by ENU. One claim granted. One filed pre-claims allowed stage.
-21 open-source repositories on github
-Many research experiments have been conducted with +500 test users
- Delivery of the innovation transfer plan
- Delivery of third DanceGraph platform prototype
- Increase public awareness via the website, social networks, academic publications, trade-fairs, press articles, technical workshops and extended stakeholder contacts to the medical sector
- Collaboration with sister projects.
-GAMES COM 2024 with +1000 visitors and participants
-Medica 2023 and other trade fairs (GDC, MWC, SIGGRAPH, CES, etc.)
-19 workshops on scientific subjects
-+20 presentations in scientific conferences and workshops (4 keynote speeches on scientific conferences)
-36 scientific publications and 16 press articles
- +650 citations of our publications at the time of reporting (2 months after the project end)
-2 Patents owned by ENU. One claim granted. One filed pre-claims allowed stage.
-21 open-source repositories on github
-Many research experiments have been conducted with +500 test users
- Delivery of the innovation transfer plan
- Delivery of third DanceGraph platform prototype
- Increase public awareness via the website, social networks, academic publications, trade-fairs, press articles, technical workshops and extended stakeholder contacts to the medical sector
- Collaboration with sister projects.
CAROUSEL advanced the state of the art by advancing human-centred AI research in respect to real-world (physical and social) AI. The outcome of CAROUSEL are novel AI-Agents capable to directly interact through dancing with humans in real-time.
CAROUSEL contribute to following socio-economic impacts and wider societal implications so far by:
- Potential for future returns in terms of societal or economic innovation or market creation by developing novel technology and applications to enable new forms of full-body embodied interaction of VR dancing. The developed VR dancing prototypes facilitate people to stay physically and social active, which supports health and wellbeing as well as combatting loneliness by feeling connected to the other dance partner(s).
- CAROUSEL contributes to the scientific and technological foundation and consolidation of a radically new future technology by developing novel human-digital interaction scenarios, applications and VR/AR/AI technologies with the focus on dancing.
-Spreading excellence and building leading innovation capacity across Europe by engaging with young talents.
-Investigation to apply the research results to different customer needs. Along these trials, the needs of several market sectors have been analysed with the focus on medical and gaming.
CAROUSEL contribute to following socio-economic impacts and wider societal implications so far by:
- Potential for future returns in terms of societal or economic innovation or market creation by developing novel technology and applications to enable new forms of full-body embodied interaction of VR dancing. The developed VR dancing prototypes facilitate people to stay physically and social active, which supports health and wellbeing as well as combatting loneliness by feeling connected to the other dance partner(s).
- CAROUSEL contributes to the scientific and technological foundation and consolidation of a radically new future technology by developing novel human-digital interaction scenarios, applications and VR/AR/AI technologies with the focus on dancing.
-Spreading excellence and building leading innovation capacity across Europe by engaging with young talents.
-Investigation to apply the research results to different customer needs. Along these trials, the needs of several market sectors have been analysed with the focus on medical and gaming.