Periodic Reporting for period 1 - XTREME (Mixed Reality Environment for Immersive Experience of Art and Culture)
Periodo di rendicontazione: 2024-01-01 al 2025-06-30
Art and culture are central to Europe’s identity, yet live performances remain difficult to access for many due to geography, cost, mobility, or limited venue capacity. They also require travel, contributing to environmental impacts. The XTREME project addresses these challenges by creating a human-centred mixed reality (MR) environment that enables audiences to experience concerts and performances virtually, without losing the richness of live events.
XTREME’s objectives are threefold: first, to advance MR technologies through a software framework that captures and reconstructs performances in 3D with high-fidelity sound and visuals; second, to demonstrate cultural and societal value through pilots with world-class artistic partners in music, opera, dance, and museums; and third, to embed ethics, inclusion, and accessibility at the core of technological development.
The project combines cutting-edge expertise in computer vision, AI, acoustics, and human–computer interaction with social sciences and humanities to ensure responsible, inclusive, and sustainable design. Expected impacts include expanding access to culture for underserved communities, reducing carbon-intensive travel, and opening new pathways for education and therapy.
By showcasing innovative MR performances and new artistic possibilities, XTREME contributes to Europe’s ambitions in Virtual Worlds and Web 4.0 supports the Green Deal, and strengthens European leadership in immersive technologies aligned with values of ethics, privacy, and inclusiveness.
XTREME’s objectives are threefold: first, to advance MR technologies through a software framework that captures and reconstructs performances in 3D with high-fidelity sound and visuals; second, to demonstrate cultural and societal value through pilots with world-class artistic partners in music, opera, dance, and museums; and third, to embed ethics, inclusion, and accessibility at the core of technological development.
The project combines cutting-edge expertise in computer vision, AI, acoustics, and human–computer interaction with social sciences and humanities to ensure responsible, inclusive, and sustainable design. Expected impacts include expanding access to culture for underserved communities, reducing carbon-intensive travel, and opening new pathways for education and therapy.
By showcasing innovative MR performances and new artistic possibilities, XTREME contributes to Europe’s ambitions in Virtual Worlds and Web 4.0 supports the Green Deal, and strengthens European leadership in immersive technologies aligned with values of ethics, privacy, and inclusiveness.
In its first phase, XTREME has established the foundations of its mixed reality (MR) environment and demonstrated early prototypes integrating diverse technical components. A major milestone has been the creation of large-scale, high-quality audio-visual datasets of music and dance. These were captured through two events: a string quartet and Irish folk music and dance recording at the University of Nottingham, and an extended session with the Irish Chamber Orchestra in Limerick, including multichannel and ambisonic audio plus multi-camera video. These datasets form the basis for current and future XR productions and enable systematic technical evaluation.
On the technical side, the consortium has advanced several key building blocks. An automated pipeline for generating music visualisations from audio has been developed using open-source AI models. Progress in spatial audio includes controlled simulation and testing of sound localisation and microphone positioning. Markerless human motion capture has been significantly improved, delivering time-consistent, fine-grained reconstruction of hands and feet for avatar animation, integrated into the Unity pipeline. Novel pipelines for scene understanding in artistic performance settings have also been created. In addition, the XTREME system and API were defined and implemented, enabling cross-partner integration and remote multiplayer environments. Prototype interaction and spatial audio frameworks for Apple Vision Pro have been realised, including gesture-based control and hybrid XR audio systems.
Complementing these advances, the project has conducted empirical user studies on co-experience in XR, measuring synchrony in skin conductance and exploring how XR can support private conversations. First MR productions — such as the immersive traditional Irish music and dance performance — provided critical insight into the challenges of staging MR events and informed the technical roadmap for the next phases. Creative partners have delivered artistic content for the first Apple Vision Pro production and initiated work on further productions.
Finally, the Living Lab has been established as a hub for co-creation and cross-disciplinary collaboration, successfully bringing together internationally renowned artists, technologists, and researchers. This process has ensured that technical development is informed by creative and social needs, laying the foundation for both innovative MR performances and new methodologies for artistic collaboration in XR.
On the technical side, the consortium has advanced several key building blocks. An automated pipeline for generating music visualisations from audio has been developed using open-source AI models. Progress in spatial audio includes controlled simulation and testing of sound localisation and microphone positioning. Markerless human motion capture has been significantly improved, delivering time-consistent, fine-grained reconstruction of hands and feet for avatar animation, integrated into the Unity pipeline. Novel pipelines for scene understanding in artistic performance settings have also been created. In addition, the XTREME system and API were defined and implemented, enabling cross-partner integration and remote multiplayer environments. Prototype interaction and spatial audio frameworks for Apple Vision Pro have been realised, including gesture-based control and hybrid XR audio systems.
Complementing these advances, the project has conducted empirical user studies on co-experience in XR, measuring synchrony in skin conductance and exploring how XR can support private conversations. First MR productions — such as the immersive traditional Irish music and dance performance — provided critical insight into the challenges of staging MR events and informed the technical roadmap for the next phases. Creative partners have delivered artistic content for the first Apple Vision Pro production and initiated work on further productions.
Finally, the Living Lab has been established as a hub for co-creation and cross-disciplinary collaboration, successfully bringing together internationally renowned artists, technologists, and researchers. This process has ensured that technical development is informed by creative and social needs, laying the foundation for both innovative MR performances and new methodologies for artistic collaboration in XR.
XTREME has already delivered results beyond the state of the art across several technical domains. In audio-visual capture, new pipelines were developed for evaluating emotions and themes in music, complemented by experiments in sound back-projection and ambisonic microphone setups. Markerless motion capture has advanced substantially: a near joint system for body, hand, and foot tracking was achieved, overcoming limits of single-view methods. Accurate foot capture and fine-grained hand reconstruction were demonstrated, and a new scene understanding model forecasting scene graph evolution outperformed benchmarks such as the Action Genome dataset. In addition, novel ambisonic rendering frameworks and loss functions accounting for microphone directionality have been introduced. On the immersive systems side, breakthroughs include intuitive gesture-based interaction with spatial audio, high-quality 3D MR video for seated AR contexts, Unity integration pipelines, and a multi-user “live-to-play” festival environment on Apple Vision Pro. These achievements expand the design space for real-time cultural XR productions and interactive performances.
Beyond the technical level, XTREME has demonstrated physiological synchrony as a reliable measure of shared XR experience, offering new tools for assessing co-presence and privacy in social XR contexts. The project has also established new creative workflows where technological and artistic development are interwoven, providing both methodological advances and novel aesthetic expressions. Finally, regulatory and policy-oriented research produced a state-of-the-art analysis of data, AI, and XR governance, addressing challenges such as human digital twins, behaviour-based avatars, and artists’ rights in relation to generative AI.
Together, these results position XTREME as a driver of both scientific and cultural innovation. Potential impacts range from new tools for immersive artistic performance to frameworks for trustworthy and sustainable XR adoption. To ensure further uptake, the project identifies the need for large-scale demonstrations, integration into creative industry workflows, access to standards and IPR support, and engagement with governance frameworks that underpin responsible innovation.
Beyond the technical level, XTREME has demonstrated physiological synchrony as a reliable measure of shared XR experience, offering new tools for assessing co-presence and privacy in social XR contexts. The project has also established new creative workflows where technological and artistic development are interwoven, providing both methodological advances and novel aesthetic expressions. Finally, regulatory and policy-oriented research produced a state-of-the-art analysis of data, AI, and XR governance, addressing challenges such as human digital twins, behaviour-based avatars, and artists’ rights in relation to generative AI.
Together, these results position XTREME as a driver of both scientific and cultural innovation. Potential impacts range from new tools for immersive artistic performance to frameworks for trustworthy and sustainable XR adoption. To ensure further uptake, the project identifies the need for large-scale demonstrations, integration into creative industry workflows, access to standards and IPR support, and engagement with governance frameworks that underpin responsible innovation.