Periodic Reporting for period 3 - GuestXR (GuestXR: A Machine Learning Agent for Social Harmony in eXtended Reality)
Berichtszeitraum: 2024-07-01 bis 2025-12-31
Immersive social spaces will soon become ubiquitous. However, there is also a warning to heed from social media. User content is the ‘lifeblood of social media’. However, it often stimulates antisocial interaction and abuse, ultimately posing a danger to vulnerable people. In the VR space this is backed up by the experience of current virtual shared spaces. While they have many positive aspects, they have also become a space full of abuse.
Our vision is to develop GuestXR, a socially interactive multisensory platform system that uses eXtended Reality as the medium to bring people together for immersive, synchronous interaction with positive social outcomes. The innovation is the intervention of artificial agents that learn over time to help the virtual social gathering realise its aims. This is an agent that exploits Machine Learning to learn how to facilitate the meeting towards specific goals. Underpinning this is neuroscience and social psychology on group behaviour, will deliver rules ABM.
The combination of AI with immersive systems, virtual and augmented reality will be a challenging research task, given the vagaries of social meetings and individual behaviour. A strong User Group made up of a diverse range of stakeholders will provide continuous feedback.
At the end of the action, GuestXR has demonstrated a fully integrated multi-user XR platform in which an adaptive AI agent can monitor, interpret and influence social dynamics in immersive environments, supporting constructive and goal-oriented interaction.
Our vision is to develop GuestXR, a socially interactive multisensory platform system that uses eXtended Reality as the medium to bring people together for immersive, synchronous interaction with positive social outcomes. The innovation is the intervention of artificial agents that learn over time to help the virtual social gathering realise its aims. This is an agent that exploits Machine Learning to learn how to facilitate the meeting towards specific goals. Underpinning this is neuroscience and social psychology on group behaviour, will deliver rules ABM.
The combination of AI with immersive systems, virtual and augmented reality will be a challenging research task, given the vagaries of social meetings and individual behaviour. A strong User Group made up of a diverse range of stakeholders will provide continuous feedback.
At the end of the action, GuestXR has demonstrated a fully integrated multi-user XR platform in which an adaptive AI agent can monitor, interpret and influence social dynamics in immersive environments, supporting constructive and goal-oriented interaction.
GuestXR focuses on group meetings in extended reality, where each meeting has specific goals, such as entertainment or conflict resolution. However, these meetings can also attract negative behaviours like discrimination and abuse, often driven by anonymity. GuestXR aims to reduce these behaviours and help meetings achieve their objectives.
The core concept involves an agent that monitors meeting behaviour and intervenes as needed. Each intervention shifts the dynamics, bringing the meeting closer to or further from its goals, which provides feedback to the agent. Over time, the agent learns to maximise meeting success.
Initially, GuestXR relied on Reinforcement Learning (RL), where the agent selected from predefined actions to support meeting objectives, such as encouraging participation. Since RL requires extensive data, WP2 developed simulations using Agent-Based Models (ABM) based on social science theories to bootstrap performance in real meetings. These simulations, together with real-time body and brain behaviour measurements, supported training and validation.
WP3 implemented the extended reality infrastructure and analysed user responses. GuestXR is multi-modal and includes haptics in WP4, enabling users to feel as well as see interactions. WP5 explored applications including virtual discussion spaces, accessibility for people with hearing disabilities, climate change dialogue, and conflict resolution.
During the project, a full GuestXR platform was realised. Large language models were integrated with multi-user VR to enable more flexible and context-sensitive interventions, allowing The Guest to act as a hybrid AI system.
By the end of the project, the consortium delivered:
• A multi-user XR platform deployed and demonstrated publicly.
• Integrated AI moderation combining reinforcement learning, simulations and language models.
• Multisensory extensions including haptic interfaces.
• Neurophysiological measurement pipelines for immersive settings.
• Validated use cases across several domains.
• An awarded patent related to haptic technology.
• A consolidated exploitation and business plan identifying Key Exploitable Results.
• Extensive dissemination, including 60 scientific publications and public demonstrations.
The project concludes with a validated integrated system and a structured pathway towards further commercial and research exploitation.
The core concept involves an agent that monitors meeting behaviour and intervenes as needed. Each intervention shifts the dynamics, bringing the meeting closer to or further from its goals, which provides feedback to the agent. Over time, the agent learns to maximise meeting success.
Initially, GuestXR relied on Reinforcement Learning (RL), where the agent selected from predefined actions to support meeting objectives, such as encouraging participation. Since RL requires extensive data, WP2 developed simulations using Agent-Based Models (ABM) based on social science theories to bootstrap performance in real meetings. These simulations, together with real-time body and brain behaviour measurements, supported training and validation.
WP3 implemented the extended reality infrastructure and analysed user responses. GuestXR is multi-modal and includes haptics in WP4, enabling users to feel as well as see interactions. WP5 explored applications including virtual discussion spaces, accessibility for people with hearing disabilities, climate change dialogue, and conflict resolution.
During the project, a full GuestXR platform was realised. Large language models were integrated with multi-user VR to enable more flexible and context-sensitive interventions, allowing The Guest to act as a hybrid AI system.
By the end of the project, the consortium delivered:
• A multi-user XR platform deployed and demonstrated publicly.
• Integrated AI moderation combining reinforcement learning, simulations and language models.
• Multisensory extensions including haptic interfaces.
• Neurophysiological measurement pipelines for immersive settings.
• Validated use cases across several domains.
• An awarded patent related to haptic technology.
• A consolidated exploitation and business plan identifying Key Exploitable Results.
• Extensive dissemination, including 60 scientific publications and public demonstrations.
The project concludes with a validated integrated system and a structured pathway towards further commercial and research exploitation.
The GuestXR concept is ahead of current technology, aiming not just to prevent abuse in meetings but to enhance their effectiveness and help achieve goals, with reducing negative behavior as a natural outcome. We expect to have a fully operational system available before the project ends, used in various applications, including those from WP5 like persistent virtual meetings, climate change, disabilities, and conflict resolution.
Additionally, the Open Call, won by NTNU in Norway, explores 'demonstrations' in the metaverse, such as political ones. NTNU and partners, especially VBW, are working on this, with the first demonstration anticipated soon.
Recent advancements in Virtual Reality (VR), including affordable stereo head-tracked displays, have shifted VR from a niche tool to a widely accessible consumer product. This has transitioned VR from university labs and industry into the mass market, with global companies now offering high-quality, cost-effective VR experiences. Despite tight ethical controls in the past, the expansion of VR and the concept of the metaverse introduces new challenges, including potential risks as it evolves into a vast, immersive platform for work and leisure.
In response, the scientific leader of GuestXR agreed to lead the European Metaverse Research Network (EMRN), which held its first conference in April 2024 in Barcelona. GuestXR partners played a key role, and future conferences are planned for 2025. This involvement aims to integrate GuestXR's insights into the broader metaverse research community.
GuestXR's long-term impact could be significant. If successful, it may offer a non-coercive way to enhance online multi-person interactions, improving the likelihood of achieving meeting objectives. Although rules and regulations can guide behavior, GuestXR will help users learn through experience what leads to successful outcomes. Reinforcement Learning, based on trial and error, adapts actions based on consequences to optimize results. This system depends on democratic norms, as participants implicitly agree to a meeting’s purpose. If GuestXR fails to act appropriately, feedback will guide it to improve.
With the integration of large language models (LLMs), the potential interventions of The Guest will be more sophisticated and impactful than previously anticipated, presenting new technical, deployment, and ethical challenges. We anticipate an exciting year ahead for GuestXR, as it continues to push the boundaries of immersive technology and AI.
In conclusion, GuestXR has established a scientific and technological foundation for AI-supported social moderation in immersive environments. Its results support future developments in corporate training, education, rehabilitation, inclusive interaction, and socially sensitive immersive applications. The project demonstrates that immersive XR systems can be designed not only to connect people, but to actively support constructive, goal-oriented and socially responsible interaction.
Additionally, the Open Call, won by NTNU in Norway, explores 'demonstrations' in the metaverse, such as political ones. NTNU and partners, especially VBW, are working on this, with the first demonstration anticipated soon.
Recent advancements in Virtual Reality (VR), including affordable stereo head-tracked displays, have shifted VR from a niche tool to a widely accessible consumer product. This has transitioned VR from university labs and industry into the mass market, with global companies now offering high-quality, cost-effective VR experiences. Despite tight ethical controls in the past, the expansion of VR and the concept of the metaverse introduces new challenges, including potential risks as it evolves into a vast, immersive platform for work and leisure.
In response, the scientific leader of GuestXR agreed to lead the European Metaverse Research Network (EMRN), which held its first conference in April 2024 in Barcelona. GuestXR partners played a key role, and future conferences are planned for 2025. This involvement aims to integrate GuestXR's insights into the broader metaverse research community.
GuestXR's long-term impact could be significant. If successful, it may offer a non-coercive way to enhance online multi-person interactions, improving the likelihood of achieving meeting objectives. Although rules and regulations can guide behavior, GuestXR will help users learn through experience what leads to successful outcomes. Reinforcement Learning, based on trial and error, adapts actions based on consequences to optimize results. This system depends on democratic norms, as participants implicitly agree to a meeting’s purpose. If GuestXR fails to act appropriately, feedback will guide it to improve.
With the integration of large language models (LLMs), the potential interventions of The Guest will be more sophisticated and impactful than previously anticipated, presenting new technical, deployment, and ethical challenges. We anticipate an exciting year ahead for GuestXR, as it continues to push the boundaries of immersive technology and AI.
In conclusion, GuestXR has established a scientific and technological foundation for AI-supported social moderation in immersive environments. Its results support future developments in corporate training, education, rehabilitation, inclusive interaction, and socially sensitive immersive applications. The project demonstrates that immersive XR systems can be designed not only to connect people, but to actively support constructive, goal-oriented and socially responsible interaction.