Periodic Reporting for period 2 - GuestXR (GuestXR: A Machine Learning Agent for Social Harmony in eXtended Reality)
Berichtszeitraum: 2023-01-01 bis 2024-06-30
Immersive social spaces will soon become ubiquitous. However, there is also a warning to heed from social media.User content is the ‘lifeblood of social media’.However, it often stimulates antisocial interaction and abuse, ultimately posing a danger to vulnerable people.In the VR space this is backed up by the experience of current virtual shared spaces. While they have many positive aspects, they have also become a space full of abuse.Our vision is to develop GuestXR, a socially interactive multisensory platform system that uses eXtended Reality as the medium to bring people together for immersive, synchronous interaction with positive social outcomes.The innovation is the intervention of artificial agents that learn over time to help the virtual social gathering realise its aims.This is an agent that exploits Machine Learning to learn how to facilitate the meeting towards specific goals.Underpinning this is neuroscience and social psychology on group behaviour, will deliver rules ABM.The combination of AI with immersive systems, virtual and augmented reality will be a challenging research task, given the vagaries of social meetings and individual behaviour.A strong User Group made up of a diverse range of stakeholders will provide continuous feedback.
GuestXR focuses on group meetings in extended reality, where each meeting has specific goals, such as entertainment or conflict resolution. However, these meetings can also attract negative behaviors like discrimination and abuse, often driven by anonymity. GuestXR aims to reduce these behaviors and help meetings achieve their objectives.
The core concept of GuestXR involves an agent that monitors meeting behavior and intervenes as needed. Each intervention can shift the dynamics, bringing the meeting closer to or further from its goals, which provides feedback to the agent. Over time, the agent learns to maximize the success of meetings.
Initially, GuestXR relied on Reinforcement Learning (RL), where the agent selected from predefined actions to help achieve meeting objectives, like encouraging participation. RL typically requires extensive data, so WP2 developed simulations using Agent-Based Models (ABM) based on social science theories to bootstrap The Guest's performance in real meetings. These simulations, coupled with real-time body and brain behavior measurements, help train the agent.
WP3 focuses on implementing the extended reality for GuestXR and analyzing user responses. Beyond standard sensory components, GuestXR is multi-modal and includes haptics in WP4, enabling users to feel, as well as see, interactions like a shoulder touch. WP5 explores applications such as virtual spaces for discussions, accessibility for people with hearing disabilities, climate change, and conflict resolution.
During the first year, the project progressed by developing the deep RL framework behind The Guest in two simulations: social dilemma games and a spatial sequential social dilemma game. Large language models were integrated with multi-user VR, essential for complex scenarios where The Guest acts as a hybrid AI system.
In the second year, the project created a full GuestXR example, focusing on the challenge of encouraging participation in small meetings. This involved collaboration among partners to design, implement, and test the application. The project now aims to expand this by involving more partners and incorporating developments from each WP, such as haptics and physiological recordings, improving auditory aspects, and preparing for broader testing.
As AI has rapidly evolved since GuestXR's inception, the project is now gradually introducing large language models (LLMs) to enhance meeting interventions, providing new ways to achieve objectives, such as giving voice to virtual characters guided by meeting goals.
The core concept of GuestXR involves an agent that monitors meeting behavior and intervenes as needed. Each intervention can shift the dynamics, bringing the meeting closer to or further from its goals, which provides feedback to the agent. Over time, the agent learns to maximize the success of meetings.
Initially, GuestXR relied on Reinforcement Learning (RL), where the agent selected from predefined actions to help achieve meeting objectives, like encouraging participation. RL typically requires extensive data, so WP2 developed simulations using Agent-Based Models (ABM) based on social science theories to bootstrap The Guest's performance in real meetings. These simulations, coupled with real-time body and brain behavior measurements, help train the agent.
WP3 focuses on implementing the extended reality for GuestXR and analyzing user responses. Beyond standard sensory components, GuestXR is multi-modal and includes haptics in WP4, enabling users to feel, as well as see, interactions like a shoulder touch. WP5 explores applications such as virtual spaces for discussions, accessibility for people with hearing disabilities, climate change, and conflict resolution.
During the first year, the project progressed by developing the deep RL framework behind The Guest in two simulations: social dilemma games and a spatial sequential social dilemma game. Large language models were integrated with multi-user VR, essential for complex scenarios where The Guest acts as a hybrid AI system.
In the second year, the project created a full GuestXR example, focusing on the challenge of encouraging participation in small meetings. This involved collaboration among partners to design, implement, and test the application. The project now aims to expand this by involving more partners and incorporating developments from each WP, such as haptics and physiological recordings, improving auditory aspects, and preparing for broader testing.
As AI has rapidly evolved since GuestXR's inception, the project is now gradually introducing large language models (LLMs) to enhance meeting interventions, providing new ways to achieve objectives, such as giving voice to virtual characters guided by meeting goals.
The GuestXR concept is ahead of current technology, aiming not just to prevent abuse in meetings but to enhance their effectiveness and help achieve goals, with reducing negative behavior as a natural outcome. We expect to have a fully operational system available before the project ends, used in various applications, including those from WP5 like persistent virtual meetings, climate change, disabilities, and conflict resolution.
Additionally, the Open Call, won by NTNU in Norway, explores 'demonstrations' in the metaverse, such as political ones. NTNU and partners, especially VBW, are working on this, with the first demonstration anticipated soon.
Recent advancements in Virtual Reality (VR), including affordable stereo head-tracked displays, have shifted VR from a niche tool to a widely accessible consumer product. This has transitioned VR from university labs and industry into the mass market, with global companies now offering high-quality, cost-effective VR experiences. Despite tight ethical controls in the past, the expansion of VR and the concept of the metaverse introduces new challenges, including potential risks as it evolves into a vast, immersive platform for work and leisure.
In response, the scientific leader of GuestXR agreed to lead the European Metaverse Research Network (EMRN), which held its first conference in April 2024 in Barcelona. GuestXR partners played a key role, and future conferences are planned for 2025. This involvement aims to integrate GuestXR's insights into the broader metaverse research community.
GuestXR's long-term impact could be significant. If successful, it may offer a non-coercive way to enhance online multi-person interactions, improving the likelihood of achieving meeting objectives. Although rules and regulations can guide behavior, GuestXR will help users learn through experience what leads to successful outcomes. Reinforcement Learning, based on trial and error, adapts actions based on consequences to optimize results. This system depends on democratic norms, as participants implicitly agree to a meeting’s purpose. If GuestXR fails to act appropriately, feedback will guide it to improve.
With the integration of large language models (LLMs), the potential interventions of The Guest will be more sophisticated and impactful than previously anticipated, presenting new technical, deployment, and ethical challenges. We anticipate an exciting year ahead for GuestXR, as it continues to push the boundaries of immersive technology and AI.
Additionally, the Open Call, won by NTNU in Norway, explores 'demonstrations' in the metaverse, such as political ones. NTNU and partners, especially VBW, are working on this, with the first demonstration anticipated soon.
Recent advancements in Virtual Reality (VR), including affordable stereo head-tracked displays, have shifted VR from a niche tool to a widely accessible consumer product. This has transitioned VR from university labs and industry into the mass market, with global companies now offering high-quality, cost-effective VR experiences. Despite tight ethical controls in the past, the expansion of VR and the concept of the metaverse introduces new challenges, including potential risks as it evolves into a vast, immersive platform for work and leisure.
In response, the scientific leader of GuestXR agreed to lead the European Metaverse Research Network (EMRN), which held its first conference in April 2024 in Barcelona. GuestXR partners played a key role, and future conferences are planned for 2025. This involvement aims to integrate GuestXR's insights into the broader metaverse research community.
GuestXR's long-term impact could be significant. If successful, it may offer a non-coercive way to enhance online multi-person interactions, improving the likelihood of achieving meeting objectives. Although rules and regulations can guide behavior, GuestXR will help users learn through experience what leads to successful outcomes. Reinforcement Learning, based on trial and error, adapts actions based on consequences to optimize results. This system depends on democratic norms, as participants implicitly agree to a meeting’s purpose. If GuestXR fails to act appropriately, feedback will guide it to improve.
With the integration of large language models (LLMs), the potential interventions of The Guest will be more sophisticated and impactful than previously anticipated, presenting new technical, deployment, and ethical challenges. We anticipate an exciting year ahead for GuestXR, as it continues to push the boundaries of immersive technology and AI.