Periodic Reporting for period 1 - GuestXR (GuestXR: A Machine Learning Agent for Social Harmony in eXtended Reality)
Reporting period: 2022-01-01 to 2022-12-31
Immersive social spaces will soon become ubiquitous. However, there is also a warning to heed from social media.User content is the ‘lifeblood of social media’.However, it often stimulates antisocial interaction and abuse, ultimately posing a danger to vulnerable people.In the VR space this is backed up by the experience of current virtual shared spaces. While they have many positive aspects, they have also become a space full of abuse.Our vision is to develop GuestXR, a socially interactive multisensory platform system that uses eXtended Reality as the medium to bring people together for immersive, synchronous interaction with positive social outcomes.The innovation is the intervention of artificial agents that learn over time to help the virtual social gathering realise its aims.This is an agent that exploits Machine Learning to learn how to facilitate the meeting towards specific goals.Underpinning this is neuroscience and social psychology on group behaviour, will deliver rules ABM.The combination of AI with immersive systems, virtual and augmented reality will be a challenging research task, given the vagaries of social meetings and individual behaviour.A strong User Group made up of a diverse range of stakeholders will provide continuous feedback.
GuestXR deals with groups of people meeting in extended reality. The idea is based on the fact that every meeting has a set of goals, ranging from entertainment through to accomplishment of specific goal like in negotiations between conflicting parties. Evidence suggests that meetings also attract some undesirable behaviours such as various forms of discrimination and abuse, especially driven by anonymity and the relative lack of consequences. GuestXR aims at helping meetings achieve their goals while reducing the possibility of negative behaviour. The GuestXR concept is that of an underlying agent that monitors meeting behaviour and intervenes through possible actions that it has at its disposal. After each action that the agent (The Guest) carries the meeting may change, which moves closer to or further away from the desired goals, thus leading The Guest to receive a positive or negative reward. The Guest learns to choose a sequence of actions that maximises its long term reward, thus leading to a greater probability of successful outcomes.
GuestXR is based on RL that typically requires a lot of data for convergence and recording thousands of meetings. WP2 builds simulations with ABM based on social science theory that are used as a bootstrap for The Guest performance in real meetings that feeds with data for the simulations. Body and brain behaviour are also measured in real time. WP3 concerns the implementation of the extended reality for GuestXR and people's responses when the system is in operation. Beyond the typical sensorial components GuestXR is multi-modal, and includes haptics in WP4 i.e. if the shoulder gets touched, people should feel, as well as see, the touch. WP5 deals with applications including a persistent virtual space in which multiple people can visit and discuss topical matters, the use of GuestXR by people with hearing disabilities, a climate change application, and a conflict resolution application.
During the first year of the project we have made progress in research defining and developing the deep RL framework behind the Guest in two simulations: (i) social dilemma games (ii) a sequential social dilemma game, which is spatial – harvest. We integrated large language models with multi-user VR, important for more complex scenarios in which The Guest would be a hybrid AI system.
GuestXR is based on RL that typically requires a lot of data for convergence and recording thousands of meetings. WP2 builds simulations with ABM based on social science theory that are used as a bootstrap for The Guest performance in real meetings that feeds with data for the simulations. Body and brain behaviour are also measured in real time. WP3 concerns the implementation of the extended reality for GuestXR and people's responses when the system is in operation. Beyond the typical sensorial components GuestXR is multi-modal, and includes haptics in WP4 i.e. if the shoulder gets touched, people should feel, as well as see, the touch. WP5 deals with applications including a persistent virtual space in which multiple people can visit and discuss topical matters, the use of GuestXR by people with hearing disabilities, a climate change application, and a conflict resolution application.
During the first year of the project we have made progress in research defining and developing the deep RL framework behind the Guest in two simulations: (i) social dilemma games (ii) a sequential social dilemma game, which is spatial – harvest. We integrated large language models with multi-user VR, important for more complex scenarios in which The Guest would be a hybrid AI system.
The GuestXR concept as a whole is beyond the state of the art. Current conceptions of ‘the metaverse’ do include simple methods for the avoidance of abuse, but to our knowledge there is proposal for the comprehensive solution that we are working towards. In particular we frame the issue in a positive form – we aim to improve meetings, we aim to help them achieve their goals – and reducing negative behaviour is just a consequence of that. So overall GuestXR is a concept beyond the state of the art. By the end of the project we expect to have a full working system that is publicly available for use, and which will have been used in a number of realistic applications – contingent ones (such as the Financial Times application that has already been done) and those based on the use cases of WP5 (persistent virtual meetings, climate change, disabilities, and conflict resolution). There will also be contributions from the planned Open Call.
In the past decade developments in Virtual Reality (VR) hardware, in particular low-cost stereo head-tracked head-mounted displays (HMD) and associated tracking technology, have transformed VR from being an expensive esoteric tool available mainly in university labs and industry, into a low-cost consumer product. Global companies have transformed the situation so that devices today can deliver high quality experiences for the home user at costs equivalent to or even less than a smartphone. Since the 1990s research in VR has been limited to university laboratories and within industrial settings (such as design applications), and has concentrated on areas that are beneficial for society. This research has been under tight ethical control through university Ethical Boards and company health and safety guidelines. However, today, with VR entering the mass market there is little or no supervision. Moreover, global companies have embraced the concept of a ‘metaverse’. This far-reaching idea is that ultimately the web as we know it today will be replaced by world-wide platforms where very large numbers of people carry out their everyday work and leisure activities in an immersive shared world, where they can see and interact with digital representations of one another (their ‘avatars’). In such a metaverse they will carry out normal activities and of course all the functions of today’s social media will be encompassed within the metaverse. This is tremendously exciting, and could unleash a new era of creativity through liberation from the constraints of physical reality, but it also contains potential dangers.
The long-term importance of GuestXR cannot be underestimated. If the concept is successful in practice it could be an important and non-coercive way to improve multi-person online experiences, where there is increased probability of meetings reaching their goals. There of course can and should be rules and regulations that inform people about behaviour that is optimal for good outcomes. However, GuestXR will help people learn by experience what produces good outcomes. Reinforcement Learning is basically a sophisticated trial and error system – it acts, examines the consequences, if that did not move closer to the desired outcome it tries another action – and over time it learns an optimal set of actions to produce the desired outcomes. This also relies on democratic norms because when people join a meeting they implicitly accept that the meeting is for a purpose (even if entertainment – by the way a highly important part of the lives of people and not to be looked down upon), and the whole idea of GuestXR is to help realise that purpose. If it carries out inappropriate actions then people simply will not respond, and The Guest will learn not to carry those out. GuestXR relies on implicit learning – both for its own operation and of the participants.
In the past decade developments in Virtual Reality (VR) hardware, in particular low-cost stereo head-tracked head-mounted displays (HMD) and associated tracking technology, have transformed VR from being an expensive esoteric tool available mainly in university labs and industry, into a low-cost consumer product. Global companies have transformed the situation so that devices today can deliver high quality experiences for the home user at costs equivalent to or even less than a smartphone. Since the 1990s research in VR has been limited to university laboratories and within industrial settings (such as design applications), and has concentrated on areas that are beneficial for society. This research has been under tight ethical control through university Ethical Boards and company health and safety guidelines. However, today, with VR entering the mass market there is little or no supervision. Moreover, global companies have embraced the concept of a ‘metaverse’. This far-reaching idea is that ultimately the web as we know it today will be replaced by world-wide platforms where very large numbers of people carry out their everyday work and leisure activities in an immersive shared world, where they can see and interact with digital representations of one another (their ‘avatars’). In such a metaverse they will carry out normal activities and of course all the functions of today’s social media will be encompassed within the metaverse. This is tremendously exciting, and could unleash a new era of creativity through liberation from the constraints of physical reality, but it also contains potential dangers.
The long-term importance of GuestXR cannot be underestimated. If the concept is successful in practice it could be an important and non-coercive way to improve multi-person online experiences, where there is increased probability of meetings reaching their goals. There of course can and should be rules and regulations that inform people about behaviour that is optimal for good outcomes. However, GuestXR will help people learn by experience what produces good outcomes. Reinforcement Learning is basically a sophisticated trial and error system – it acts, examines the consequences, if that did not move closer to the desired outcome it tries another action – and over time it learns an optimal set of actions to produce the desired outcomes. This also relies on democratic norms because when people join a meeting they implicitly accept that the meeting is for a purpose (even if entertainment – by the way a highly important part of the lives of people and not to be looked down upon), and the whole idea of GuestXR is to help realise that purpose. If it carries out inappropriate actions then people simply will not respond, and The Guest will learn not to carry those out. GuestXR relies on implicit learning – both for its own operation and of the participants.