Periodic Reporting for period 1 - GuestXR (GuestXR: A Machine Learning Agent for Social Harmony in eXtended Reality)
Periodo di rendicontazione: 2022-01-01 al 2022-12-31
GuestXR is based on RL that typically requires a lot of data for convergence and recording thousands of meetings. WP2 builds simulations with ABM based on social science theory that are used as a bootstrap for The Guest performance in real meetings that feeds with data for the simulations. Body and brain behaviour are also measured in real time. WP3 concerns the implementation of the extended reality for GuestXR and people's responses when the system is in operation. Beyond the typical sensorial components GuestXR is multi-modal, and includes haptics in WP4 i.e. if the shoulder gets touched, people should feel, as well as see, the touch. WP5 deals with applications including a persistent virtual space in which multiple people can visit and discuss topical matters, the use of GuestXR by people with hearing disabilities, a climate change application, and a conflict resolution application.
During the first year of the project we have made progress in research defining and developing the deep RL framework behind the Guest in two simulations: (i) social dilemma games (ii) a sequential social dilemma game, which is spatial – harvest. We integrated large language models with multi-user VR, important for more complex scenarios in which The Guest would be a hybrid AI system.
In the past decade developments in Virtual Reality (VR) hardware, in particular low-cost stereo head-tracked head-mounted displays (HMD) and associated tracking technology, have transformed VR from being an expensive esoteric tool available mainly in university labs and industry, into a low-cost consumer product. Global companies have transformed the situation so that devices today can deliver high quality experiences for the home user at costs equivalent to or even less than a smartphone. Since the 1990s research in VR has been limited to university laboratories and within industrial settings (such as design applications), and has concentrated on areas that are beneficial for society. This research has been under tight ethical control through university Ethical Boards and company health and safety guidelines. However, today, with VR entering the mass market there is little or no supervision. Moreover, global companies have embraced the concept of a ‘metaverse’. This far-reaching idea is that ultimately the web as we know it today will be replaced by world-wide platforms where very large numbers of people carry out their everyday work and leisure activities in an immersive shared world, where they can see and interact with digital representations of one another (their ‘avatars’). In such a metaverse they will carry out normal activities and of course all the functions of today’s social media will be encompassed within the metaverse. This is tremendously exciting, and could unleash a new era of creativity through liberation from the constraints of physical reality, but it also contains potential dangers.
The long-term importance of GuestXR cannot be underestimated. If the concept is successful in practice it could be an important and non-coercive way to improve multi-person online experiences, where there is increased probability of meetings reaching their goals. There of course can and should be rules and regulations that inform people about behaviour that is optimal for good outcomes. However, GuestXR will help people learn by experience what produces good outcomes. Reinforcement Learning is basically a sophisticated trial and error system – it acts, examines the consequences, if that did not move closer to the desired outcome it tries another action – and over time it learns an optimal set of actions to produce the desired outcomes. This also relies on democratic norms because when people join a meeting they implicitly accept that the meeting is for a purpose (even if entertainment – by the way a highly important part of the lives of people and not to be looked down upon), and the whole idea of GuestXR is to help realise that purpose. If it carries out inappropriate actions then people simply will not respond, and The Guest will learn not to carry those out. GuestXR relies on implicit learning – both for its own operation and of the participants.