Periodic Reporting for period 4 - MoTIVE (Moments in Time in Immersive Virtual Environments)
Période du rapport: 2022-07-01 au 2023-12-31
To explore this, the project reconstructed a Dire Straits concert ('Sultans of Swing': https://youtu.be/bOSWaKT88j4) and Massiel's winning Eurovision performance (https://youtu.be/dGcmRNYFtt8). This required: creating realistic virtual bodies of the performers from photos and videos, building responsive virtual audiences, and overcoming problems of using VR such as simulator sickness.
Moreover, we could go beyond reality where participants could be onstage as part of the band. The project explored how the sense of owning the virtual body and its actions (agency) would influence behaviour. Additionally, 'VR United' software was developed to enable multiple users to share a VR space and engage in social interaction even across different continents.
VR's increasing consumer availability shows the need to understand its various, including ethical, implications an issue considered during the project, leading to a new spin-off project funded by the Spanish Government. VR United has commercial potential and is, for example, now being explored as a tool to combat loneliness among older adults. Most notably, the project aims to help older people relive their past in reconstructed events using an enhanced method of ‘reminiscence therapy’. Initial results, based on pilot studies, suggest these VR experiences can positively improve some negative aspects of ageing.
The project developed a novel method (Adaptive Multimodal Matching – A3M) to find optimal VR settings for presence in real-time based on participant choices, rather than post-experience questionnaires. A3M, a Reinforcement Learning (AI) model, successfully determined optimal configurations for presence. An experiment in conjunction with Facebook was used to determine optimal configurations for a virtual TED talk.
VR reminiscence therapy was explored, with older adults reliving a salient Eurovision performance from their youth. While analysis is pending, pilot results suggest these immersive experiences can positively impact negative aspects of ageing.
The project also demonstrated how VR can be a potent tool to address biases and encourage pro-social behavior. We enhanced earlier findings that embodying people in virtual bodies of a different race can reduce their implicit biases against people of that race. Here we found an increase in implicit bias when the embodiment occurred in a stressful scenario, challenging the notion of VR as an automatic empathy machine. Continuing this line, our ‘Golden Rule Embodiment Paradigm’, where participants witness their own negative actions or acquiescent inaction from the embodied perspective of a victim, was successful in reducing bias and encouraging helpful interventions in a VR scenario in a United States police department, a project in collaboration with Google Jigsaw (https://youtu.be/tb9QAUkZWic). This method is widely used by our spin-off company kiin.tech to address workplace discrimination.
Several technical advances were made: a system to create realistic virtual bodies from photos, a solution for natural walking movement in VR despite space constraints, and the QuickVR programming library. The VR United software allows remote users to interact in VR with with others with each person with a virtual body that looks like themselves. It facilitated journalistic interviews (https://www.youtube.com/watch?v=1dACicAYdYg https://youtu.be/njVlI8409fs) and a conference panel discussion including Albert Einstein driven by ChatGPT (https://youtu.be/qkN1F3QAhp8).
We went beyond the original project specification by studying ‘Change blindness’, a phenomenon where people can be oblivious to major changes in their environment. We found that in VR people were surprisingly oblivious to gradual changes in the appearances of themselves and others (https://youtu.be/XPkUIjBKqUU). This concept was applied to a single-session therapy for helping people overcome their fear of public speaking, in comparison with a traditional 5 session exposure therapy.
We developed a new technique that allows participants to walk through a VR scenario even though the physical space in which they are located is much smaller than the virtual space in which they can walk.
Finally, we found that gradually increasing an environment from dark to light allows faster movement through the environment, but without causing simulator sickness. This is very important because it offers a way forward for people to use VR without feeling the symptoms of simulator sickness.
We introduced the AI based version of our previous 3M technique for optimising VR experiences, leading to further use of this method (at least 7 external papers) and continued use in our lab (6 studies). This approach offers a more objective way to assess factors that impact presence and other responses, without the need for questionnaires.
We found a strong inverse relationship between the level of order (entropy) of eye movements and the sense of presence in VR. This suggests a new ‘familiarity’ dimension, where internal world models play a significant role in how participants experience VR. The connection of presence with eye movement behaviour demonstrates that presence is a genuine psychophysiological phenomenon and not just a conceptual construction of researchers.
The many uses of VR United and its components (look-alike self-representations) provide an important step forward in the exploitation of VR in the developing idea of the metaverse.
Our work on change blindness in VR, demonstrating how people can miss dramatic changes, even to their own bodies and those of others and even though directly observed. This has has far-reaching potential applications that we are persuing.
The 'Golden Rule Embodiment Paradigm' is a novel tool for promoting pro-social behavior. Its success shows a potential to shape attitudes and actions.
We have discovered a new way to mitigate simulator sickness by manipulating environmental brightness, improving accessibility and comfort for VR users.
An extended version of this report can be found on https://www.motiverc.org/extended-final-report/