Periodic Reporting for period 3 - UnReal (A Phenomenological, Computational and Neural model of the Sense of Reality)
Reporting period: 2023-03-01 to 2024-08-31
We all rely on our senses to convey veridical information about the world around us, which we call “Reality”. However, distortions of reality in the form of hallucinations or illusions originating from neurological, psychiatric, pharmacological or psychological origins are common. Thus, the sense of reality which is the ability to discriminate between true and false perceptions of the world, is a central criterion for neurological and psychiatric health. Despite the critical role of the sense of reality in our daily life, little is known about how this is formed in the mind and brain. The “UnReal” project proposes a novel theoretical and technological framework to uncover how the sense of Reality the phenomenological, computational and neural processes underlying the Sense of Reality (SoR). The proposed experiments employ a specialized virtual reality environment allowing precise and controlled manipulations of visual reality across different domains (AliceVR). Using this VR platform, we will first test the impact of parametrically manipulating visual reality on judgments of reality in order to build a phenomenological and computational model of SoR (WP1). We will then investigate the neural mechanisms underlying SoR by combining our VR paradigm with high resolution fMRI (WP2). Finally, WP3 will investigate conditions of altered SoR, either of psychiatric origin or pharmacological origin. Here, we will empirically test SoR under conditions of altered reality processing in schizophrenia patients with hallucinations and in a pharmacological model through administration of ketamine in healthy participants. This ambitious project will thus reveal how humans discriminate reality at the phenomenological, cognitive and neural levels. The results will have important implications not only in defining a neurocognitive model of reality in healthy and clinical populations, but also shedding new light upon the fundamental philosophical question of “how do we know what is real?”.
Understanding SoR is also of great importance to society at large. First, SoR is a central benchmark in determining psychiatric and neurological wellbeing. Indeed, symptoms of impaired SoR such as hallucinations, delusions and derealization are a key diagnostic criterion for psychosis and are also present in epilepsy, dementia, Parkinson’s disease and many other conditions. As such, it is critical that we acquire a fuller understanding of how SoR is constructed by the brain and mind, allowing us to provide better diagnostic and therapeutic tools. Furthermore, recent years have brought an unprecedented surge in technological measures mimicking human perceptual experience. Only 120 years ago silent video images began captivating and generating simulated perceptual experiences. Today, consumer grade virtual and augmented reality can create astonishingly immersive experiences 86,159,160 raising ethical, health and societal concerns132,161. A central issue here is the unknown impact of virtual and augmented reality (AR) on the distinction between true percepts and digitally simulated ones. Thus, VR and AR can be considered “cognitively disruptive technology” posing a novel and unique challenge to the SoR. In a world in which the delineation between real and unreal perceptions is becoming more demanding, a strong scientific understanding of the underlying cognitive and neural mechanisms of SoR is essential.
Understanding SoR is also of great importance to society at large. First, SoR is a central benchmark in determining psychiatric and neurological wellbeing. Indeed, symptoms of impaired SoR such as hallucinations, delusions and derealization are a key diagnostic criterion for psychosis and are also present in epilepsy, dementia, Parkinson’s disease and many other conditions. As such, it is critical that we acquire a fuller understanding of how SoR is constructed by the brain and mind, allowing us to provide better diagnostic and therapeutic tools. Furthermore, recent years have brought an unprecedented surge in technological measures mimicking human perceptual experience. Only 120 years ago silent video images began captivating and generating simulated perceptual experiences. Today, consumer grade virtual and augmented reality can create astonishingly immersive experiences 86,159,160 raising ethical, health and societal concerns132,161. A central issue here is the unknown impact of virtual and augmented reality (AR) on the distinction between true percepts and digitally simulated ones. Thus, VR and AR can be considered “cognitively disruptive technology” posing a novel and unique challenge to the SoR. In a world in which the delineation between real and unreal perceptions is becoming more demanding, a strong scientific understanding of the underlying cognitive and neural mechanisms of SoR is essential.
We have developed several ecological, realistic virtual reality environments in which participants interact with objects and perform sensorimotor tasks. Moreover, we have embedded within this VR environment objects which exist both in the virtual world but also have exact counterparts in the real world (e.g. a virtual table which is a replica of a real table the participants sit next to). This causes an enhanced sense of realism to the virtual world through allowing multisensory (visual, tactile and proprioceptive) correspondences and embedding a feeling of continuity between the real and virtual worlds. We tested several such environments through extensive pilot testing to ensure an optimal setting inducing a strong sense of realism in our participants. We then constructed “virtual hallucinations” in which we alter nine different aspects of reality of three domains: Perceptual changes, in which the visual appearance (e.g. colors, acuity) of the scene are manipulated. (2) Laws of Nature, in which we will manipulate visual aspects of laws of nature (e.g. gravity, passage of time. (3) Changes of Self, in which we will manipulate the participants’ sense of self through conflicts between visual signals and self-related information (e.g. sensorimotor conflicts, changes in first person perspective). These alterations are a hallmark of altered reality in several neurological, psychiatric and mystical experiences. We conducted several experiments on more than 200 human participants including measurements of behavior, eye tracking, pupillometry, ECG and kinematic data to assess how virtual hallucination impact phenomenological, cognitive and physiological processing. We have impressive results from preregistered experiments including internal replications in which we found that virtual hallucinations across different domains have different impacts on the sense of Reality. Moreover, we could classify the sense of reality from physiological responses. We have built a computational model based on this data showing how decisions regarding reality are achieved using a Bayesian inference process.
The UnReal project has harnessed technological advancements in virtual reality and through them enabled a new field of study which has not been investigated. The ability to create such alterations of reality has provided a novel paradigm allowing ecological and well controlled manipulation of the Sense of Reality. This in turn allowed to study previously neglected aspects of human cognition such as SoR. Finally this methodology and the discovery of the computational model of the cognitive and neural mechanisms of SoR is of high interest to not only cognitive scientists but also to mental health professionals and developers of XR technologies. Now that the experimental paradigms are well established we expect to learn how the sense of reality is processed in the brain (using fMRI), an dhow this relates to psychiatric conditions such as psychosis and depersonalization/ derealization.