Periodic Reporting for period 2 - ILLUSIVE (Foundations of Perception Engineering)
Reporting period: 2023-02-01 to 2024-07-31
Virtual reality (VR) technology has enormous potential to transform society, especially as the world faces unprecedented challenges of remote work and environmentally friendly travel. VR creates perceptual illusions that can uniquely enhance education, collaborative design, health care, and social interaction, all from a distance. Further benefits include highly immersive computer interfaces, data visualization, and storytelling.
In spite of its potential, the VR industry has struggled through hype cycles as devices built from co-opted component technologies fail to meet the expectations of each generation. The struggle is largely due to the lack of a rigorous foundation that draws from the human-centered sciences of perceptual psychology, neuroscience, and physiology. We are working to reduce the gap between engineering and these sciences by forging a new discipline called perception engineering. In this pioneering view, the object being engineered is the perceptual illusion itself, and the physical devices that achieve it are auxiliary.
We are developing mathematical foundations for perception engineering that unify engineering concepts from robotics and control theory with their biological-system counterparts, such as predictive coding and the free energy principle. A cornerstone is von Neumann-Morgenstern information spaces, introduced for games with hidden information. Our team includes experts from VR, neuroscience, perceptual psychology, robotics, control theory, and pure mathematics. The work will lay a valuable foundation for principled engineering approaches to design, simulation, prediction, and analysis of sustained, targeted perceptual experiences. This will offer valuable guidance and deeper insights into VR, robotics, and possibly the sciences that study perception.
In spite of its potential, the VR industry has struggled through hype cycles as devices built from co-opted component technologies fail to meet the expectations of each generation. The struggle is largely due to the lack of a rigorous foundation that draws from the human-centered sciences of perceptual psychology, neuroscience, and physiology. We are working to reduce the gap between engineering and these sciences by forging a new discipline called perception engineering. In this pioneering view, the object being engineered is the perceptual illusion itself, and the physical devices that achieve it are auxiliary.
We are developing mathematical foundations for perception engineering that unify engineering concepts from robotics and control theory with their biological-system counterparts, such as predictive coding and the free energy principle. A cornerstone is von Neumann-Morgenstern information spaces, introduced for games with hidden information. Our team includes experts from VR, neuroscience, perceptual psychology, robotics, control theory, and pure mathematics. The work will lay a valuable foundation for principled engineering approaches to design, simulation, prediction, and analysis of sustained, targeted perceptual experiences. This will offer valuable guidance and deeper insights into VR, robotics, and possibly the sciences that study perception.
The main objective has been to introduce a theoretical foundation that helps lift VR to a more established engineering discipline, with related objectives as gaining deeper insights into perception engineering from unification attempts, better explaining and predicting existing VR phenomena through specializations, establishing benefits of VR for robots and other autonomous systems, and having broader impacts on education and industry. Our interdisciplinary team of post-doctoral researchers in the areas of mathematics, cognitive neuroscience, virtual reality, control theory, and robotics have worked collaboratively and arduously toward these goals. We have made substantial progress toward the main goal, with contributions growing for the related objectives.
For the main objective, the first year involved expanding the game-theoretic formulation from the proposal, based on von Neumann-Morgernstern information, into a general mathematical model of agency that applies to both biological and engineered systems. As originally proposed, an enactivist philosophy was followed to avoid the need for symbolic representations that might be common in engineering, but are unsuitable for biological systems. The mathematical framework is expressed at a very general, set-theoretic level, with mappings defined for external state spaces, internal information spaces, sensors, and actuation. One astonishing result is that we proved that minimal 'brains' always exist and are unique, for a well-specified set of tasks.
Once this model of agency was developed, the next step was to build upon it to define perception engineering, in a general, precise, and clear mathematical way using set theory and functions, so that it can be easily specialized to many practical applications. We defined two kinds of agents that operate within a shared environment: producer agents, who intentionally modify the environment to deliver targeted perceptual experiences to receiver agents. To constitute perception engineering (corresponding closely to our intuitive notion of VR), the perceptual experience must achieve two conditions: 1) it must be plausible, in the sense that the receiver does not receive unexpected or highly unusual stimuli, and 2) it must be illusory, meaning that the experience does not correspond to what is happening in reality (which has been carefully defined). Within the framework, we have introduced notions of plausibility robustness and forced fusion magnitude, both of which are inspired by concepts from VR literature but now applied in a rigorously defined way to both biological and engineered systems.
As originally proposed, the work involves both top-down development of general theory, discussed above, and bottom-up approaches, which involve experiments on specific systems that include humans and/or robots. Substantial progress has been made in this direction as well. We established that humans have incorrect expectations about interacting with the physical world when they are convinced that they are reduced in scale. We then determined the rates at which they adapt to abnormal gravity in basic throwing tasks. In several works, we gained a better understanding of how to make more plausible and comfortable tele-embodiment experiences using robots, panoramic cameras, and VR. These works involve studying discrepancies between human expectations and their targeted experience.
We have also invested substantial effort into developing the lab infrastructure and methodology to perform electroencephalogram (EEG)-based studies based on the well-known technique of event-related potentials (ERP) from neuroscience. We are one of the first groups in the world to adapt these techniques to VR studies, and they have allowed us to directly measure cognitive responses to stimuli presented in VR. This allows us to avoid many problems associated with the common method of having human participants fill out questionnaires after the experience. Our first study based on this technique characterizes the degradation of attention due to the onset of VR sickness.
For the main objective, the first year involved expanding the game-theoretic formulation from the proposal, based on von Neumann-Morgernstern information, into a general mathematical model of agency that applies to both biological and engineered systems. As originally proposed, an enactivist philosophy was followed to avoid the need for symbolic representations that might be common in engineering, but are unsuitable for biological systems. The mathematical framework is expressed at a very general, set-theoretic level, with mappings defined for external state spaces, internal information spaces, sensors, and actuation. One astonishing result is that we proved that minimal 'brains' always exist and are unique, for a well-specified set of tasks.
Once this model of agency was developed, the next step was to build upon it to define perception engineering, in a general, precise, and clear mathematical way using set theory and functions, so that it can be easily specialized to many practical applications. We defined two kinds of agents that operate within a shared environment: producer agents, who intentionally modify the environment to deliver targeted perceptual experiences to receiver agents. To constitute perception engineering (corresponding closely to our intuitive notion of VR), the perceptual experience must achieve two conditions: 1) it must be plausible, in the sense that the receiver does not receive unexpected or highly unusual stimuli, and 2) it must be illusory, meaning that the experience does not correspond to what is happening in reality (which has been carefully defined). Within the framework, we have introduced notions of plausibility robustness and forced fusion magnitude, both of which are inspired by concepts from VR literature but now applied in a rigorously defined way to both biological and engineered systems.
As originally proposed, the work involves both top-down development of general theory, discussed above, and bottom-up approaches, which involve experiments on specific systems that include humans and/or robots. Substantial progress has been made in this direction as well. We established that humans have incorrect expectations about interacting with the physical world when they are convinced that they are reduced in scale. We then determined the rates at which they adapt to abnormal gravity in basic throwing tasks. In several works, we gained a better understanding of how to make more plausible and comfortable tele-embodiment experiences using robots, panoramic cameras, and VR. These works involve studying discrepancies between human expectations and their targeted experience.
We have also invested substantial effort into developing the lab infrastructure and methodology to perform electroencephalogram (EEG)-based studies based on the well-known technique of event-related potentials (ERP) from neuroscience. We are one of the first groups in the world to adapt these techniques to VR studies, and they have allowed us to directly measure cognitive responses to stimuli presented in VR. This allows us to avoid many problems associated with the common method of having human participants fill out questionnaires after the experience. Our first study based on this technique characterizes the degradation of attention due to the onset of VR sickness.
We consider our new mathematical framework that explains VR and perception engineering to be the most significant achievement of the project to date. It is a breakthrough because to the best or our knowledge there has been no formal characterization of what VR actually is. Without this, the research field is often prone to the ebbs and flows of business and technology hypes, without grounding in a stable, scientific basis that tend to support other, more established fields. We expect to further extend and adapt this theory, using to explain existing phenomena in the context of VR applied to both biological and engineered systems.