The main objective has been to introduce a theoretical foundation that helps lift VR to a more established engineering discipline, with related objectives as gaining deeper insights into perception engineering from unification attempts, better explaining and predicting existing VR phenomena through specializations, establishing benefits of VR for robots and other autonomous systems, and having broader impacts on education and industry. Our interdisciplinary team of post-doctoral researchers in the areas of mathematics, cognitive neuroscience, virtual reality, control theory, and robotics have worked collaboratively and arduously toward these goals. We have made substantial progress toward the main goal, with contributions growing for the related objectives.
For the main objective, the first year involved expanding the game-theoretic formulation from the proposal, based on von Neumann-Morgernstern information, into a general mathematical model of agency that applies to both biological and engineered systems. As originally proposed, an enactivist philosophy was followed to avoid the need for symbolic representations that might be common in engineering, but are unsuitable for biological systems. The mathematical framework is expressed at a very general, set-theoretic level, with mappings defined for external state spaces, internal information spaces, sensors, and actuation. One astonishing result is that we proved that minimal 'brains' always exist and are unique, for a well-specified set of tasks.
Once this model of agency was developed, the next step was to build upon it to define perception engineering, in a general, precise, and clear mathematical way using set theory and functions, so that it can be easily specialized to many practical applications. We defined two kinds of agents that operate within a shared environment: producer agents, who intentionally modify the environment to deliver targeted perceptual experiences to receiver agents. To constitute perception engineering (corresponding closely to our intuitive notion of VR), the perceptual experience must achieve two conditions: 1) it must be plausible, in the sense that the receiver does not receive unexpected or highly unusual stimuli, and 2) it must be illusory, meaning that the experience does not correspond to what is happening in reality (which has been carefully defined). Within the framework, we have introduced notions of plausibility robustness and forced fusion magnitude, both of which are inspired by concepts from VR literature but now applied in a rigorously defined way to both biological and engineered systems.
As originally proposed, the work involves both top-down development of general theory, discussed above, and bottom-up approaches, which involve experiments on specific systems that include humans and/or robots. Substantial progress has been made in this direction as well. We established that humans have incorrect expectations about interacting with the physical world when they are convinced that they are reduced in scale. We then determined the rates at which they adapt to abnormal gravity in basic throwing tasks. In several works, we gained a better understanding of how to make more plausible and comfortable tele-embodiment experiences using robots, panoramic cameras, and VR. These works involve studying discrepancies between human expectations and their targeted experience.
We have also invested substantial effort into developing the lab infrastructure and methodology to perform electroencephalogram (EEG)-based studies based on the well-known technique of event-related potentials (ERP) from neuroscience. We are one of the first groups in the world to adapt these techniques to VR studies, and they have allowed us to directly measure cognitive responses to stimuli presented in VR. This allows us to avoid many problems associated with the common method of having human participants fill out questionnaires after the experience. Our first study based on this technique characterizes the degradation of attention due to the onset of VR sickness.
