Periodic Reporting for period 4 - AMPLIFY (Amplifying Human Perception Through Interactive Digital Technologies) Reporting period: 2020-12-01 to 2022-09-30 Summary of the context and overall objectives of the project The AMPLIFY project explores how digital technologies can enhance, augment, and amplify human perception. In the last decade, technical sensor systems have been developed that are superior to human perception, e.g. cameras can be built that provide higher spatial and temporal resolution than the human eye and cover a wider spectral range. But a human sense is much more than a sensor. Technologies can be created that seamlessly augment and enhance human perception and cognition. In this project, we aim to increase our understanding of such technologies on a technical and conceptual level, but also in terms of their impact on the individual, the perception of an individual using such technologies, and also on a societal level. Augmenting, amplifying, and extending human perception will open new perspectives for human interaction with the environment, and these advances have the potential to revolutionize human perception without creating information overload. They will enhance what they see, hear, and feel - this goes beyond perception and is fundamental to our experience and ultimately the way we think.The project has four main objectives: (1) foundations for implementing new human senses, (2) foundations for augmenting human senses, (3) user interfaces for explicit and implicit control, and (4) measures for quantifying the effectiveness and quality of perceptual augmentation. Using a human-centered and experimental approach that relies heavily on the creation and study of functional prototypes, we conduct research in human-computer interaction and ubiquitous computing to achieve our goals. We employ an iterative design process with key elements of inspiration, exploration, and validation. At its core, the project is a computer science research topic, but has strong interdisciplinary aspects. The key driver for innovation is the idea of amplifying the human mind and improving human perception, which is at the intersection of understanding humans and creating new technologies. Inspiration comes from human psychology and neuroscience. For the technical exploration and to create fundamental and usable systems, we need expertise in computer science and electronics. Our interdisciplinary research team includes scientists and students with backgrounds in neuroscience, psychology, electronics, and computer science.During the development of the prototypes, we focused on the feasibility of creating artificial human senses that provide new perceptual channels to the human mind without increasing the experienced cognitive load of the person. Our focus was on creating intuitive and natural control mechanisms for the augmented senses using eye gaze, muscle activity, and brain signals.In a human-centered research process, we have created a large number of prototypes and used them as a means of exploring and advancing our understanding of human augmentation. The specific prototypes and the empirical work with them are a vehicle for our conceptual research in human-computer interaction. Through our publications, events, and workshops, we have fostered an international community researching human perceptual and cognitive augmentation and enhancement. In addition to the many technical contributions we have made, we have also advanced the methodology of human-computer interaction by discovering the placebo effect for AI-based technological augmentations. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far Our research in this project followed the vision to create and study digital technologies that provide novel sensory experiences and new perceptual capabilities for humans in physical reality, augmented reality, and virtual reality. The central design goal was to create these new experiences while providing natural and intuitive control. We experimentally explored the enhancement of human vision and hearing. We built several functional prototypes and conducted studies. One case study was an augmented vision system that uses augmented reality to alter the speed of visual perception [1]. Navigation and spatial perception was another area we investigated. Here, a research probe was a device that provided implicit navigation information for swimming [2]. This allowed us to study how we can create a new sense and its impact on the user. We extended our work to the study of perception in VR. Here we discovered the Perspective Continuum, which outlines the design space for VR interfaces beyond the first and third perspectives [3]. To create intuitive and easy-to-use interfaces for human augmentation, we explored implicit input techniques using EEG and EMG. One result was the EMBody toolkit [4, 5], which simplifies experimentation with physiological inputs for control.The amplification of human abilities raises ethical and societal questions. We have conducted research on the perception of human amplification and augmentation. In an international study, we captured people's expectations and feelings about digital technologies that augment human perceptual and cognitive abilities [6].While conducting our technical and empirical research, we discovered the placebo effect of AI-based technological enhancements [7]. If participants in studies expect their abilities to be enhanced, even if they are not, they will behave and perform differently. This challenges current approaches and evaluation methods in human-computer interaction and interactive systems, and calls for the inclusion of a placebo condition in studies. [1] https://doi.org/10.1145/3384657.3384659 [2] https://doi.org/10.1145/3290605.3300467 [3] https://doi.org/10.1145/3491102.3517447 [4] https://dl.acm.org/doi/10.1145/3457142[5] https://github.com/HCUM/embody [6] https://doi.org/10.1145/3544548.3581485 [7] https://doi.org/10.1145/3529225 Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) We have created a number of functional prototypes that demonstrate the feasibility of creating amplified senses. These prototypes include visual and auditory augmentation as well as novel senses. Each of these prototypes contributes to human-computer interaction or ubiquitous computing. At the same time, these prototypes have enabled us to conduct human-centered empirical research on digital technologies for augmenting human perception. Here we have contributed to the understanding of how such novel interactive systems can be designed and implemented. In particular, seamless control and interaction with augmented senses is a challenge, and here we have contributed implicit and explicit interaction mechanisms and toolkits.An important methodological contribution is the discovery of the placebo effect for human augmentation technologies using AI. This finding was triggered by the many studies we conducted. This led to the hypothesis, and we then conducted a new experiment to validate it. This finding challenges the current practice of evaluating system performance in human-computer interaction. We are currently running workshops and tutorials to educate follow-on researchers about this, and suggest that studies need an additional placebo condition.On the conceptual and theoretical side, we have advanced our understanding of the feasibility and limitations of augmenting human senses, systematically explored a design space for implicit control of sensors, and advanced methods for developing and evaluating sensor-based human augmentations. User study with amplifyed directional hearing Sketch of a envisioned body-word device Sketch of a envisioned head-worn device Prototype of a device to provide a sense of orientation while swimming Prototype of an enhanced visual sense Supporting orientation in open water
