Periodic Reporting for period 4 - AMPLIFY (Amplifying Human Perception Through Interactive Digital Technologies)
Período documentado: 2020-12-01 hasta 2022-09-30
Technologies can be created that seamlessly augment and enhance human perception and cognition. In this project, we aim to increase our understanding of such technologies on a technical and conceptual level, but also in terms of their impact on the individual, the perception of an individual using such technologies, and also on a societal level.
Augmenting, amplifying, and extending human perception will open new perspectives for human interaction with the environment, and these advances have the potential to revolutionize human perception without creating information overload. They will enhance what they see, hear, and feel - this goes beyond perception and is fundamental to our experience and ultimately the way we think.
The project has four main objectives: (1) foundations for implementing new human senses, (2) foundations for augmenting human senses, (3) user interfaces for explicit and implicit control, and (4) measures for quantifying the effectiveness and quality of perceptual augmentation. Using a human-centered and experimental approach that relies heavily on the creation and study of functional prototypes, we conduct research in human-computer interaction and ubiquitous computing to achieve our goals. We employ an iterative design process with key elements of inspiration, exploration, and validation.
At its core, the project is a computer science research topic, but has strong interdisciplinary aspects. The key driver for innovation is the idea of amplifying the human mind and improving human perception, which is at the intersection of understanding humans and creating new technologies. Inspiration comes from human psychology and neuroscience. For the technical exploration and to create fundamental and usable systems, we need expertise in computer science and electronics. Our interdisciplinary research team includes scientists and students with backgrounds in neuroscience, psychology, electronics, and computer science.
During the development of the prototypes, we focused on the feasibility of creating artificial human senses that provide new perceptual channels to the human mind without increasing the experienced cognitive load of the person. Our focus was on creating intuitive and natural control mechanisms for the augmented senses using eye gaze, muscle activity, and brain signals.
In a human-centered research process, we have created a large number of prototypes and used them as a means of exploring and advancing our understanding of human augmentation. The specific prototypes and the empirical work with them are a vehicle for our conceptual research in human-computer interaction.
Through our publications, events, and workshops, we have fostered an international community researching human perceptual and cognitive augmentation and enhancement. In addition to the many technical contributions we have made, we have also advanced the methodology of human-computer interaction by discovering the placebo effect for AI-based technological augmentations.
We experimentally explored the enhancement of human vision and hearing. We built several functional prototypes and conducted studies. One case study was an augmented vision system that uses augmented reality to alter the speed of visual perception [1]. Navigation and spatial perception was another area we investigated. Here, a research probe was a device that provided implicit navigation information for swimming [2]. This allowed us to study how we can create a new sense and its impact on the user. We extended our work to the study of perception in VR. Here we discovered the Perspective Continuum, which outlines the design space for VR interfaces beyond the first and third perspectives [3]. To create intuitive and easy-to-use interfaces for human augmentation, we explored implicit input techniques using EEG and EMG. One result was the EMBody toolkit [4, 5], which simplifies experimentation with physiological inputs for control.
The amplification of human abilities raises ethical and societal questions. We have conducted research on the perception of human amplification and augmentation. In an international study, we captured people's expectations and feelings about digital technologies that augment human perceptual and cognitive abilities [6].
While conducting our technical and empirical research, we discovered the placebo effect of AI-based technological enhancements [7]. If participants in studies expect their abilities to be enhanced, even if they are not, they will behave and perform differently. This challenges current approaches and evaluation methods in human-computer interaction and interactive systems, and calls for the inclusion of a placebo condition in studies.
[1] https://doi.org/10.1145/3384657.3384659
[2] https://doi.org/10.1145/3290605.3300467
[3] https://doi.org/10.1145/3491102.3517447
[4] https://dl.acm.org/doi/10.1145/3457142
[5] https://github.com/HCUM/embody
[6] https://doi.org/10.1145/3544548.3581485
[7] https://doi.org/10.1145/3529225
An important methodological contribution is the discovery of the placebo effect for human augmentation technologies using AI. This finding was triggered by the many studies we conducted. This led to the hypothesis, and we then conducted a new experiment to validate it. This finding challenges the current practice of evaluating system performance in human-computer interaction. We are currently running workshops and tutorials to educate follow-on researchers about this, and suggest that studies need an additional placebo condition.
On the conceptual and theoretical side, we have advanced our understanding of the feasibility and limitations of augmenting human senses, systematically explored a design space for implicit control of sensors, and advanced methods for developing and evaluating sensor-based human augmentations.