Periodic Reporting for period 1 - TOUCHLESS (Touchless Haptic Experiences with Neurocognitive AI)
Reporting period: 2021-01-01 to 2021-12-31
The social distancing measures introduced to restrict the spread of coronavirus and help halt the COVID-19 pandemic have made us realise just how important interpersonal physical contact is to our overall well-being, and which we are currently missing. Even though there are many advanced haptic technologies, they cannot replace tactile interaction that enables bonding physical sensation of care through the stimulation of skin receptors during touching. This project aims to overcome that challenge through development of a next-generation touchless haptic technology. In addition to new hardware, it will construct neurocognitive models of experience and a novel AI framework to enable touchless digital inducement of the sensation of touch through the receptor response. It will also enable soothing, affective, social and cognitive experiences, opening up a variety of new digital interactions and the affective enhancement of everyday applications.
In our first year, the TOUCHLESS project has made good progress in multiple directions relating to our overarching vision of touchless technologies and affective experiences. Namely, we have advanced the state of the art of ultrasonic mid-air haptic devices and published open-source guidelines of how to build your own display. We have performed delicate psychophysical experiments to improve our understanding of touch and the receptors involved. We have developed AI platforms and algorithms to remotely detect a person’s bio-signals (e.g. heartrate) thereby getting a glimpse into their current emotional state. We have developed demonstrators of touchless e-commerce interactions at a vending machine and have created a haptic vocabulary of those experiences and tactile sensations. Finally, we have brainstormed and fleshed out multiple applications and use cases of the technology we are developing and responsibly reflected on its ethical implications towards society. These activities can help increase the effectiveness and impact of our research. Some of the results mentioned above are already published on our website ( www.touchlessAI.eu ) while others are still being researched in our labs.
More specifically, here below is the work that has been performed during the first 12 months of TOUCHLESS in technical WPs:
- WP1: Defined the general experimental setup in D1.1 by gathering feedback from the project partners and the early experience that we have had when facing the design of future experiments. Published instructions on how to build an ultrasonic phased-array for haptic stimulation with extensive work carried out on the simulation, electronic design, and software but also on video editing for the creation of the step-by-step instructions. Designed, built and tested a large ultrasonic array called GranCapitan, capable of generating sensations on the back of the hand and the forearm. Simulations on the technique of PhantomFields.
- WP2: Preparation of custom stimuli for mid-air psychophysics testing. Pre-registration and data collection of a frequency-resolve and intensity-balanced study of spatial resolution in human mid-air tactile sensations. Preliminary investigation of rich haptic experiences and psychometric evaluation using a structured qualitative model of human sensory experience. Specification, design and preliminary study of mid-air haptic role in high-level user experiences such as user affect and agency.
- WP3: Development of AI Operational Infrastructure for efficient collaboration and research code exchange between multiple consortium participants: the resulting infrastructure is based on Microsoft Azure platform and allows for efficient codebase management, data management, AI model training and monitoring. Elaboration of a high-level architecture for a Hybrid AI system that combines machine perception and modulation blocks. Design of the Interface layers which abstract away from a developer low level signal processing features via API/SDK and allow for basic integration of new modules. Running initial research activities on experience sensing and modulation focused on building initial demonstrator that utilizes AI engine: to this end we designed a fully touchless experience communication system and run an initial user study.
- WP4: First version of the Touch Style Library created and used to build a demonstrator. Basic research into the design of touchless buttons and stimulations for relevant end-user experiences conducted and related to the Touch Style Library.
- WP5: Developed a novel training offer by combining generic RRI training with specific elements relevant to haptic/body engineering research and technology sectors. The learnings of RRI training were carried forward into the core research activities of the project (WP4). The specific ethical challenges of haptic/body engineering were identified, and used to plan an ethics research agenda suitable for embedding within the R&I activities of the TOUCHLESS partners.
- WP6: Several dissemination and outreach activities related to the project results and vision, active engagement with sister projects. Live content available on project website, and planning regular updates with content, e.g. new publications, videos, blogs, and project updates. Monitoring of website traffic through analytics: approximately 1700 unique visitors. Launch of the Haptics Helix in December, a great opportunity to network and accelerate the impact of the project. Creation of a comprehensive cross-project collaboration plan and engagement with sister projects on several occasions. Organisation of a workshop at a leading Haptics conference in July (IEEE WorldHaptics), thereby raising awareness about our project and engaging with the scientific community. Active and diverse list of communication and outreach activities, including the launch of a Haptix Helix, reaching a total of 29,000 people across the world.
More specifically, here below is the work that has been performed during the first 12 months of TOUCHLESS in technical WPs:
- WP1: Defined the general experimental setup in D1.1 by gathering feedback from the project partners and the early experience that we have had when facing the design of future experiments. Published instructions on how to build an ultrasonic phased-array for haptic stimulation with extensive work carried out on the simulation, electronic design, and software but also on video editing for the creation of the step-by-step instructions. Designed, built and tested a large ultrasonic array called GranCapitan, capable of generating sensations on the back of the hand and the forearm. Simulations on the technique of PhantomFields.
- WP2: Preparation of custom stimuli for mid-air psychophysics testing. Pre-registration and data collection of a frequency-resolve and intensity-balanced study of spatial resolution in human mid-air tactile sensations. Preliminary investigation of rich haptic experiences and psychometric evaluation using a structured qualitative model of human sensory experience. Specification, design and preliminary study of mid-air haptic role in high-level user experiences such as user affect and agency.
- WP3: Development of AI Operational Infrastructure for efficient collaboration and research code exchange between multiple consortium participants: the resulting infrastructure is based on Microsoft Azure platform and allows for efficient codebase management, data management, AI model training and monitoring. Elaboration of a high-level architecture for a Hybrid AI system that combines machine perception and modulation blocks. Design of the Interface layers which abstract away from a developer low level signal processing features via API/SDK and allow for basic integration of new modules. Running initial research activities on experience sensing and modulation focused on building initial demonstrator that utilizes AI engine: to this end we designed a fully touchless experience communication system and run an initial user study.
- WP4: First version of the Touch Style Library created and used to build a demonstrator. Basic research into the design of touchless buttons and stimulations for relevant end-user experiences conducted and related to the Touch Style Library.
- WP5: Developed a novel training offer by combining generic RRI training with specific elements relevant to haptic/body engineering research and technology sectors. The learnings of RRI training were carried forward into the core research activities of the project (WP4). The specific ethical challenges of haptic/body engineering were identified, and used to plan an ethics research agenda suitable for embedding within the R&I activities of the TOUCHLESS partners.
- WP6: Several dissemination and outreach activities related to the project results and vision, active engagement with sister projects. Live content available on project website, and planning regular updates with content, e.g. new publications, videos, blogs, and project updates. Monitoring of website traffic through analytics: approximately 1700 unique visitors. Launch of the Haptics Helix in December, a great opportunity to network and accelerate the impact of the project. Creation of a comprehensive cross-project collaboration plan and engagement with sister projects on several occasions. Organisation of a workshop at a leading Haptics conference in July (IEEE WorldHaptics), thereby raising awareness about our project and engaging with the scientific community. Active and diverse list of communication and outreach activities, including the launch of a Haptix Helix, reaching a total of 29,000 people across the world.
The consortium of partners in Touchless consists of experts in hardware, psychophysics, neuroscience, AI, Human Computer Interaction, and responsible innovation. Collectively however, our ambition is to go beyond incremental improvements in each of our expertise and to create synergies that shape a whole new paradigm of intelligent touch. Specifically, we want to enable computer systems to intelligently re-create the tactile experiences that were previously lost in the virtual transition such as the feelings of agency, bonding, and attachment. This technological possibility has far reaching implications for online platforms that facilitate social interactions, e.g. in gaming, healthcare, education, and autonomous systems (e.g. cars, assistants, robots). This is also why we aim to develop the first ethics framework for theorizing ethical considerations around remote, contactless interactions with the human body and the self. Finally, our project results aim to spawn opportunities within Europe for technology entrepreneurship and fuel academic and industry research on novel extended reality (XR) applications in a range of sectors.