Predictive Haptic COding Devices In Next Generation interfaces

Project Information

ph-coding

Grant agreement ID: 829186

DOI

10.3030/829186

Project closed

EC signature date 17 December 2018

Start date 1 January 2019

End date 30 September 2022

Funded under

• EXCELLENT SCIENCE - Future and Emerging Technologies (FET)

Total cost

€ 2 988 285,00

EU contribution

€ 2 988 285,00

Coordinated by

LUNDS UNIVERSITET
Sweden

Periodic Reporting for period 2 - ph-coding (Predictive Haptic COding Devices In Next Generation interfaces)

Reporting period: 2020-01-01 to 2022-09-30

Summary of the context and overall objectives of the project

PH-CODING has set the stage for the next generation of ultra-intuitive user interfaces for applications in virtual reality and remote operation by resolving critical scientific gaps and generating corresponding technological solutions in haptics. This target domain is in turn a key to future development of successful man-machine interfaces and machines being more intuitively interacting with humans. Developing intuitive user function requires understanding and appealing to the fundamentals of brain operation in its bodily interaction with the environment. The major difference between the brain and present-day interfaces is the incredibly rich and diversified representations that are possible in the brain.

The critical factor to solve this problem is to understand the foundations of haptic perception. This in turn contains two part problems, one is the mechanical dynamics of the skin and the other is the neural mechanisms implemented in the brain to predict, or mirror, these dynamics. PH-CODING achieved these goals by delivering new foundational models in both domains. These models were in turn applied to the development of new haptic user interfaces, bendable electronics solutions and autonomous intelligent network systems including an example of their physical instantiation in robotics.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The PH-CODING project investigated the foundations of tactile perception. To this end, the project integrated two interrelated subject areas: the mechanical dynamics of the skin and the neural mechanisms implemented in the brain to take advantage of these dynamics to achieve efficient tactile perception. It was found that the skin’s dynamics not only enhanced the information available from contact but also enabled the predictive mechanism that is behind perception. The findings of the studies carried out in the project have generated a variety of exploitable results, within automotive industry, VR/metaverse, and next generation robotics and AI technology.

Two major outputs of ph-coding are: (i) fundamental insights into the biological tactile sensing mechanisms of the skin underlying haptics and (ii) brain circuitry mechanisms of perception of touch (predictive coding). These results have been used to identify critical features in biological and articial skin, which has been developed in applications for virtual reality, and (ii) new critical design features for autonomous intelligent systems based on biological neuronal circuitry mechanisms of perception, which has been applied to generate novel AI technology and robotic systems with haptic intelligence. The project has provided a few examples of how such artificial skin, or human-machine haptics interfaces can be designed, while many further designs are made possible based on these principles. This set of desirable design features provide a seed for the field of sensorised flexible electronics to be applied to the sensing in mechanical interactions. The results on autonomous intelligent perception system provide the seed for future development of applied such systems, within robotics and other domains. These results have contributed to the generation of new haptic devices at the PH-CODING SME partner Actronika, the generation of a spin-off company in assistive and rehabilitation robotics (Human Robotix), and the generation of a spin-off company IntuiCell, a deeptech company within AI. PH-CODING partners have also been highly active in dissemination, both in the scientific and public domains, including organizing workshops and conferences on central PH-CODING themes.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

One the objectives of the ph-coding project was the direct application of the concept of predictive coding to the domain of haptic interfaces. Haptic interfaces combine movement and tactile information to create new types of human machine interfaces with applications in many domains such as vehicle cockpits, virtual reality technologies, accessibility, well-being, and many more domains. Partner company Actronika SAS [https://www.actronika.com/] has assiduously applied the results of the ph-coding project to create new types of haptic interfaces that provide movement-induced sensations characterised by a high degree of realism and which do not require any training from their users nor any attentional engagement to be used efficiently.

The study of the brain circuitry mechanisms underlying haptics perception has led to a foundational technology development in Artificial Intelligence based on predictive coding perception, rather than on machine learning-style inference. It is a breakthrough technology that promises qualitative advances in AI, and has the potential to outclass traditional artificial neural network (ANN) technology in many domains. The objective is to provide a platform to achieve Articial General Intelligence defined as including artificial perceptual processes into the inferential process. This breakthrough technology will make AI more useful and amenable to real world applications. IntuiCell AB [https://intuicell.com/] is a spin-off company created from ph-coding results that implements predictive coding technology. Being based on self-discovered, self-acquired knowledge in articial neural networks, this technology creates intelligent systems that are truly autonomous and applicable to a wide range of problem domains.

Both the skin technology and the network technology, as well as the central principle of predictive coding, have been implemented in a haptic intelligent robot system that can learn to optimally interact with its environment during movement.

As a bridge to the domain of robotics control, and the path towards autonomously intelligent robot systems, ph-coding work has also generated an interfacing technology, building on similar network technology as the novel AI perception technology above, but which map information about the domain of the body, or the robotic ‘plant’ configuration, to the domain of dynamic network representations. This generic circuitry model brain sensorimotor control within ph-coding represents a first step towards creating a new technology platform for future better integration between AI network technology and haptically intelligent robotic systems.

ph-coding summary