Skip to main content

IMMERsive multi-modal interactive preSENCE

Objective

The main objective of this Integrated Project (IP) is to enable highly realistic multi-modal interactive immersion into virtual and augmented reality environments. Its focus will be on visual, haptic and auditive sensory components, while addressing an even broader range of human senses. Hand-based interaction will serve as a prototypical framework, but the multimodal feedback of all relevant information will be considered.

Three driving scenarios will cover the full spectrum of problems arising when implementing interactive environments: manipulation of an object by a human; direct contact between persons e.g. during a handshake or social dancing; and object mediated collaboration between humans, like jointly carrying a heavy object. These targets can only been reached by a concerted action, unifying competences in cognitive neuroscience, in psychophysics and in technology development. The indispensable scientific fundament will be established by basic research looking for new insights into the psychophysical and neurological mechanisms underlying multimodal interaction.

A novel, data-driven algorithmic paradigm will be investigated for the generation of sensory feedback. During a first recording stage all relevant data characterizing interaction are collected by multimodal sensing suites. The recorded data will be processed in order to convert the raw signals into abstract descriptors. This abstraction will not only enable to faithfully replay the interactive session using the collected data while replacing the real counterpart by a virtual one, but will provide the fundament for prediction schemes to create sensory feedback for interaction which never has been observed before. The resulting prototype systems will be systematically evaluated by novel measures of presence to be developed in the course of the project. The IP will lead directly to trend-setting applications in science, medical therapy, entertainment, training, and education with immense potential.

Funding Scheme

IP - Integrated Project

Coordinator

TECHNISCHE UNIVERSITAET MUENCHEN

Participants (9)

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE
France
Address
3, Rue Michel-ange
Paris Cedex 16
EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE ZÜRICH
Switzerland
Address
Raemistrasse 101
Zuerich
MAX PLANCK GESELLSCHAFT ZUR FOERDERUNG DER WISSENSCHAFTEN E.V.
Germany
Address
Hofgartenstrasse 8
Muenchen
TECHNION - ISRAEL INSTITUTE OF TECHNOLOGY
Israel
Address
Technion City
Haifa
THE UNIVERSITY OF BIRMINGHAM
United Kingdom
Address
Edgbaston
Birmingham
UNIVERSIDAD POLITECNICA DE MADRID
Spain
Address
Grupo De Tratamiento De Imágenes, E.t.s. Ing. De Telecomunicación, Avenida Ramiro De Maeztu 7
28040 Madrid
UNIVERSITA DI PISA
Italy
Address
Lungarno Pacinotti 43/44
Pisa
UNIVERSITAT POLITECNICA DE CATALUNYA
UNIVERSITE D'EVRY-VAL D'ESSONNE
France
Address
Boulevard Francois Mitterand 23
91025 Evry