Skip to main content
Go to the home page of the European Commission (opens in new window)
English en
CORDIS - EU research results
CORDIS

Language Augmentation for Humanverse

Project description

Next-generation language augmented extended reality systems

The EU-funded LUMINOUS project aims to transform extended reality by using natural language and multimodal large language models to create more intuitive, personalised and adaptive user experiences. Users will be able to fluently interact with their environment, with instant access to global and domain-specific knowledge to complete tasks. Accompanied by visual aids and procedural steps, these tasks will be communicated through a speech interface or adaptive avatar, whose language style, facial expressions and attitude will assist generalised and situation-aware communication. Finally, three thematic prototypes of language augmented extended reality systems will be demonstrated and tested in connection with neurorehabilitation for speech-impaired stroke survivors, immersive industrial safety training and 3D architectural design.

Objective

LUMINOUS aims at the creation of the next generation of Language Augmented XR systems, where natural language-based communication and Multimodal Large Language Models (MLLM) enable adaptation to individual, not predefined user needs and unseen environments. This will enable future XR users to interact fluently with their environment, while having instant access to constantly updated global as well as domain- specific knowledge sources to accomplish novel tasks. We aim to exploit MLLMs injected with domain specific knowledge for describing novel tasks on user demand. These are then communicated through a speech interface and/or a task adaptable avatar (e.g. coach/teacher) in terms of different visual aids and procedural steps for the accomplishment of the task. Language driven specification of the style, facial expressions, and specific attitudes of virtual avatars will facilitate generalisable and situation-aware communication in multiple use cases and different sectors. LLMs will benefit in parallel in identifying new objects that were not part of their training data and then describing them in a way that they become visually recognizable. Our results will be prototyped and tested in three pilots, focussing on neurorehabilitation (support of stroke patients with language impairments), immersive industrial safety training, and 3D architectural design review. A consortium of six leading R&D institutes experts in six different disciplines (AI, Augmented Vision, NLP, Computer Graphics, Neurorehabilitation, Ethics) will follow a challenging workplan, aiming to bring about a new era at the crossroads of two of the most promising current technological developments (LLM/AI and XR), made in Europe.

Fields of science (EuroSciVoc)

Keywords

Programme(s)

Topic(s)

Funding Scheme

HORIZON-RIA - HORIZON Research and Innovation Actions

See all projects funded under this funding scheme

Call for proposal

(opens in new window) HORIZON-CL4-2023-HUMAN-01-CNECT

See all projects funded under this call

Coordinator

DEUTSCHES FORSCHUNGSZENTRUM FUR KUNSTLICHE INTELLIGENZ GMBH
Net EU contribution
€ 1 455 728,75
Address
TRIPPSTADTER STRASSE 122
67663 Kaiserslautern
Germany
Region
Rheinland-Pfalz Rheinhessen-Pfalz Kaiserslautern, Kreisfreie Stadt
Activity type
Research Organisations
Links
Total cost
€ 1 455 728,75

Participants (8)

Partners (4)