eXtended Reality Modelling (RIA)

Recent advances in the field of Artificial Intelligence (AI) giving machines the ability to understand and derive meaning from human languages, have shown that automatic systems can exhibit human‑like performance. Machine translation, speech recognition or personal assistants are now part of our daily lives. Recent progress in AI has also enabled systems to generalise from one task to another, from one language to another, from one modality to another. Large pre-trained multilingual language models can handle different languages, even with little or no training data. The same models can cover completely different language-related tasks, such as text translation or summarisation, speech transcription, or sentiment analysis. Natural language Understanding and Natural Language Generation state-of-art techniques are expected to take advantage of the latest advances in research. Advances in user and environment modelling and progress in data analytics allow systems to be increasingly context-aware and efficiently support users in their decisions.

Drawing on the above-mentioned recent advances, the proposals will:

• Develop pre‑trained eXtended Reality models capable of adapting to a large variety of forms of expression, interaction, languages, domains, styles and intent. Taking into account surrounding real or virtual environments, contexts, preferences and abilities of the user, the models will contribute to the general understanding of the environments and users’ knowledge, preferences, believes, abilities, intent and goals.
• Demonstrate the adaptation and generalisation of the eXtended Reality models, including through the integration of structured knowledge, by developing solutions capable of carrying genuine human-like interaction before, during and after an eXtended Reality experience.
• Integrate the solutions into several eXtended Reality use‑cases scenarios, such as media, collaborative telepresence, learning, personal assistants or information retrieval.

Beyond supporting a large set of languages and modalities, the work will focus on enabling new forms of interactions, avoiding bias, whilst ensuring accessibility, privacy, transparency and explainability.

To compensate the increase of model complexity, the proposed solutions should be energy efficient thanks to optimised protocols and algorithms with equivalent performance during both training and implementation.

The proposal will ensure reproducibility and repeatability of the research results, promote an open data and interfaces standardisation, avoiding narrow de-facto standards and demonstrate clear and efficient integration paths for the European industry take up.

To further extend the application domains, address sector specific constrains, ensure reproducibility and demonstrate their integration paths, proposals are expected to organise a number of competitive calls with financial support to third parties (FSTP) and further extend the use-cases. At least 20% of the funding should be dedicated to FSTP. To that aspect, the consortium will provide guidelines and technical support in engineering integration, testing and validation to support the development of such use-cases.