Project description
XR experiences combining vision and sound
Extended reality (XR) technologies are on the verge of dominating the human-computer interaction (HCI) scene by overtaking traditional approaches. Two other fields that are experiencing similar blooming are natural language processing (NLP) and computer vision (CV), mainly due to the emergence of data-driven methods in the areas of machine learning (ML) and artificial intelligence (AI). VOXreality aspires to fuse these parallel fields to design and develop AI-models that will integrate language as a core interaction medium, together with visual understanding. The focus is on producing pre-trained XR-models entangling the spatial and semantic knowledge of XR and NLP systems. This could kick-start a new era of applications built around the holistic understanding of the users’ goals, away from devices and controllers.
Objective
VOXReality is an ambitious project whose goal will be to facilitate and exploit the convergence of two important technologies, natural language processing (NLP) and computer vision (CV). Both technologies are experiencing a huge performance increase due to the emergence of data-driven methods, specifically machine learning (ML) and artificial intelligence (AI). On the one hand, CV/ML are driving the extended reality (XR) revolution beyond what was possible up to now, and, on the other, speech-based interfaces and text-based content understanding are revolutionising human-machine and human-human interaction. VOXReality will employ an economical approach to combine these two. VOXReality will pursue the integration of language- and vision-based AI models with either unidirectional or bidirectional exchanges between the two modalities. Vision systems drive both AR and VR, while language understanding adds a natural way for humans to interact with the back-ends of XR systems or create multimodal XR experiences combining vision and sound. The results of the project will be twofold: 1) a set of pretrained next-generation XR models combining, in various levels and ways, language and vision AI and enabling richer, more natural immersive experiences that are expected to boost XR adoption, and 2) a set of applications using these models to demonstrate innovations in various sectors. The above technologies will be validated through three use cases: 1) Personal Assistants that are an emerging type of digital technology that seeks to support humans in their daily tasks, with their core functionalities related to human-to-machine interaction; 2) Virtual Conferences that are completely hosted and run online, typically using a virtual conferencing platform that sets up a shared virtual environment, allowing their attendees to view or participate from anywhere in the world; 3) Theaters where VOXReality will combine language translation, audiovisual user associations and AR VFX triggered by predetermined speech.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: https://op.europa.eu/en/web/eu-vocabularies/euroscivoc.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: https://op.europa.eu/en/web/eu-vocabularies/euroscivoc.
- social sciences political sciences political transitions revolutions
- humanities arts performing arts dramaturgy
You need to log in or register to use this function
We are sorry... an unexpected error occurred during execution.
You need to be authenticated. Your session might have expired.
Thank you for your feedback. You will soon receive an email to confirm the submission. If you have selected to be notified about the reporting status, you will also be contacted when the reporting status will change.
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.2.4 - Digital, Industry and Space
MAIN PROGRAMME
See all projects funded under this programme -
HORIZON.2.4.6 - Next Generation Internet
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-RIA - HORIZON Research and Innovation Actions
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) HORIZON-CL4-2021-HUMAN-01
See all projects funded under this callCoordinator
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
47822 SANTARCANGELO DI ROMAGNA
Italy
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.