Skip to main content

Unified Transcription and Translation for Extended Reality

Objective

The aim of UTTER is to leverage large language models to build the next generation of multimodal eXtended reality (XR) technologies for transcription, translation, summarisation, and minuting. We will make these technologies scalable, adaptable, contextualised, robust, explainable, and emotion-aware. We will increase the context-sensitivity of the technologies, so they can take into account the full history of the conversation, as well as its wider context. We will introduce confidence-aware models, which can take into account their own limitations. We will develop explainable models, so the human user can know why the model made the decisions it did. We will improve adaptation, so that domain-specific and language-specific models can be quickly rolled out. For these advances we will make use of pre-trained eXtended reality (XR) models, which optimally combine text and speech signals, and are trained efficiently with adapters and prompting. We will also develop efficient methods to deploy such large and complex models, so that they can put into production in an energy-efficient manner. Our use-case prototypes will cover (i) A personal assistant for meetings that can improve communication in the online world and (ii) an advanced customer service assistant to support global markets. These prototypes will be developed and tested throughout the project, with annual releases and evaluations. Through our cascaded grant programme, and our release of tools to facilitate the use of pre-trained XR models, will enable the take-up and development of these technologies throughout Europe.

Coordinator

UNIVERSITEIT VAN AMSTERDAM
Net EU contribution
€ 1 998 946,89
Address
Spui 21
1012WX Amsterdam
Netherlands

See on map

Region
West-Nederland Noord-Holland Groot-Amsterdam
Activity type
Higher or Secondary Education Establishments
Other funding
€ 4 469,11

Participants (3)

UNBABEL UNIPESSOAL, LDA
Portugal
Net EU contribution
€ 750 000,00
Address
Rua Viscone De Santarem 67b
1000-286 Lisboa

See on map

SME

The organization defined itself as SME (small and medium-sized enterprise) at the time the Grant Agreement was signed.

Yes
Region
Continente Área Metropolitana de Lisboa Área Metropolitana de Lisboa
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00
INSTITUTO DE TELECOMUNICACOES
Portugal
Net EU contribution
€ 652 000,00
Address
Campus Universitario De Santiago Universidade De Aveiro
3810 193 Gloria E Vera Cruz

See on map

Activity type
Research Organisations
Other funding
€ 0,00
NAVER FRANCE
France
Net EU contribution
€ 669 375,00
Address
89 Rue Reaumur
75002 Paris

See on map

Region
Ile-de-France Ile-de-France Paris
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00

Partners (1)

THE UNIVERSITY OF EDINBURGH
United Kingdom
Net EU contribution
No data
Address
Old College, South Bridge
EH8 9YL Edinburgh

See on map

Region
Eastern Scotland Edinburgh
Activity type
Higher or Secondary Education Establishments
Other funding
No data