The VOXReality project orients to the delivery of AI models and XR applications exploiting those models. First and foremost, it develops pretrained AI models on (a) automatic speech recognition (ASR) that provide textual outputs from user speech, (b) neural machine translation (NMT) that provide translation text to given textual inputs in other languages, (c) visual language models (VLM) that provide textual outputs from visual content, and (d) generative language models in the form of conversation agents (CA) to assist with instructional training and indoor navigation. These models are properly trained and openly available under permissive licensing schemes so as to address the needs of interested stakeholders that intend to use them in various settings and domains. Secondly, it provides – through the Open Calls - funds for (a) integrating models into new XR applications, (b) extending models / tools, and (c) performing both activities. Third, the VOXReality consortium has identified four deployment options to be provided to deploy the components developed as a result of the project activities:
1. Source code: Inference codes to be open to allow modifications and deployment of the components with the desired configuration of the users
2. Containerization: Docker containers that enable the deployment of the components as API services with minimum intervention from the users required
3. Kubernetes: Means to deploy the components for the application pipelines that utilize Kubernetes framework
4. ONNX: Models in the ONNX format to ensure easy installation of the models into Unity applications.
Fourth, the VOXReality models are compliant with the AI Act whereas the datasets collected follow the provisions of the GDPR. Project's results can be exploited so as to change the way business and entertainment are conducted. VR Conference can be employed to facilitate communication even if the users do not speak a common language and get assistance by conversation agents in a remote venue. Using the Training Assistant, engineers may continuously update their skills without having to move and get supported by conversation agents that can support them. In AR theatres, foreign visitors may attend local language theatrical plays and experience the play in their language, enjoying visual effects driven by actor’s speech.
Furthermore, it manages to create strategic impact to the increasing need of the EU market to provide viable alternatives and innovative services that will enable European citizens to continue their activities in the post pandemic landscape. The language models and applications demonstrate their perspectives, values and requirements in terms of privacy and security but also as open science outcomes since they are delivered publicly through widely known repositories. Moreover, it supports the European landscape to maintain its presence in the XR applications market through the provision of new interactive services based on the AI VOXReality language models. Alongside, it supports European SMEs through its Open Calls and the open available models to gain competitiveness in the AI and XR applications global market, following the EU guidelines provided through the AI Act. Indirectly, it contributes to the objective of the EU of having 20 million ICT professionals in Europe by 2030, through the employment in the project team of ICT researchers with advanced digital skills and supporting 3 PhD theses candidates. The Scientific impact of VOXReality is addressed through the numerous publications as well as the open delivery of the project results. Additionally, it contributes to the objective of the Green Deal as the AI models and XR applications it delivers support respectively active remote participation to conferences and the effective remote development of skills of engineer professionals. Last, it creates a cultural impact as through the AR theatre creates new immersive multilingual experiences to audiences that was not possible for them beforehand to enjoy them.
