Hybrid Extended reAliTy

Información del proyecto

HEAT

Identificador del acuerdo de subvención: 101135637

DOI

10.3030/101135637

Fecha de la firma de la CE 2 Abril 2024

Fecha de inicio 1 Junio 2024

Fecha de finalización 31 Mayo 2027

Financiado con arreglo a

• Digital, Industry and Space

Coste total

€ 6 993 207,50

Aportación de la UE

€ 6 993 207,50

Coordinado por

UNIVERSITA DEGLI STUDI DI CAGLIARI
Italy

Periodic Reporting for period 1 - HEAT (Hybrid Extended reAliTy)

Período documentado: 2024-06-01 hasta 2025-11-30

Resumen del contexto y de los objetivos generales del proyecto

Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

The primary objective of WP1 (Management) is to ensure coordination, cooperation and cohesion in all aspects of the project’s lifetime ensuring that all the foreseen activities are carried out accordingly with the approved work plan.
The objective of WP2 (Requirements, Scenarios, Platform Architecture and QoE Methodology) is to lay the foundation for the project by defining: (i) the target use cases and scenarios; (ii) the resulting requirements; (iii) the platform architecture and (iv) the methodology for the evaluation of the QoE.
The objective of WP3 (Technological Components and Architectural Integration) is the design and development of the technological components required for the realisation of the HEAT project and their integration in the proposed architecture. It includes the preparation and deployment of the involved technologies.
The WP4 (Pilots, Content Production and Evaluation) will validate the project contributions in four pilot actions of increasing complexity and will demonstrate their benefits by opportunely applying appropriate QoE evaluation methodologies developed in WP2.
The core objectives of WP5 (Dissemination and Impact Maximisation) can be summarised as follows: (i) Design and implement a Dissemination & Communication (D&C) strategy to maximise the visibility and outreach of the project outcomes; (ii) Publish scientific publications in conferences and journals, monitor and prepare contributions relevant to standardisation activities, provide open-source contributions, and participate in industrial events (e.g. with talks and demos); and (iii) Determine overall conditions for successful exploitation and knowledge transfer, by exploring IPR generation and shares, identifying licensing opportunities and creating synergistic collaborations with other partners from the same or different funding opportunities.
The objective of WP6 (Ethics requirements) is to ensure compliance with the 'ethics requirements' set out in this work package.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

HEAT will go beyond the state of the art:
- by providing novel technological solutions to enable the agile volumetric (photo-) realistic real-time capture by using a set of off-the-shelf and low-cost sensors and achieving a (semi-) automatic calibration process.
- by developing a holographic representation that can embrace NeRFs and pave the way for the use of standardised processing and coding techniques. In addition, the results obtained will be proposed for standardisation within the main regularisation bodies (JPEG, MPEG, ETSI, IEEE).
- by enabling an effective and strategic integration of heterogeneous content modalities and formats, overcoming challenges not only in terms of performance, but also in terms of storytelling and providing increased feelings of realism and immersion beyond 3DoF+ scenarios, at reduced costs.
- by further investigating the use of miniature olfaction devices and proposing solutions for seamless integration of scents into immersive XR experiences.
- by developing feasible means of conveying and rendering meaningful user feedback to all involved users in both virtual and mixed modes.
- by devising and adopting novel semantic-, position- and viewport-aware strategies for content encoding and segmentation, exploiting AI-based techniques, like autoencoders, as well as denoising and completion methods.
- by exploiting the capabilities of network virtualization and Cloud Continuum (e.g. Edge Computing), to maximise scalability and interoperability
- by offloading heavy processing functions from the clients to virtualized in-cloud media processing functions.
- by devising, implementing and validating novel low-latency and scalable streaming strategies.
- by effectively supporting media synchronisation and prioritisation methods.

The main components of HEAT and their interaction

Pilot development timeline

HEAT high-level system architecture