Hybrid Extended reAliTy

Project Information

HEAT

Grant agreement ID: 101135637

DOI

10.3030/101135637

EC signature date 2 April 2024

Start date 1 June 2024

End date 31 May 2027

Funded under

• Digital, Industry and Space

Total cost

€ 6 993 207,50

EU contribution

€ 6 993 207,50

Coordinated by

UNIVERSITA DEGLI STUDI DI CAGLIARI
Italy

Periodic Reporting for period 1 - HEAT (Hybrid Extended reAliTy)

Reporting period: 2024-06-01 to 2025-11-30

Summary of the context and overall objectives of the project

Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The primary objective of WP1 (Management) is to ensure coordination, cooperation and cohesion in all aspects of the project’s lifetime ensuring that all the foreseen activities are carried out accordingly with the approved work plan.
The objective of WP2 (Requirements, Scenarios, Platform Architecture and QoE Methodology) is to lay the foundation for the project by defining: (i) the target use cases and scenarios; (ii) the resulting requirements; (iii) the platform architecture and (iv) the methodology for the evaluation of the QoE.
The objective of WP3 (Technological Components and Architectural Integration) is the design and development of the technological components required for the realisation of the HEAT project and their integration in the proposed architecture. It includes the preparation and deployment of the involved technologies.
The WP4 (Pilots, Content Production and Evaluation) will validate the project contributions in four pilot actions of increasing complexity and will demonstrate their benefits by opportunely applying appropriate QoE evaluation methodologies developed in WP2.
The core objectives of WP5 (Dissemination and Impact Maximisation) can be summarised as follows: (i) Design and implement a Dissemination & Communication (D&C) strategy to maximise the visibility and outreach of the project outcomes; (ii) Publish scientific publications in conferences and journals, monitor and prepare contributions relevant to standardisation activities, provide open-source contributions, and participate in industrial events (e.g. with talks and demos); and (iii) Determine overall conditions for successful exploitation and knowledge transfer, by exploring IPR generation and shares, identifying licensing opportunities and creating synergistic collaborations with other partners from the same or different funding opportunities.
The objective of WP6 (Ethics requirements) is to ensure compliance with the 'ethics requirements' set out in this work package.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

HEAT will go beyond the state of the art:
- by providing novel technological solutions to enable the agile volumetric (photo-) realistic real-time capture by using a set of off-the-shelf and low-cost sensors and achieving a (semi-) automatic calibration process.
- by developing a holographic representation that can embrace NeRFs and pave the way for the use of standardised processing and coding techniques. In addition, the results obtained will be proposed for standardisation within the main regularisation bodies (JPEG, MPEG, ETSI, IEEE).
- by enabling an effective and strategic integration of heterogeneous content modalities and formats, overcoming challenges not only in terms of performance, but also in terms of storytelling and providing increased feelings of realism and immersion beyond 3DoF+ scenarios, at reduced costs.
- by further investigating the use of miniature olfaction devices and proposing solutions for seamless integration of scents into immersive XR experiences.
- by developing feasible means of conveying and rendering meaningful user feedback to all involved users in both virtual and mixed modes.
- by devising and adopting novel semantic-, position- and viewport-aware strategies for content encoding and segmentation, exploiting AI-based techniques, like autoencoders, as well as denoising and completion methods.
- by exploiting the capabilities of network virtualization and Cloud Continuum (e.g. Edge Computing), to maximise scalability and interoperability
- by offloading heavy processing functions from the clients to virtualized in-cloud media processing functions.
- by devising, implementing and validating novel low-latency and scalable streaming strategies.
- by effectively supporting media synchronisation and prioritisation methods.

The main components of HEAT and their interaction

Pilot development timeline

HEAT high-level system architecture