Hybrid Extended reAliTy

Projektinformationen

HEAT

ID Finanzhilfevereinbarung: 101135637

DOI

10.3030/101135637

EK-Unterschriftsdatum 2 April 2024

Startdatum 1 Juni 2024

Enddatum 31 Mai 2027

Finanziert unter

• Digital, Industry and Space

Gesamtkosten

€ 6 993 207,50

EU-Beitrag

€ 6 993 207,50

Koordiniert durch

UNIVERSITA DEGLI STUDI DI CAGLIARI
Italy

Periodic Reporting for period 1 - HEAT (Hybrid Extended reAliTy)

Berichtszeitraum: 2024-06-01 bis 2025-11-30

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

The primary objective of WP1 (Management) is to ensure coordination, cooperation and cohesion in all aspects of the project’s lifetime ensuring that all the foreseen activities are carried out accordingly with the approved work plan.
The objective of WP2 (Requirements, Scenarios, Platform Architecture and QoE Methodology) is to lay the foundation for the project by defining: (i) the target use cases and scenarios; (ii) the resulting requirements; (iii) the platform architecture and (iv) the methodology for the evaluation of the QoE.
The objective of WP3 (Technological Components and Architectural Integration) is the design and development of the technological components required for the realisation of the HEAT project and their integration in the proposed architecture. It includes the preparation and deployment of the involved technologies.
The WP4 (Pilots, Content Production and Evaluation) will validate the project contributions in four pilot actions of increasing complexity and will demonstrate their benefits by opportunely applying appropriate QoE evaluation methodologies developed in WP2.
The core objectives of WP5 (Dissemination and Impact Maximisation) can be summarised as follows: (i) Design and implement a Dissemination & Communication (D&C) strategy to maximise the visibility and outreach of the project outcomes; (ii) Publish scientific publications in conferences and journals, monitor and prepare contributions relevant to standardisation activities, provide open-source contributions, and participate in industrial events (e.g. with talks and demos); and (iii) Determine overall conditions for successful exploitation and knowledge transfer, by exploring IPR generation and shares, identifying licensing opportunities and creating synergistic collaborations with other partners from the same or different funding opportunities.
The objective of WP6 (Ethics requirements) is to ensure compliance with the 'ethics requirements' set out in this work package.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

HEAT will go beyond the state of the art:
- by providing novel technological solutions to enable the agile volumetric (photo-) realistic real-time capture by using a set of off-the-shelf and low-cost sensors and achieving a (semi-) automatic calibration process.
- by developing a holographic representation that can embrace NeRFs and pave the way for the use of standardised processing and coding techniques. In addition, the results obtained will be proposed for standardisation within the main regularisation bodies (JPEG, MPEG, ETSI, IEEE).
- by enabling an effective and strategic integration of heterogeneous content modalities and formats, overcoming challenges not only in terms of performance, but also in terms of storytelling and providing increased feelings of realism and immersion beyond 3DoF+ scenarios, at reduced costs.
- by further investigating the use of miniature olfaction devices and proposing solutions for seamless integration of scents into immersive XR experiences.
- by developing feasible means of conveying and rendering meaningful user feedback to all involved users in both virtual and mixed modes.
- by devising and adopting novel semantic-, position- and viewport-aware strategies for content encoding and segmentation, exploiting AI-based techniques, like autoencoders, as well as denoising and completion methods.
- by exploiting the capabilities of network virtualization and Cloud Continuum (e.g. Edge Computing), to maximise scalability and interoperability
- by offloading heavy processing functions from the clients to virtualized in-cloud media processing functions.
- by devising, implementing and validating novel low-latency and scalable streaming strategies.
- by effectively supporting media synchronisation and prioritisation methods.

The main components of HEAT and their interaction

Pilot development timeline

HEAT high-level system architecture