Periodic Reporting for period 1 - XReco (XR mEdia eCOsystem)
Reporting period: 2022-09-01 to 2024-02-29
Modern media have transitioned away from their single-dimensional and unidirectional nature (as traditional broadcast, on-demand movies, printed press). Particularly the possibility to present media spatially, through eXtended Reality (XR) technologies, opens new possibilities of bringing across media content in a more captivating and immersive manner. XReco seeks to address these opportunities by enabling efficient media data and especially XR content creation, sharing, discovery, recommendations, and transformations, involving different types and sizes of media organisations.
The project's main goal is to establish a data-driven approach for fostering development and production of novel XR media experiences by delivering a platform for inter-organisation data sharing. Content and data providers can benefit from distributing their data (monetisation but also access to external contributions) and media creators will benefit from increased access to content. This will allow them to create novel media content and XR experiences using the platform’s reuse services, faster, with less effort and better addressing their consumers.
One of the core goals is to address a diverse range of stakeholders, from large organisations to SMEs, as well as across industries and thus contributing to an XR ecosystem. Feedback from the XR community has already proven the relevance of the wide variety of activities the XReco consortium pursues.
The project's main goal is to establish a data-driven approach for fostering development and production of novel XR media experiences by delivering a platform for inter-organisation data sharing. Content and data providers can benefit from distributing their data (monetisation but also access to external contributions) and media creators will benefit from increased access to content. This will allow them to create novel media content and XR experiences using the platform’s reuse services, faster, with less effort and better addressing their consumers.
One of the core goals is to address a diverse range of stakeholders, from large organisations to SMEs, as well as across industries and thus contributing to an XR ecosystem. Feedback from the XR community has already proven the relevance of the wide variety of activities the XReco consortium pursues.
Based on the available technologies and competencies, the user partners represented in this project have formulated their requirements. In several iterations, these were validated by focus groups, market research, and consultations with potential users and customers of the B2B platform the consortium aims to build an XR ecosystem for creating, managing, and sharing XR assets and experiences. Building upon this solid foundation, the technology partners adapted the technology R&D roadmap to the needs of the users, designed a system architecture, developed a first set of components and services, and integrated it to the extent that, 18 months into the project, user partners have started using the core platform and its individual components to create their demonstrator applications. The first demonstrator phase, which will end in May 2024, will provide concrete feedback on the platform. The XR experiences produced by the use case partners with the help of the XReco technologies will not only show what the aspired B2B platform will enable future customers to do. They will also serve to demonstrate the tools and workflows that led to these results. Currently, the platform provides (1) the Neural Media Repository (NMR) that exploits neural based technologies to enable search and retrieval of multimodal content from the repository of the platform or from connected external repositories, such as broadcast archives or public content collections. Several media domains have already been included in the system and are now available for search, such as image, video and 3D content. (2) The search front-end is integrated with a content marketplace, facilitating content exchange. A first version of the tools for defining content licenses and registering them on a blockchain is available. (3) XReco is developing a set of solutions for the creation of novel media content, incl. techniques such as 3D Neural Reconstructions (NeRF and SfM), Free Viewpoint Video, Offline Volumetric Video and Holoportation. (4) To facilitate XR content and experience creation, we are developing XR Capsule, a no-code easy-to-use solution that any creator can use without prior experience in Extended Reality, and provide a simplified guided interface to Unity 3D, both applicable to a wide range of sectors. Finally, the AR CMS and editor will enable users to create location-based experiences fast and easily. This work is complemented by an in-depth analysis of the legal framework, in particular concerning privacy and copyright, and created a list of risks and opportunities for each for the core functionalities of the project to support risk management.
A core aim of XReco is to support widespread uptake by making available technologies more affordable and easy to use, rather than developing something new from scratch. Thus, the main achievement beyond the state of the art is to adapt state of the art technologies to provide no-code services to enable people without coding skills to produce compelling XR assets and experiences.
Beyond that, tThe technical work produced some results advancing the state of the art:
• Advances in multimodal search, VR- and MR-based search interfaces, and contributions to improving the evaluation methodologies in leading interactive search benchmarks.
• Advances in real-time free viewpoint video and holoportation pipelines, including integration with the Unity Authoring Tool
• Modification of NeERF pipelines to improve training on custom data and facilitate parameterisation.
Beyond that, tThe technical work produced some results advancing the state of the art:
• Advances in multimodal search, VR- and MR-based search interfaces, and contributions to improving the evaluation methodologies in leading interactive search benchmarks.
• Advances in real-time free viewpoint video and holoportation pipelines, including integration with the Unity Authoring Tool
• Modification of NeERF pipelines to improve training on custom data and facilitate parameterisation.