Enriching 360 media with 3D storytelling and personalisation elements

Project Information

Hyper360

Grant agreement ID: 761934

DOI

10.3030/761934

Project closed

EC signature date 2 June 2017

Start date 1 October 2017

End date 30 September 2020

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 4 464 292,50

EU contribution

€ 3 736 254,75

Coordinated by

ENGINEERING - INGEGNERIA INFORMATICA SPA
Italy

Periodic Reporting for period 2 - Hyper360 (Enriching 360 media with 3D storytelling and personalisation elements)

Reporting period: 2018-11-01 to 2020-09-30

Summary of the context and overall objectives of the project

The media sector is important for the economy since it provides high value added jobs and growth potential. The European media can build upon a history of excellent creativity - deeply rooted in the centenary European culture. However, it faces challenges from both traditional competitors and emerging ones.
New technologies can help the European media sector align to compete worldwide, while unleashing new forms of expressive creativity.
Hyper360 results being a complete solution for the capture, production, enhancement, delivery and consumption of free viewpoint videos (FVV) to OTT media, Envisioning the convergence of omnidirectional (360°) and 3D content for increasingly effective immersive experiences Hyper360 will add new storytelling features to current 360° videos.
As a result, the European media sector will have a better base to compete in the global arena.
An example of this is the possibility of providing clickable dynamic placeholder objects in conjunction with user profiling and preferences gathering. By leveraging these features broadcasters using the Hyper360 toolset will be able to offer more effective advertisements and this in turn will allow them to obtain higher revenues from advertising or higher level of engagement in entertainment or journalistic content offer.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the first reporting period, the consortium defined the requirements and the overall architecture and decided to research with iterative cycles, to refine features and architectural choices.
In the following periods, the consortium realized the planned Hyper360 toolset as a software suite supporting three main phases, namely capture (or recording), post-production (or annotation), and personalized delivery.
- Capture phase
The capturing tool OmniCap supports professional 360° video recording with camera setups ranging from simple dual lenses integrated devices to sophisticated multicamera 360° rigs. DRK achieved satisfactory features and performance and is connected to the JRS Quality Check component to analyze the recorded footage and identify defects like blurriness, flicker, noise and others.
CERTH CapTion’s 3D capturing tool collects and processes multi-view RGB-D data by leveraging a new, easy-to-use multi-sensor calibration process. A modular design is open to the integration of future sensors. The tool is fully functional. Besides, CERTH developed a ray-casting fusion engine that will drive and - in the future guide - the merging of the two heterogeneous media (360° video and 3D free-viewpoint video). For guidance, preliminary research produced state-of-the-art results in 360° scene understanding and a new dataset that we made public. The final engine will use AI to guide the fusion process and achieve higher photorealism when merging 360° and 3D video.
- Post-production phase
The post-production phase relies on OmniConnect (ENG).
OmniConnect can add interactive elements to 360° background videos. A final prototype is fully operational and supports the overlay of sophisticated interactive items (text elements, HTML, audio, and video). The performance is satisfactory. OmniConnect makes it possible to add different configurations (sets of overlay elements) to any 360° background video file, thereby increasing flexibility in managing different experiences for diverse viewers.
OmniConnect can also fuse the 3D storytelling characters (Mentors) produced by OmniCap as narrative elements.
- Presentation phase
The presentation phase relies on the following main components:
1) OmniPlay is the set of Hyper360 players for all mainstream platforms: standard PC browsers. Android, iOS, and virtual reality HMDs.
2) personalization services defined by the synergic action of the Profiling and Recommendation Engines. The Profiling Engine performs the understanding of implicit user preferences (interests vs. disinterests) based on user viewing behavior. The Recommendation Engine carries out inferences based on these user-specific data plus domain knowledge to provide some level of personalization for the content delivery.
3) OmniCloud is the delivery environment hosting all Hyper360 components.
4) Automatic camera path, which generates a standard 2D video (for consumption on TV sets) from a 360° video in an automatic way, by focusing on the salient areas in the 360° video.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The 360° video domain is a dynamic and rapidly evolving sector in which technical progress happens at a sustained rate.
In relation to the state-of-the-art, Hyper360 has delivered a state-of-the-art low-cost multi-view RGB-D capturing system that supports the latest RGB-D sensor technology advancements (Intel Realsense D415 and Microsoft Kinect for Azure) as well as a novel easy-to-use multi-view spatial calibration method. In addition, a new fast performance capture method based on multi-view RGB-D data is delivered. Moreover, state-of-the-art 360° depth, surface normal as well as low-frequency illumination estimation methods were achieved using a new dataset generated in the Hyper360 project.
Moreover, Hyper360 has extended and delivered a state-of-the-art, lightweight yet effectively expressive fuzzy reasoning service, used for real-time content-to-user preference matchmaking in the context of content recommendation, as well as a user-pertinent, cross-cutting ontology for the networked media domain.
Hyper360 results are aligned at the evolving state of the art, with some strong points:
- Information layered by OmniConnect on 360° background videos is a separate set of data, added on the fly only during playback. This choice allows a more effective and flexible post production process.
- Hyper360 adopts open standards, where existing.
- Automatic camera path generation from 360° video
- A set of AI modules (pose estimation, object detection and tracking, …) for semantic analysis of the 360° video
- Hyper360 supports the personalization of content based on past user viewer choices.

Hyper360 Logo