During the first reporting period, the consortium defined the requirements and the overall architecture and decided to research with iterative cycles, to refine features and architectural choices.
In the following periods, the consortium realized the planned Hyper360 toolset as a software suite supporting three main phases, namely capture (or recording), post-production (or annotation), and personalized delivery.
- Capture phase
The capturing tool OmniCap supports professional 360° video recording with camera setups ranging from simple dual lenses integrated devices to sophisticated multicamera 360° rigs. DRK achieved satisfactory features and performance and is connected to the JRS Quality Check component to analyze the recorded footage and identify defects like blurriness, flicker, noise and others.
CERTH CapTion’s 3D capturing tool collects and processes multi-view RGB-D data by leveraging a new, easy-to-use multi-sensor calibration process. A modular design is open to the integration of future sensors. The tool is fully functional. Besides, CERTH developed a ray-casting fusion engine that will drive and - in the future guide - the merging of the two heterogeneous media (360° video and 3D free-viewpoint video). For guidance, preliminary research produced state-of-the-art results in 360° scene understanding and a new dataset that we made public. The final engine will use AI to guide the fusion process and achieve higher photorealism when merging 360° and 3D video.
- Post-production phase
The post-production phase relies on OmniConnect (ENG).
OmniConnect can add interactive elements to 360° background videos. A final prototype is fully operational and supports the overlay of sophisticated interactive items (text elements, HTML, audio, and video). The performance is satisfactory. OmniConnect makes it possible to add different configurations (sets of overlay elements) to any 360° background video file, thereby increasing flexibility in managing different experiences for diverse viewers.
OmniConnect can also fuse the 3D storytelling characters (Mentors) produced by OmniCap as narrative elements.
- Presentation phase
The presentation phase relies on the following main components:
1) OmniPlay is the set of Hyper360 players for all mainstream platforms: standard PC browsers. Android, iOS, and virtual reality HMDs.
2) personalization services defined by the synergic action of the Profiling and Recommendation Engines. The Profiling Engine performs the understanding of implicit user preferences (interests vs. disinterests) based on user viewing behavior. The Recommendation Engine carries out inferences based on these user-specific data plus domain knowledge to provide some level of personalization for the content delivery.
3) OmniCloud is the delivery environment hosting all Hyper360 components.
4) Automatic camera path, which generates a standard 2D video (for consumption on TV sets) from a 360° video in an automatic way, by focusing on the salient areas in the 360° video.
