Skip to main content

An end-to-end system for the production and delivery of photorealistic social immersive virtual reality experiences

Article Category

Article available in the folowing languages:

Photorealistic avatars for more immersive VR experiences

Anyone familiar with ‘The Sims’ video games would have had a feeling of déjà vu when they tried social Virtual Reality for the first time. But whilst the cartoonish characters came across as a breath of fresh air way back in 2000, they can’t help but feel a bit awkward when ported to wannabe immersive Virtual Reality experiences. Is it time for a photorealistic approach to social Virtual Reality?

Digital Economy

Immersive is easily the first term that comes to mind when you think about Virtual Reality (VR). But there is a second one not too far behind: social. As the recent boom of social VR apps shows, immersion goes hand-in-hand with the capacity of users to interact just like they would in the real world. The EU-funded project VRTogether (An end-to-end system for the production and delivery of photorealistic social immersive virtual reality experiences) stands at this much-coveted crossroads between immersive and social VR. The project promises groundbreaking VR experiences built around ‘social photorealistic immersive content’, at an affordable cost. To cut a long story short, VRTogether wants VR users to just forget about the avatars they’ve had to deal with so far, and rather enter VR experiences looking exactly as they do in the real world. “We have focused our work on the creation of the tools necessary to take multiple users, capture them in real time as volumetric videos, and bring them in virtual environments. We now have an end-to-end lightweight holo-conferencing platform that pushes technological boundaries to seamlessly integrate people – or rather a digital version of themselves – in VR or Extended Reality (XR) environments,” Sergi Fernandez, Media and Internet director at i2CAT and coordinator of VRTogether, summarises. On the path to their photorealistic avatars, the project team left no stones unturned. “A platform, per se, is composed of multiple components that can both be used individually and add value to specific parts of the chain. We have generated tools for capture, streaming and communication between multiple users using volumetric video. These components may also serve for other purposes like pure volumetric video streaming for the next-generation contents or realistic capture of characters, to be later post-produced as VFX,” says Fernandez. Beyond the 3D capture and reconstruction of users as a point cloud, the consortium also developed: solutions to ensure low latency; a new compression format to save bandwidth; media orchestration to create consistent and synchronised experiences for all users; the capacity to broadcast content in real time; and protocols to make object interactions more worldly. Icing on the cake: the support of home usage with a low entry burden for users. The technology can be compared to that used in Hollywood CGI blockbusters, only at a fraction of the cost.

From gaming to dating applications

Potential applications are legion. In Fernandez’ own words, “VRTogether is the seed for the next generation of communication and collaboration tools where the boundaries between real and digital become blurred. Our technology can be applied to content watching, gaming, training, education, culture, industry, or anything in which human collaboration in hybrid or virtual environments requires a higher quality of experience than what current framed video calls offer.” To ensure these experiences will eventually come to life, the project team has developed three content pilots to demonstrate the technology and attract potential customers. In these pilots, the users can participate in a murder investigation. “It’s a threefold story with one chapter per pilot,” Fernandez explains. “Users have to discover the perpetrator of a crime and collaborate in groups of two to six to interrogate suspects at a police station, participate in a TV show, and inspect a crime scene.” Collaboration with partners interested in applying VRTogether technology to specific use cases has already begun, and the team expects first product commercialisation as early as 2022. Meanwhile, project partners have started working on project follow-ups. i2CAT for instance is developing two virtual escape rooms, as well as a watch-together-in-VR experience for a broadcaster. All in all, VRTogether brings us closer to live-like experiences in VR, where realism is pushed to the extreme.

Keywords

VRTogether, avatar, photorealistic, virtual reality, VR, social VR

Discover other articles in the same domain of application