Periodic Reporting for period 2 - AdMiRe (Advanced Mixed Realities)
Período documentado: 2021-09-01 hasta 2022-08-31
At present, TV audiences can interact with the shows they are watching only through social media or, to a lesser extent, hybrid broadcast broadband TV. Comments, video and audio from the audience may be incorporated into TV program formats, but this is done in such a way that viewer engagement is quite limited and there is not much potential for innovation.
AdMiRe will radically change this by producing solutions that will allow viewers at home to be incorporated into the live TV program they are watching and to interact with people in the TV studio. The key innovation here is to use a regular mobile phone to capture the viewer, extract him/her from the background and finally composite the viewer into the studio footage as an MR object.
The main goal of AdMiRe is to develop, validate and demonstrate innovation solutions, based on Mixed Reality (MR) technology, which will allow for audiovisual media audiences a step change in interactivity, and that will bring for content creators a radical improvement in talent immersion and interaction with computer generated elements.
AdMiRe addresses all open challenges by having the following general objectives:
• Producing and demonstrating the potential products, tools and services that will allow TV viewers at home to send their video image to be inserted live into a TV program, appearing as if they were physically present in the studio, and using for this purpose regular mobile phone cameras.
• Producing and demonstrating the potential products, tools and services that will allow actors and other talent to seamlessly interact with computer generated imagery, demonstrating it for the use case of very large scale live performances.
• Piloting and large-scale validation of potential products allowing to composite image elements that come from different sources, ensuring that the appearance of all elements is consistent in terms of colour, contrast, scene geometry and lighting.
• Validating all solutions with rigorous Quality of Experience (QoE) tests.
• Designing and implementing dissemination and exploitation strategies.
AdMiRe will radically change this by producing solutions that will allow viewers at home to be incorporated into the live TV program they are watching and to interact with people in the TV studio. The key innovation here is to use a regular mobile phone to capture the viewer, extract him/her from the background and finally composite the viewer into the studio footage as an MR object.
The main goal of AdMiRe is to develop, validate and demonstrate innovation solutions, based on Mixed Reality (MR) technology, which will allow for audiovisual media audiences a step change in interactivity, and that will bring for content creators a radical improvement in talent immersion and interaction with computer generated elements.
AdMiRe addresses all open challenges by having the following general objectives:
• Producing and demonstrating the potential products, tools and services that will allow TV viewers at home to send their video image to be inserted live into a TV program, appearing as if they were physically present in the studio, and using for this purpose regular mobile phone cameras.
• Producing and demonstrating the potential products, tools and services that will allow actors and other talent to seamlessly interact with computer generated imagery, demonstrating it for the use case of very large scale live performances.
• Piloting and large-scale validation of potential products allowing to composite image elements that come from different sources, ensuring that the appearance of all elements is consistent in terms of colour, contrast, scene geometry and lighting.
• Validating all solutions with rigorous Quality of Experience (QoE) tests.
• Designing and implementing dissemination and exploitation strategies.
During the first year of the project, the consortium has been focused on defining and gathering the end-users requirements for AdMiRe. Several consultations has been designed and held by NTNU in order to know first-hand which expectations of the potential customers are (based on the final users). Simultaneously, the technological partners has been worked on the AdMiRe specifications in order to develop and efficient and achievable system able to give response to the end users’ demands. End users (Premier Sports, NRK and TVR) have been very active in the definition of the use cases and the requirements for AdMiRe based on those cases.
Brainstorm, EPFL, UPF and disguise have started with the development of the AdMiRe technical modules
Regarding the capture and streaming module, the end to end streaming architecture design has been completed and a large part of the developments included in this module have been developed and implemented.
As for the video workflow & video processing applications, the three applications that comprise this module are in a very advanced state of development.
Regarding improvements to interaction between performers and virtual objects, disguise has started to improve and adapt its technology based on LED walls so that it can adopt AdMiRe outputs, increasing the versatility of live broadcast and productions, and the fidelity of the interaction and " presence". All related developments are very advanced.
NTNU has also led during the first half of the project the definition of the procedure to validate the project as well as the assessment methodology to be applied when the system is integrated and ready to implement at user’s sites.
During the second and final year of the project, the consortium has focused on the completion of the modules that make up AdMire. All modules have been integrated into the AdMiRe pipeline and have been validated internally in a first stage, and externally with end users in a second stage.
NTNU has carried out together with end users NRK, TVR and Premier Media a validation of the AdMiRe system and a Quality of Experience assessment with the technical support of all technical partners, especially Brainstorm.
Brainstorm and disguise have integrated the AdMiRe system into the XR stage based on disguise LED walls with very promising results.
Finally, with the continuous feedback obtained during the validations carried out with the users, the AdMiRe modules have been refined and improved obtaining a close to market solution.
The project has been extensively disseminated in many channels fulfilling the proposed KPIs. A consolidated business plan for the project results has been prepared being the route for the exploitation of the projects results.
Brainstorm, EPFL, UPF and disguise have started with the development of the AdMiRe technical modules
Regarding the capture and streaming module, the end to end streaming architecture design has been completed and a large part of the developments included in this module have been developed and implemented.
As for the video workflow & video processing applications, the three applications that comprise this module are in a very advanced state of development.
Regarding improvements to interaction between performers and virtual objects, disguise has started to improve and adapt its technology based on LED walls so that it can adopt AdMiRe outputs, increasing the versatility of live broadcast and productions, and the fidelity of the interaction and " presence". All related developments are very advanced.
NTNU has also led during the first half of the project the definition of the procedure to validate the project as well as the assessment methodology to be applied when the system is integrated and ready to implement at user’s sites.
During the second and final year of the project, the consortium has focused on the completion of the modules that make up AdMire. All modules have been integrated into the AdMiRe pipeline and have been validated internally in a first stage, and externally with end users in a second stage.
NTNU has carried out together with end users NRK, TVR and Premier Media a validation of the AdMiRe system and a Quality of Experience assessment with the technical support of all technical partners, especially Brainstorm.
Brainstorm and disguise have integrated the AdMiRe system into the XR stage based on disguise LED walls with very promising results.
Finally, with the continuous feedback obtained during the validations carried out with the users, the AdMiRe modules have been refined and improved obtaining a close to market solution.
The project has been extensively disseminated in many channels fulfilling the proposed KPIs. A consolidated business plan for the project results has been prepared being the route for the exploitation of the projects results.
AdMiRe advances beyond the state-of-the-art on several fronts:
It will pilot and demonstrate the products and tools that will allow TV viewers at home to send their video image to be inserted live into a TV program, appearing as if they were physically present in the studio, using a regular mobile phone camera for this purpose. This is a completely novel application that broadcasters will be able to provide, with a huge potential for increased user engagement and viewer ratings.
It will pilot and demonstrate the products and tools for virtual production that will allow actors and other talent to seamlessly interact with computer generated imagery, demonstrating it for the use case of very large scale live performances. These tools have the potential to be a game-changer in the industry.
It will pilot and validate products that will ensure that the appearance of composite elements coming from different sources is consistent in terms of colour, contrast, scene geometry and lighting. These tools have the potential to improve workflows and speed-up the transition from standard production to high dynamic range and wide colour gamut production, that offer greater quality of experience and sense of immersion, with the associated huge potential for the creative sector.
Different technologies and tools will be exploitable outcomes of AdMire:
Capture module. Responsible to obtain the texture and depth information of the captured spectator in such a way that it can be streamed properly, and including a capture wizard that, based on the captured data, and in real time, will provide advice on how to optimize it in terms of composition and image quality.
Streaming module. Based on WebRTC / NDI and centred on the communication with the smart phone.
Virtual Control Room. It will permit curating the group of users willing to intervene in the show and will manage all the provided video streams.
Video processing module. This module will be focused on real time high quality background extraction and also on obtaining super-resolution based on the spectator video stream and deep learning applied on it.
Video effects module. This module, more centred on the final image result, will be the responsible to adjust the spectator’s insertion in the scene in a compelling way.
It will pilot and demonstrate the products and tools that will allow TV viewers at home to send their video image to be inserted live into a TV program, appearing as if they were physically present in the studio, using a regular mobile phone camera for this purpose. This is a completely novel application that broadcasters will be able to provide, with a huge potential for increased user engagement and viewer ratings.
It will pilot and demonstrate the products and tools for virtual production that will allow actors and other talent to seamlessly interact with computer generated imagery, demonstrating it for the use case of very large scale live performances. These tools have the potential to be a game-changer in the industry.
It will pilot and validate products that will ensure that the appearance of composite elements coming from different sources is consistent in terms of colour, contrast, scene geometry and lighting. These tools have the potential to improve workflows and speed-up the transition from standard production to high dynamic range and wide colour gamut production, that offer greater quality of experience and sense of immersion, with the associated huge potential for the creative sector.
Different technologies and tools will be exploitable outcomes of AdMire:
Capture module. Responsible to obtain the texture and depth information of the captured spectator in such a way that it can be streamed properly, and including a capture wizard that, based on the captured data, and in real time, will provide advice on how to optimize it in terms of composition and image quality.
Streaming module. Based on WebRTC / NDI and centred on the communication with the smart phone.
Virtual Control Room. It will permit curating the group of users willing to intervene in the show and will manage all the provided video streams.
Video processing module. This module will be focused on real time high quality background extraction and also on obtaining super-resolution based on the spectator video stream and deep learning applied on it.
Video effects module. This module, more centred on the final image result, will be the responsible to adjust the spectator’s insertion in the scene in a compelling way.