An Artificial Intelligence Enhancing Video Quality Locally to Limit Internet Traffic Tied to Video Streaming

Projektinformationen

ENHANCEplayer

ID Finanzhilfevereinbarung: 831109

DOI

10.3030/831109

Projekt abgeschlossen

EK-Unterschriftsdatum 7 Dezember 2018

Startdatum 1 November 2018

Enddatum 31 Januar 2021

Finanziert unter

• Horizon 2020 Framework Programme

Gesamtkosten

€ 3 343 750,00

EU-Beitrag

€ 2 400 000,00

Koordiniert durch

ARTOMATIX LIMITED
Ireland

Periodic Reporting for period 2 - ENHANCEplayer (An Artificial Intelligence Enhancing Video Quality Locally to Limit Internet Traffic Tied to Video Streaming)

Berichtszeitraum: 2019-11-01 bis 2021-01-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

What is the problem/issue being addressed?
As live streaming on phones becomes more and more popular across the world, broadcasters and live stream content creators have a bandwidth challenge. ENHANCEplayer address this by moving some of the cost away from bandwidth to local processing using upres models developed by Artomatix.
Why is it important for society?
More information can be shared more widely by more people in real time. This has very clear ramifications in mobile first developing economies, where bandwidth is often a strong barrier to entry for companies sharing content that the majority of users will watch on mobile devices.
What are the overall objectives?
Overall objectives is to create upres models that will enable broadcasters to push livestream content at a lower resolution and then have the ENHANCEplayer feature upres it in real time at a high quality.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

We were successfully able to achieve the following result:
We achieved an upscale of 240p to 480p at 25fps on a wide range of Android phones.
We achieved an upscale of 360p to 720p at 25fps on an iPhone 11, thanks to the performing neural processing unit in the iPhone 11.
We established an MVP, and a Marketing and Sales strategy, and developed all supporting material, and reached out to a first set of parties to pitch the ENHANCEplayer concept.
We prepared for a public demonstration at NAB, but unfortunately had to replace this by a virtual NAB with a pre-recorded demonstration. The same holds for IBC.
We prepared a final MVP definition, including roadmap going forward.
We collected feedback from real users.

Unfortunately, we were not able to overcome the technical difficulties on Android devices, and after trying a variety of potential solutions, we concluded that the technology is not yet ready for this solution.

We've put together a selection of recommendations for future research, as we see technology developing in the next two to three years that should enable this technology to be successful and have a strong market impact.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

Recommendations for future research:

These recommendations are not only for Android but the Enhance Player project.
Overall, we have encountered following challenges during the 2 years:
Upres Quality
Temporal Consistency
Limitation on model size
Model capacity
Compression Artifacts
Face definition
Text definition
Audience attention
Performance: real time processing for high definition targets (like 1080p)
25 FPS
HD targets
Our experiments successfully solved all challenges in Upres Quality category and achieved acceptable results for Performance on iPhone devices. For future research, our suggestion is to focus on performance to allow quicker market occupation.

Research direction 1: Integrate upres as part of video codec. Lots of codec like MPEG4 have key frames which can be utilised to reduce the needs of upres frame by frame. Before decoding the video, upres the key frame in the video stream only.

Research direction 2: Integrate upres as part of video displaying pipeline with customised acceleration chipset. Instead of upres the frames at computer side, this approach upres them at monitor or TV side with customised chipset just before displaying. Same hardware design can be placed in the graphic card as well.

Research direction 3: Customise the streaming pipeline to allow upresing on part of the video on older mobile devices and browsers. As the goal is to reduce bandwidth cost, given the large user base number of low-medium range mobile devices and browser players, enabling partial upresing will save a significant amount of bandwidth in overall.

Upres comparison between ENHANCE and signal processing