AI-based video compression for emerging technologies

Project Information

AISTREAM

Grant agreement ID: 965502

DOI

10.3030/965502

Project closed

EC signature date 24 November 2020

Start date 1 January 2021

End date 31 December 2022

Funded under

• Horizon 2020 Framework Programme

Total cost

€ 4 139 897,50

EU contribution

€ 2 999 928,00

Coordinated by

TECHNISCHE UNIVERSITAET WIEN
Austria

Periodic Reporting for period 4 - AISTREAM (AI-based video compression for emerging technologies)

Reporting period: 2022-07-01 to 2022-12-31

Summary of the context and overall objectives of the project

**Problem + Importance for Society**
The annual global consumption of internet data has increased exponentially more than tenfold from 120 Exabytes to 1,870 Exabytes from 2008 to 2018 and is set to increase to 3,276 Exabytes by 2021. The average adult in the US now spends 6.3 hours per day with digital media. Image and video data comprises more than 85% of all global internet traffic. This exponential growth trend shows no sign of slowing, and the volume of data will rise further with upcoming services like 4K, 5K, 8K, Virtual and Augmented Reality streaming and cloud gaming. Additionally, the number of mobile internet users is expected to grow from 3.6 billion in 2018 to 5.3 billion by 2025. Every two years, the global consumption of data doubles; this is not a sustainable situation. Even South Korea, with the fastest average domestic internet connection (28.6 Megabit/sec.), cannot reliably stream 4K or use cloud gaming. Jeff Hecht states "Researchers are scrambling to repair and expand data pipes worldwide to keep the information revolution from grinding to a halt".

Deep Render has been developing a highly innovative approach to video compression not based on previous compression technologies or codecs. Built from scratch, we have re-invented the entire domain of compression around modern frameworks, creating a radically new class of compression methods. We combine artificial intelligence, machine learning, statistics, and information theory, in a non-linear approach to video compression that mirrors the neurological processing of the best video compressor known, the human eye — aka, Biological Compression.

The importance for society is straightforward given the problems mentioned above. Without radically better compression technology, the entire digital revolution is grinding to a halt, impacting the economy, the ease of communicating and sharing data, the governments (non-popular policies + ISP subsidies) and future digital innovation speed. There is also the importance of saving money and getting fair bandwidth access. Currently, bandwidth access is not fairly distributed in the world and not fairly distributed within countries. Typically, more remote areas (the countryside) are bandwidth-starved, contributing to an unequal society split between urban and rural areas. Governments are aware of this problem and try to fix it with ISP subsidies, with not much success.

**Objectives**
In this project, we will take our proven video compression algorithm and develop a full demonstration video codec market-ready for commercialisation for video-on-demand (VoD), with a compression efficiency 80% better than the best existing codecs. We will test and pilot this codec using a real-life VoD streaming service provided by Contentflow, thereby identifying, and solving all integration issues, ensuring we can begin commercialisation post project. TU Wien will use its specialist technical skills in computer vision to extend our codec for four use cases, each with specific challenges — medical imaging, satellite imaging, stereo virtual reality, and autonomous cars. Codecs have numerous applications but are rarely translated to niche use cases due to the time and cost of development. Our codec can rapidly re-deploy, and we wish to confirm this by extending it to these applications. The knowledge gained will be vital for future application to other markets and cross-sector.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

**Deep Render:**

• Technology: We further refined our AI-based Compression technology; right now, we are 66% better than the state-of-the-art of traditional compression (goal at project end: 80%).

• Runtime: We worked on porting our compression pipeline to mobile NPUs (finished for the image pipeline, work remaining on video).

• Data Center: We expanded our IT data centre to have sufficient resources for large-scale neural network training.

• IP Protection: We filed numerous provisional patent and PCT applications, comprising over 35 concepts/innovations.

• Publications: We published three papers (2x CVPR workshop, 1x Preprint).

• Website/Branding: We finished re-branding and launching a new website; including an invite-only web demo and a public blog sharing our research (6 posts online).

• Consortium Collaboration: We worked together with TU Wien to onboard their researchers in the domain of AI-based Compression and aided them in their extension studies. We worked together with Contentflow to get help in container design and early-stage exploration of integrating the codec in their services.

• Commercial: We reached out to Microsft and Meta and entered into a technical evaluation phase with them.


**Contentflow**

• Preparation of our platform for the new codec; allowing more formats via ingest and outgest.

• We can now outgest also the format Icecast and connect directly to any given social network (implemented their APIs).

• Elemental changes to the mechanism of how our platform operates to allow implementation of the base-code.


**TU Wien**
• Literature research on state-of-the-art methods as well as public datasets in learned image compression

• Development of two stereo compression models that are currently the state of the art in learned stereo compression, satellite compression outperforming traditional codecs, and a volumetric compression model for medical MRI and CT data outperforming modern traditional codecs.

• Publication of our work on stereo image compression at the IEEE Conference for Computer Vision and Pattern Recognition (CVPR) and the Computer Vision Winter Workshop (CVWW)

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

**Expected Results Beyond the State Of the Art**

• An AI-based compression codec that is 80% more efficient than the state-of-the-art.

• The world's first exotic-data specialised, AI-based Compression codec. Meaning a compression codec that is finetuned for specific domain data (e.g. stereo imaging).


**Impact**
Our 80% better compression multiplies bandwidth supply by 5x supporting the EU Digital Single Market by:
• Ensuring that technology works for all people (e.g. bandwidth speed in rural areas).
• Ensuring a fair and competitive digital economy through freeing up bandwidth globally.
• Ensuring net neutrality can be preserved. No bandwidth "traffic jams" means no need to prioritise access.
• Positioning the UK and the EU as a global digital player for video compression products.

Ease of information sharing increases the quality of life, health, wellbeing, and inclusion for citizens; protects freedoms, and supports crisis communications; including:
• Reducing personal or geographical isolation, improving relationships via high-quality video chat.
• New recreational activities from new technologies. E.g. cloud-gaming, VR, AR, and 360°-video.
• Supporting the creative hobby sector, as amateur video streaming and encoding becomes easier.
• A fairer distribution of internet availability for bandwidth-starved regions (e.g. the countryside), following Europe's plan of "no person and no place is left behind"

The project helps towards Climate Change by:
• Helping to contribute towards the EU's climate change mitigation through reduced CO2 generation.
• Transitioning the VoD sector to a clean economy by significantly reducing its energy and carbon footprint.
• Increasing bandwidth availability extends existing networks' lifespans, increases robustness, and avoids the need for disruption and CO2 emissions in producing/ installing new capacity.

aistream-project-img.jpg