AI and Process Automation for Sustainable Entertainment and Media

Project Information

EMERALD

Grant agreement ID: 101119800

DOI

10.3030/101119800

Project closed

EC signature date 23 May 2023

Start date 1 October 2023

End date 31 March 2026

Funded under

• Digital, Industry and Space

Total cost

€ 2 689 186,25

EU contribution

€ 2 191 223,75

Coordinated by

UNIVERSIDAD POMPEU FABRA
Spain

Periodic Reporting for period 1 - EMERALD (AI and Process Automation for Sustainable Entertainment and Media)

Reporting period: 2023-10-01 to 2024-09-30

Summary of the context and overall objectives of the project

EMERALD is a 30-month IA to develop and demonstrate exemplary tools for the digital entertainment and media industries using AI Machine Learning and Big Data technologies, to automate and speed processing, increase production efficiency, use less energy and increase the quality of content. There is a massive increase in the volume of video-based and extended reality content, with an unsustainable demand for skilled human resources, data processing and energy. EMERALD aims to meet the challenge by a series of actions that will have a multiplier effect across major segments of the entertainment and media industry. In particular, EMERALD is developing process automation for sustainable media creation; creating a testbed for measuring the energy used in media computation; developing more efficient data use for AI/ML in entertainment and media applications; reducing the power demands for large-scale media data processing; and creating acceptance and demand for AI and sustainable production technologies in the entertainment and media industries. Tools will be based on more efficient data use. We will explore hardware processing that reduces the energy demand. We will develop a framework enabling more precise measuring and estimation of energy consumption. We will foster acceptance and demand for AI and sustainable entertainment and media production, with special attention to metaverse creators for scalability, and specifically explore how these techniques could make media consumption more sustainable.

EMERALD’s planned results leading to outcomes and impacts include:
● AI-supported processes and tools to automate video-based content analysis, processing, modification and reuse (WP2)
● Industry guidelines and White Papers on energy assessment and measurement in digital content processing pipelines (WP3)
● An open-source toolchain, API, framework and measuring tools for quantifying the energy consumption of media processing pipelines and computing operations, at a granular level, for commonly used processes such as
Foundry’s Nuke and specific tools produced by EMERALD (WP3)
● More efficient content processing for sustainable production by the reduction of datasets for training, smart streaming, content aware control of codecs in workflows and improved big data Cloud-based production pipelines (WP4)
● Novel methods for working with compressed video, exploiting AV1 compression in existing general-purpose hardware, to increase efficiency and reduce energy use (WP4)
● Software tools, optimised codecs and hardware implementations for reducing the energy consumed by video processing and content creation, by the application of AI algorithms, FPGAs and low-power computing devices (WP5)
● Industry-based evaluations and demonstrations of sustainable AI-based processes and big data media pipelines in professional industry installations and at major trade fairs and events (WP6)
● Promotion of energy reduction and computational efficiency increases to the media technology and content production sectors, standards bodies, policy makers and the public through workshops, dissemination and communication activities (WP6)

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Towards EMERALD's objectives, the project has different milestones and, in particular, as of month 12 (M12), the project has two milestones:
- Milestone 1. Procedures and initiation.
Related Work Packages: WP1 and WP6. Due to M6.
Means of verification: Deliverables D6.1 D1.1 D6.2 D1.2 D1.3
- Milestone 2. Algorithms, initial datasets and tools.
Related Work Packages: WP2, WP4 and WP5. Due to M12.
Means of verification: Deliverables D2.1 D2.2 D2.3 D2.4 D2.5 D4.1 D5.1 D6.3

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Until now we have obtained several results with impact, some of them have been published and/or presented in appropriate venues. We list here some of them:
1. François Pitié, “Introduction to Video Compression Research and Emerald”, in VideoLAN Developer Days 2023, Dublin, Ireland.
23/09/2023
Relevant Work Package: WP5
2. Vibhoothi, J. Zouien, F. Pitié, and A. Kokaram, “Unravelling the Power of Single-Pass Look-Ahead in Modern Codecs for Optimized Transcoding Deployment,”in NAB 2024
15/04/2024
Relevant Work Package: WP5T1
3. Hareesh Veekanchery , “Energy Efficiency in AI Tools for Post Production workflows” at Researcher’s Day in School of Engineering, Trinity College Dublin
25/04/2024
Relevant Work Package: WP3T2
4. A. Kokaram, Vibhoothi, J. Zouien, F. Pitié, C. Nash, J. Bentley, and P. Coulam-Jones, “Demystifying the use of Compression in Virtual Production” in SMPTE Media Summit 2024
21/10/2024
Relevant Work Package: WP5T1
5. Shashwat Khandelwal, Optimising AI with hardware acceleration; From edge devices to the cloud”, at 2nd Researcher Symposium in School of Engineering, Trinity College Dublin
22/10/2024
Relevant Work Package: WP5T2
6. A. Kokaram, “Introduction to Compression in Virtual Production” in Netflix and YouTube Office, USA
01/07/2024
Relevant Work Package: WP5T1
7. A. Kokaram, “Media Future Workshop” AOMedia at IBC held at Netflix offices Amsterdam,
15/09/2024
Relevant Work Package: WP5T1/2
8. J. Zouein, Vibhoothi, A. Kokaram, “Hardware Video Encoding Quality for Post-production Applications” in VideoLAN Developer Days 2024
21/10/2024
Relevant Work Package: WP5T1
9. Vibhoothi, J. Zouein, F. Pitié, and A. Kokaram, “Using Single-Pass Look-Ahead in Modern Codecs for Optimized Transcoding Deployment” in Motion Imaging Journal 2024
01/10/2024
Relevant Work Package: WP5T1
10. Shreejith Shanker, “Fast, Flexible and Energy-Efficient Deep Learning with Dataflow Accelerators”, at eFutures Edge AI Event at AMD, Dublin
12/11/2024
Relevant Work Package: WP5T2
11. A. Kokaram,”Future Media Production” Keynote, Creative Industries Forum, Enterprise Ireland
06/11/2024
Relevant Work Package: WP5
12. A. Cartas. “Two Weakly Supervised Approaches for Role Classification of Soccer Players”. talk at ACM MMSports '24 on November 1st, 2024, related to the publication by A. Cartas, C. Ballester, and G. Haro. 2024. Two Weakly Supervised Approaches for Role Classification of Soccer Players. Paper accepted for presentation at the 7th International ACM Workshop on Multimedia Content Analysis in Sports (MMSports '24), Melbourne, Australia.
01/11/2024
Relevant Work Package: WP2
13. Coloma Ballester. Multimodal Self-Supervised Learning and Applications to Visual Data Understanding. Invited Plenary Speaker at SIAM Conference on Imaging Science (IS24). Atlanta (United States). May 28 - 31, 2024.
30/05/2024
Relevant Work Package: WP2
14. Coloma Ballester. Artificial Intelligence for Visual Data Understanding and EMERALD. Invited talk at the 2024 Research Week. UPF. Barcelona.
16/11/2024
Relevant Work Package: WP2
15. FilmLight IBC Trade Show
Demonstrating FilmLight Remote at the AWS Booth and on private demos
12-15/09/2024
Relevant Work Package: WP4
16. FilmLight:Presentation MTH Conference Babelsberg
Presented a joint collaboration workflow with Telekom, Sony, Quibb and Pharos
25/09/2024
Relevant Work Package: WP4
17. Graeme Phillipson. Gave industrial talk Applications for Novel View Synthesis in Media. Conference on Visual Media Production (CVMP), London, UK
01/12/2023
Relevant Work Package: WP2T4
18. Jack Alston presented, as student, Computer Vision for Wildlife Archive Enrichment at BMVA Summer School, Durham University, UK
15/07/24
Relevant Work Package: WP5T3

emerald-project-evolution.jpg