Smart Asset re-Use in Creative Environments

Project Information

SAUCE

Grant agreement ID: 780470

DOI

10.3030/780470

Project closed

EC signature date 15 November 2017

Start date 1 January 2018

End date 31 December 2020

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 3 999 375,00

EU contribution

€ 3 999 375,00

Coordinated by

UNIVERSIDAD POMPEU FABRA
Spain

Periodic Reporting for period 2 - SAUCE (Smart Asset re-Use in Creative Environments)

Reporting period: 2019-07-01 to 2020-12-31

Summary of the context and overall objectives of the project

SAUCE is a three-year EU Research and Innovation project between 9 partners to create a step-change in allowing creative industry companies to re-use existing digital assets for future productions. Currently, within film, TV or games, searching for and enabling a previously created digital asset for a new production or context requires a high cost due to the lack of automated processes and the fact that assets can be tied to a particular tool-chain.

The goal of SAUCE is to produce, pilot and demonstrate a set of professional tools and techniques that reduce the costs for the production of enhanced digital content for the creative industries by increasing the potential for re-purposing and re-use of content as well as providing significantly improved technologies for digital content production and management. SAUCE will contribute to make content “smarter” by making it fully adaptive, in a broad, unprecedented manner: adaptive to context (which facilitates re-use), to purpose (among or within industries), to the user (improving the viewing experience), and to the production environment (so that it’s “future-proof”). This smart content will allow companies to easily re-use it, share it with other companies, or provide it for use in other sectors.

Specifically, SAUCE has the following definite and measurable objectives:
- Create tools to unlock value in previously-created content by allowing to edit or automatically adapt its properties (e.g. appearance, scale, motion) to the current production needs.
- Create tools to allow content created in the future to be more easily re-used and re-purposed, by developing light-field technology for the creative industries in terms of capture, storage, distribution and processing.
- Create management, animation and production tools to keep digital content accessible, discoverable and malleable by using procedural techniques and high level semantic knowledge.

Achieving these three aims will allow companies to work much more cost-effectively and efficiently. It will also open up the possibility of sharing assets with other sectors – especially those that are taking up interest in virtual production or AR/VR for various scenarios where content creation can be a barrier due to its prohibitive cost.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

SAUCE has achieved its first half of the project being able to successfully address all its objectives: Objective 1 has been achieved. Objectives 2-8 are all being addressed by the project, and satisfactory progress is being made against the project DoA.
Deliverable production has been satisfactory: almost all deliverables have been produced on time, are well written and professionally presented.

Scenario Requirements helped identify the key use cases and scenarios for a smart search and transformation framework. DNEG produced a report (D2.3) which documented their findings and proposed a new generalised asset descriptor that would provide the basis and foundation for subsequent work packages.

The primary goals of New Technologies for Asset Creation have been progressively accomplished during the first half tenure of the project. USAAR with inputs from both internal and external experts have done very successful lightfields productions, ‘Elements’ and ‘Unfolding’. The ‘Elements’ shoot was the very first shoot done within the SAUCE project with unstructured content (fire and smoke). ‘Unfolding’ assets include different versions of the voices of a cello composition written and performed by a professional cello player, specifically for this project.

Unlocking Existing Assets: Taking the findings from WP2, DNEG delivered a working, fully functional, proof of concept for the smart search framework, which included data architecture, technical architecture and a number of reference implementations.

In Transforming Assets, DNEG focused on two main areas. The first was developing and demonstrating an extendable and composable transformation framework, which like the search framework describe in WP4, provides a means for transformation services and capabilities to be advertised to the search framework and chained together. The second focused on developing tools to alter of animation timing and trajectory. The concepts used to develop these tools provided a framework which was extended to support asset synthesis and the synthesis of new material.

Semantic Animation Production: The common theme of the tasks within WP6 revolve around methodology to generate ‘smart assets’ and unlock existing assets for re-use furthering the state-of-the-art within the animation industry. To meet these goals, months 1 to 18 within the SAUCE project have seen the completion of one task (WP6T2) and the on-going development of four tasks (WP6T1/WP6T3/WP6T5/WP6T6) to satisfy the goals outlined in the SAUCE project.

Intelligent Storage of Assets primarily focuses on the creation of a type agnostic and geolocatable asset store. The primary work completed within this WP is the separation of the front and back end of Flix and to enable the backend server to be used as a stand alone application. This will form the base of the asset store. As part of this separation and refactoring we created a rich api to allow for extensions/plugins to allow it to house the partners work.

For Experimental Production, Evaluation and Innovation Assessment, Filmakademie planned, prepared and realized a lightfield production under realistic movie production conditions. The ‘Unfolding’ production was shot in a professional studio at Saarländischer Rundfunk in cooperation with the Director of Photography (DoP) Matthias Bolliger. In the preparation phase, a virtual previsualization was created. The developed 3D scene was used to plan shots and timings as well as the overall look. In addition, the material was used to test algorithms developed within SAUCE even before the actual footage was captured. After the shoot in January 2019, the postproduction was carried out at Filmakademie.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

By extending the state of the art in the media production field with respect to the storage, reuse, and generation of assets, the SAUCE project will contribute to industry wide efficiency gains, allowing for the market's needs to be met with less resources. This will not only allow production players to improve their bottom line, but could also contribute to a reduction in their environmental impact and increase their bandwidth to accomodate for the creation of more diverse and less mainstream content, ultimately contributing to the enrichment of the entertainment landscape. The project aims to accomplish this by creating smarter tools for asset storage and recall (semantic labelling, re-synthesis, and transformation of high-level asset components) as well as by facilitating future asset generation tools like light field capture and smart animation. These tools will not only promote the reuse of assets, allowing for creators to cut down on redundant asset synthesis (estimated to make up approximately 20% of new asset creation currently), but will also facilitate innovative workflows where re-usability is taken into account at the production phase. Considering the 2.1 trillion USD projected value of the worldwide entertainment sector at the close of the project in 2020, it is likely that the eventual economic impact of the project will be immense, even ignoring the potential impact on related fields of product design, architecture, or engineering.

Project Overview