Periodic Reporting for period 1 - AutoPost (Deformable Surface Tracking and Alpha Matting for the Automation of Post-production Workflows)
Reporting period: 2015-01-01 to 2016-06-30
Summary of the context and overall objectives of the project
The H2020 EU project “Deformable Surface Tracking and Alpha Matting for the Automation of Post-production Workflows” aims to improve the efficiency of labour-intensive VFX tasks, automating major parts of the daily workload in current post-production and visual effects workflows. The goal of the AutoPost project is to automate major parts of the daily workload in small and medium post houses and with it to make post-production more efficient by reducing time-consuming and costly manual processing. This objective will be achieved by bridging the existing gap between commercially-available post-production tools and state-of-the-art research in the domain of deformable surface tracking and natural video matting, alleviating the operator workload on material that lacks planning with respect to post-production requirements.
The AutoPost project aims to develop new methods applicable to general movie productions filmed without specific constraints, since they constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses. To ensure industry adoption, the algorithms will be integrated into “The AutoPost Plugin Suite”, an extensible, close-to-market multi-platform post-production tool targeting regular 2D and 3D productions, Visual effects and 2D-to-3D conversion.
The technical and operational objectives of the project are:
O1. To define the requirements of the tools as determined by the industry needs.
O2. To develop deformable surface tracking methods that estimate temporally consistent surface motion, deformation, and shading changes, even in presence of temporary occlusions under real-world conditions.
O3. To develop natural video matting methods that provide accurate and more realistic mattes for VFX and post-production processes with particular attention to motion blur and deformable surfaces under real-world conditions.
O4. To develop Software Development Kits (SDKs) for the tracking and matting algorithms. The SDKs will be the basis for the development of the Plugin Suite and will come along with functional prototypes that demonstrate their use.
O5. To develop The AutoPost Plugin Suite that will include tracking and matting plugins for industry-standard tools. The Plugin Suite will provide data interfaces that are widely used and accepted by VFX and 2D-to-3D conversion artists and user interaction tools.
O6. To evaluate and demonstrate the developed plugins under real working conditions by SME project partners from creative industries, developers of related post-production tools as well as post-houses and end-users.
O7. To define the strategies and policies for managing and exploiting the intellectual property derived from the activities of the project and to define and implement a business exploitation model and its corresponding business plan.
The AutoPost project aims to develop new methods applicable to general movie productions filmed without specific constraints, since they constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses. To ensure industry adoption, the algorithms will be integrated into “The AutoPost Plugin Suite”, an extensible, close-to-market multi-platform post-production tool targeting regular 2D and 3D productions, Visual effects and 2D-to-3D conversion.
The technical and operational objectives of the project are:
O1. To define the requirements of the tools as determined by the industry needs.
O2. To develop deformable surface tracking methods that estimate temporally consistent surface motion, deformation, and shading changes, even in presence of temporary occlusions under real-world conditions.
O3. To develop natural video matting methods that provide accurate and more realistic mattes for VFX and post-production processes with particular attention to motion blur and deformable surfaces under real-world conditions.
O4. To develop Software Development Kits (SDKs) for the tracking and matting algorithms. The SDKs will be the basis for the development of the Plugin Suite and will come along with functional prototypes that demonstrate their use.
O5. To develop The AutoPost Plugin Suite that will include tracking and matting plugins for industry-standard tools. The Plugin Suite will provide data interfaces that are widely used and accepted by VFX and 2D-to-3D conversion artists and user interaction tools.
O6. To evaluate and demonstrate the developed plugins under real working conditions by SME project partners from creative industries, developers of related post-production tools as well as post-houses and end-users.
O7. To define the strategies and policies for managing and exploiting the intellectual property derived from the activities of the project and to define and implement a business exploitation model and its corresponding business plan.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
The main scientific and technical results of the AUTOPOST project can be grouped in the following categories:
Functional and technical requirements
------------------------------------------------------
The main objective in this area was the definition of requirements related to the tracking and matting tools from a user’s perspective as well as the specification of interoperability interfaces with the aim to guarantee that the solutions produced could be integrated into existing postproduction platforms and are effective in real productions.
As a first stage, the most important use cases where efficient matting and tracking tools are required for visual effects were identified and reviewed. The use cases were derived from common visual effects tasks and actual projects that have been executed by the post-production partners. In order to come up with a representative list of use cases the world-wide connections with other post-production companies were also exploited. Identified use cases included the tracking of rigid objects (e.g. changing a logo on a coffee mug), or deforming surfaces (e.g. adding a tattoo on a body part or digital make-up), extracting people with motion blur and heads with hair, enhancing or replacing the background of a scene (e.g. adding snow to a landscape), or selecting objects for colour grading and finishing.
Based on these use cases the functional and technical requirements and the initial specification of the AutoPost surface tracking and alpha matting tools was derived. Taking advantage of the consortium members experience with existing tools and methods, as well as research on a wide scope of potential applications, typical user requirements were identified and transferred into required features and specifications of the interfaces and data formats. These were reviewed during the project based on systematic user evaluations of existing tools and workflows, discussions and collaborative work between the consortium members and advancements in the tracking and matting algorithms/plugins.
Extension and adaptation of state-of-the-art algorithms
-------------------------------------------------------------------------------
Existing state-of-the-art algorithms for deformable surface tracking and natural video matting were redesigned, extended and adapted to meet the user needs and requirements defined in the project in order to make them suitable for post-production environments and tasks. Data interfaces were designed to conform to established industry standards for easy adoption. Furthermore, programming interfaces were designed to allow for easy integration into SDKs and eventually plugins, including means for user interaction. Incremental versions of both tracking and matting algorithms were developed and implemented into the corresponding SDKs.
An improved algorithmic tracking framework, providing fully deformable surface tracking functionality as well as low level programming interfaces for configuration and data interaction, was developed to improve both the effectiveness and the efficiency of tracking tasks. The new algorithm operates on arbitrary continuous surfaces and is not limited to planarity or rigid motion. Also, shading and motion blur, two of the most common changes in scene content, are incorporated into the pure geometric displacement estimation. Last but not least, a variety of parameters and interactions are available allowing to achieve better results by bringing the user into the loop.
An improved algorithmic matting framework was implemented to improve and extend the state-of-the-art in natural matting for video sequences. This framework uses effectively motion information to automatically propagate predefined trimaps to assist the user through the whole sequence, to incorporate temporal coherence between alpha mattes and to deal with blurred mattes produced by objects in motion. Effective interaction tools allow for semiautomatic correction of possible errors in propagated trimaps or alpha mattes.
AutoPost Plugin Suite development
-------------------------------------------------------------------------------
Software development kits (SDKs) for deformable surface tracking and natural video matting were implemented constituting the high level programming interface for configuration and data exchange of the algorithmic frameworks developed and the basis for the development of the AutoPost Plugin Suite. SDKs include the abstraction of the low level interfaces to make them more intuitive and useful to the visual effects community, data interfaces that conform to established industry standards for easy adoption and user interfaces for interaction with the algorithms.
The tracking SDK provides an easy to use interface to facilitate surface tracking. Exchange formats are almost exclusively based on simple 2D textures, including region, shading and motion maps, which simplify the integration process into plugins for post-production tools. True user interaction is included, allowing the assessment of tracking results after each frame and the rerun of the algorithm with additional input to improve the results.
The matting SDK provides effective tools to perform and interact with a natural matting framework for video sequences. The SDK comprises a set of independent but compatible modules for alpha matte estimation, trimap propagation and refinement, foreground colour propagation, and motion regularization, in order to provide flexible usage, efficient user interaction and a wide configuration range for different use-cases and also different workflows. The data formats used in the matting interface including images, trimaps, mattes and optical flow maps.
Based on the SDKs, the AutoPost Plugin Suite, a set of matting and tracking plugins, was developed. It extends existing postproduction software such as Nuke with more sophisticated matting and tracking tools for improved quality and efficiency. Plugins provide the full functionality of the SDKs with graphical user interfaces and suitable interaction workflows and tools allowing a seamless integration into established tools and workflows. The AutoPost Plugin Suite was primarily developed for Nuke on Windows. The matting plugins were implemented based on the OFX API which is supported by a large variety of hosts and the tracking plugin was implemented based on the proprietary NDK supported only by Nuke. The plugin packages also include detailed documentation and usage examples.
Both the SDKs and the plugins have been developed by adopting an incremental and iterative development methodology with initial versions released early in the project and periodic updates until the end of the project.
Interoperability and evaluation
-------------------------------------------------------------------------------
An extensive collection of existing datasets coming from other national and international research activities and from previous productions of postproduction was compiled and reviewed to select a suitable test set which represents typical use cases for matting and tracking tools.
Due to the limitations of the existing datasets (uses cases and usage rights) a new dataset was acquired which combines visual effects shots that require both matting and tracking under varying conditions into a narrative story. This dataset was used to evaluate and demonstrate the AutoPost tools in a real-world production scenario. A subset of this data is publicly shared to facilitate its reuse and the comparison with other approaches.
All the plugins were tested successfully through interoperability tests to verify both the syntactic and semantic interoperability. Interoperability tests were carried at SDK level to prove the correct behaviour of the SDK, at plugin level to verify that the plugin uses the SDK correctly and at host level to ensure that the plugins work on platforms and hosts. Technical evaluation covering the quality, comparing obtained results with original or ground-truth data, and speed of tracking and matting plugins were also conducted to assess the performance of the plugins.
The AutoPost plugins were further assessed from an end-user perspective. For that the usability, flexibility, efficiency and acceptance have been evaluated in a realistic production scenario covering different use cases. Benchmark tests where the same footage is processed with different workflows and tools under the same conditions were conducted to validate the ambition of the AutoPost project and compare the performance of the AutoPost plugins against competing solutions. This evaluation has shown that the AutoPost plugins meet the expectations providing a better solution than existing commercial tools. The user evaluation was conducted in the form of an experimental production where the AutoPost Plugin Suite was used for creating the visual effects for a short film.
Dissemination and exploitation planning
---------------------------------------------------------------------------------
AutoPost has been very active in dissemination actions. AutoPost created particular means and tools for disseminating the project, such as its corporative image, the AutoPost website (www.autopost-project.eu) AutoPost Vimeo channel (www.vimeo.com/autopost) or the AutoPost Zenodo public repository (http://zenodo.org/collection/user-autopost(opens in new window)) as part of the participation of AutoPost in the H2020 Open Research Data Pilot. Several dissemination material were designed and published, including the project leaflet, posters, dissemination videos, and press releases. Scientific publications were also made and presented at conferences such as CVMP 2015.
Of special importance was the presentation of the project at public events in order to reach major audiences targeted and key for the success of the project: end-users, stakeholders and R+D community. AutoPost participated and presented the project by means of a project booth in the two major trade fairs in the post-production sector, the IBC 2015 (September, Amsterdam, the Netherlands) and the NAB 2016 (April, Las Vegas, USA). The AutoPost presence at these events has helped to spread the word about the AutoPost Plugin Suite, to capture the attention of potential end-users, to present the results in more detail, to build a better relationship with interested attendees and to get valuable feedback and insights for the path to a future commercialization of project results.
Regarding exploitation planning, partners have monitored the post-production and visual effects market, have studied current and emerging matting and tracking technologies, commercially available tools, and trends and challenges which are related to AutoPost. Based on the collected information, an exploitation plan has been defined to bring the AutoPost solution to the market.
Functional and technical requirements
------------------------------------------------------
The main objective in this area was the definition of requirements related to the tracking and matting tools from a user’s perspective as well as the specification of interoperability interfaces with the aim to guarantee that the solutions produced could be integrated into existing postproduction platforms and are effective in real productions.
As a first stage, the most important use cases where efficient matting and tracking tools are required for visual effects were identified and reviewed. The use cases were derived from common visual effects tasks and actual projects that have been executed by the post-production partners. In order to come up with a representative list of use cases the world-wide connections with other post-production companies were also exploited. Identified use cases included the tracking of rigid objects (e.g. changing a logo on a coffee mug), or deforming surfaces (e.g. adding a tattoo on a body part or digital make-up), extracting people with motion blur and heads with hair, enhancing or replacing the background of a scene (e.g. adding snow to a landscape), or selecting objects for colour grading and finishing.
Based on these use cases the functional and technical requirements and the initial specification of the AutoPost surface tracking and alpha matting tools was derived. Taking advantage of the consortium members experience with existing tools and methods, as well as research on a wide scope of potential applications, typical user requirements were identified and transferred into required features and specifications of the interfaces and data formats. These were reviewed during the project based on systematic user evaluations of existing tools and workflows, discussions and collaborative work between the consortium members and advancements in the tracking and matting algorithms/plugins.
Extension and adaptation of state-of-the-art algorithms
-------------------------------------------------------------------------------
Existing state-of-the-art algorithms for deformable surface tracking and natural video matting were redesigned, extended and adapted to meet the user needs and requirements defined in the project in order to make them suitable for post-production environments and tasks. Data interfaces were designed to conform to established industry standards for easy adoption. Furthermore, programming interfaces were designed to allow for easy integration into SDKs and eventually plugins, including means for user interaction. Incremental versions of both tracking and matting algorithms were developed and implemented into the corresponding SDKs.
An improved algorithmic tracking framework, providing fully deformable surface tracking functionality as well as low level programming interfaces for configuration and data interaction, was developed to improve both the effectiveness and the efficiency of tracking tasks. The new algorithm operates on arbitrary continuous surfaces and is not limited to planarity or rigid motion. Also, shading and motion blur, two of the most common changes in scene content, are incorporated into the pure geometric displacement estimation. Last but not least, a variety of parameters and interactions are available allowing to achieve better results by bringing the user into the loop.
An improved algorithmic matting framework was implemented to improve and extend the state-of-the-art in natural matting for video sequences. This framework uses effectively motion information to automatically propagate predefined trimaps to assist the user through the whole sequence, to incorporate temporal coherence between alpha mattes and to deal with blurred mattes produced by objects in motion. Effective interaction tools allow for semiautomatic correction of possible errors in propagated trimaps or alpha mattes.
AutoPost Plugin Suite development
-------------------------------------------------------------------------------
Software development kits (SDKs) for deformable surface tracking and natural video matting were implemented constituting the high level programming interface for configuration and data exchange of the algorithmic frameworks developed and the basis for the development of the AutoPost Plugin Suite. SDKs include the abstraction of the low level interfaces to make them more intuitive and useful to the visual effects community, data interfaces that conform to established industry standards for easy adoption and user interfaces for interaction with the algorithms.
The tracking SDK provides an easy to use interface to facilitate surface tracking. Exchange formats are almost exclusively based on simple 2D textures, including region, shading and motion maps, which simplify the integration process into plugins for post-production tools. True user interaction is included, allowing the assessment of tracking results after each frame and the rerun of the algorithm with additional input to improve the results.
The matting SDK provides effective tools to perform and interact with a natural matting framework for video sequences. The SDK comprises a set of independent but compatible modules for alpha matte estimation, trimap propagation and refinement, foreground colour propagation, and motion regularization, in order to provide flexible usage, efficient user interaction and a wide configuration range for different use-cases and also different workflows. The data formats used in the matting interface including images, trimaps, mattes and optical flow maps.
Based on the SDKs, the AutoPost Plugin Suite, a set of matting and tracking plugins, was developed. It extends existing postproduction software such as Nuke with more sophisticated matting and tracking tools for improved quality and efficiency. Plugins provide the full functionality of the SDKs with graphical user interfaces and suitable interaction workflows and tools allowing a seamless integration into established tools and workflows. The AutoPost Plugin Suite was primarily developed for Nuke on Windows. The matting plugins were implemented based on the OFX API which is supported by a large variety of hosts and the tracking plugin was implemented based on the proprietary NDK supported only by Nuke. The plugin packages also include detailed documentation and usage examples.
Both the SDKs and the plugins have been developed by adopting an incremental and iterative development methodology with initial versions released early in the project and periodic updates until the end of the project.
Interoperability and evaluation
-------------------------------------------------------------------------------
An extensive collection of existing datasets coming from other national and international research activities and from previous productions of postproduction was compiled and reviewed to select a suitable test set which represents typical use cases for matting and tracking tools.
Due to the limitations of the existing datasets (uses cases and usage rights) a new dataset was acquired which combines visual effects shots that require both matting and tracking under varying conditions into a narrative story. This dataset was used to evaluate and demonstrate the AutoPost tools in a real-world production scenario. A subset of this data is publicly shared to facilitate its reuse and the comparison with other approaches.
All the plugins were tested successfully through interoperability tests to verify both the syntactic and semantic interoperability. Interoperability tests were carried at SDK level to prove the correct behaviour of the SDK, at plugin level to verify that the plugin uses the SDK correctly and at host level to ensure that the plugins work on platforms and hosts. Technical evaluation covering the quality, comparing obtained results with original or ground-truth data, and speed of tracking and matting plugins were also conducted to assess the performance of the plugins.
The AutoPost plugins were further assessed from an end-user perspective. For that the usability, flexibility, efficiency and acceptance have been evaluated in a realistic production scenario covering different use cases. Benchmark tests where the same footage is processed with different workflows and tools under the same conditions were conducted to validate the ambition of the AutoPost project and compare the performance of the AutoPost plugins against competing solutions. This evaluation has shown that the AutoPost plugins meet the expectations providing a better solution than existing commercial tools. The user evaluation was conducted in the form of an experimental production where the AutoPost Plugin Suite was used for creating the visual effects for a short film.
Dissemination and exploitation planning
---------------------------------------------------------------------------------
AutoPost has been very active in dissemination actions. AutoPost created particular means and tools for disseminating the project, such as its corporative image, the AutoPost website (www.autopost-project.eu) AutoPost Vimeo channel (www.vimeo.com/autopost) or the AutoPost Zenodo public repository (http://zenodo.org/collection/user-autopost(opens in new window)) as part of the participation of AutoPost in the H2020 Open Research Data Pilot. Several dissemination material were designed and published, including the project leaflet, posters, dissemination videos, and press releases. Scientific publications were also made and presented at conferences such as CVMP 2015.
Of special importance was the presentation of the project at public events in order to reach major audiences targeted and key for the success of the project: end-users, stakeholders and R+D community. AutoPost participated and presented the project by means of a project booth in the two major trade fairs in the post-production sector, the IBC 2015 (September, Amsterdam, the Netherlands) and the NAB 2016 (April, Las Vegas, USA). The AutoPost presence at these events has helped to spread the word about the AutoPost Plugin Suite, to capture the attention of potential end-users, to present the results in more detail, to build a better relationship with interested attendees and to get valuable feedback and insights for the path to a future commercialization of project results.
Regarding exploitation planning, partners have monitored the post-production and visual effects market, have studied current and emerging matting and tracking technologies, commercially available tools, and trends and challenges which are related to AutoPost. Based on the collected information, an exploitation plan has been defined to bring the AutoPost solution to the market.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
The AutoPost Plugin Suite aims to boost the competitiveness of the European post-production industry in the global market, targeting specifically small and medium post-production companies. AutoPost has the ambition of improving the efficiency of labour-intensive visual effects task through the development of a multi-platform plugin suite by integrating advanced scientific state-of-the-art solutions for tracking deformable surfaces and natural video matting into common workflows and production pipelines of visual effect artists.
AutoPost has been highly successful in its efforts to meet the complex and ambitious scientific, technological and industrial challenges, and has produced the AutoPost Plugin Suite, a set of close-to-market matting and tracking plugins that go beyond the current state-of the art of the industry and that can be integrated into existing post-production platforms (such as the widely used Nuke), extending them with more sophisticated tools for improved quality and efficiency.
The AutoPost Plugin Suite brings together several innovative features that address specifically the needs of European post houses. These functionalities make the matting and tracking plugins suitable for postproduction workflows and a large variety of visual effects including object and skin manipulation, scene extension and replacement, appearance modification (digital makeup, ageing), compositing and object extraction from arbitrary and natural backgrounds and object selection for colour grading and finishing. In particular, the plugins are applicable to general movie productions filmed without specific constraints such as markers, availability of 3D-models, individual object layers, green screens, which currently constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses.
The AutoPost plugins automate and improve the efficiency of visual effects and 2D-to-3D conversion workflows, providing time savings in labour-intensive manual process. Evaluation results have shown that in average an 80% of time savings can be achieved with AutoPost tools, leading to a 20% reduction of the postproduction costs.
AutoPost will not only help post-houses to reduce post-production costs but will also help them to produce higher-quality results thanks to the integration of new advanced technologies. This fact has been validated in tests with real-users, achieving operator approval in more than 80% of the tests.
Furthermore, providing the developed tools as plugins facilitates standardization, minimizes the risk of excessive solution diversification and makes possible a fast and natural arrival to the end users.
The impact from the use of AutoPost tools is expected to go beyond the local markets of European post-houses and reach the global market where infrastructure and labour costs can dictate where films are finally produced. AutoPost technology will help to reduce the weight of the labour cost factor, thus emphasizing the other decision-making factors in which the European postproduction industry is strong.
Indeed, it is expected that the impact on the small and medium post-production companies in Europe also creates a positive effect for the European post-production industry at a general level, given the fact, that the European postproduction landscape is primarily composed of small and medium facilities.
Apart from its potential impact in the competitiveness of the post-production industry, AutoPost also has a positive impact on the scientific community as well as the industry and the market. As AutoPost is an innovation project, the positive aspects amongst the scientific community do not reside particularly in the advancement of the state of the art but in the opportunity to convert research results into solutions for the industry and the strong collaborative relationship that is established between ICT R&D organisations and the media industry. The R&D tasks carried out in this project have also improved the internal skills of the academic partners to produce research results closer to user needs. Finally, the close collaboration with R&D partners and the insight in their cutting-edge technology have encouraged the end-user partners to think about new innovative products and how they can be used for their own competitive advantage.
AutoPost has been highly successful in its efforts to meet the complex and ambitious scientific, technological and industrial challenges, and has produced the AutoPost Plugin Suite, a set of close-to-market matting and tracking plugins that go beyond the current state-of the art of the industry and that can be integrated into existing post-production platforms (such as the widely used Nuke), extending them with more sophisticated tools for improved quality and efficiency.
The AutoPost Plugin Suite brings together several innovative features that address specifically the needs of European post houses. These functionalities make the matting and tracking plugins suitable for postproduction workflows and a large variety of visual effects including object and skin manipulation, scene extension and replacement, appearance modification (digital makeup, ageing), compositing and object extraction from arbitrary and natural backgrounds and object selection for colour grading and finishing. In particular, the plugins are applicable to general movie productions filmed without specific constraints such as markers, availability of 3D-models, individual object layers, green screens, which currently constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses.
The AutoPost plugins automate and improve the efficiency of visual effects and 2D-to-3D conversion workflows, providing time savings in labour-intensive manual process. Evaluation results have shown that in average an 80% of time savings can be achieved with AutoPost tools, leading to a 20% reduction of the postproduction costs.
AutoPost will not only help post-houses to reduce post-production costs but will also help them to produce higher-quality results thanks to the integration of new advanced technologies. This fact has been validated in tests with real-users, achieving operator approval in more than 80% of the tests.
Furthermore, providing the developed tools as plugins facilitates standardization, minimizes the risk of excessive solution diversification and makes possible a fast and natural arrival to the end users.
The impact from the use of AutoPost tools is expected to go beyond the local markets of European post-houses and reach the global market where infrastructure and labour costs can dictate where films are finally produced. AutoPost technology will help to reduce the weight of the labour cost factor, thus emphasizing the other decision-making factors in which the European postproduction industry is strong.
Indeed, it is expected that the impact on the small and medium post-production companies in Europe also creates a positive effect for the European post-production industry at a general level, given the fact, that the European postproduction landscape is primarily composed of small and medium facilities.
Apart from its potential impact in the competitiveness of the post-production industry, AutoPost also has a positive impact on the scientific community as well as the industry and the market. As AutoPost is an innovation project, the positive aspects amongst the scientific community do not reside particularly in the advancement of the state of the art but in the opportunity to convert research results into solutions for the industry and the strong collaborative relationship that is established between ICT R&D organisations and the media industry. The R&D tasks carried out in this project have also improved the internal skills of the academic partners to produce research results closer to user needs. Finally, the close collaboration with R&D partners and the insight in their cutting-edge technology have encouraged the end-user partners to think about new innovative products and how they can be used for their own competitive advantage.