Innovative Volumetric Capture and Editing Tools for Ubiquitous Storytelling

Informations projet

INVICTUS

N° de convention de subvention: 952147

DOI

10.3030/952147

Projet clôturé

Date de signature de la CE 10 Juin 2020

Date de début 1 Octobre 2020

Date de fin 31 Decembre 2022

Financé au titre de

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Coût total

€ 2 357 492,69

Contribution de l’UE

€ 1 997 107,50

Coordonné par

UNIVERSITE DE RENNES
France

CORDIS fournit des liens vers les livrables publics et les publications des projets HORIZON.

Les liens vers les livrables et les publications des projets du 7e PC, ainsi que les liens vers certains types de résultats spécifiques tels que les jeux de données et les logiciels, sont récupérés dynamiquement sur OpenAIRE .

Livrables

Documents, reports (9)

D4.4 Creative XR Productions and Evaluation Report 2

We will perform final experimental XR productions to evaluate the final achievements of the INVICTUS project

D1.1 Report on architecture for volumetric motion capture, tracking, and database generation

Workpackage 1 addresses the development of algorithms necessary to create interactive representations of volumetric content This deliverable specifies the architecture of software components for the processing pipeline This includes algorithms for geometry reconstruction temporal registration automatic rigging animation and hybrid content creation

D3.1 Collaborative VR scene editing tool

Report on the implemented features of the Collaborative VR scene editing tool typically Scene Layout Lighting and cinematography

D5.1 First Plan for the Exploitation and Dissemination of the Results

A detailed report presenting a first plan for the dissemination and exploitation of the results within a the different scientific communities and b the targeted industries specified within the INVICTUS project ie VFX CG and animation and interactive industries

D5.3 Final Plan for the Exploitation and Dissemination of the Results

A final report presenting a detailed plan for the dissemination of the results within a the different scientific communities and b the targeted industries specified within the project ie VFX CG and animation and interactive industries In addition to that the report will contain a strategy for commercializing the knowledge that was generated within the project in alignment with the exploitation plan driven by the Innovation Board and will include a the management of IPR issues b ownership of the results and c licencing of the scientific outcomes

D5.2 Communication material to promote the project's results

A report presenting the communication material prepared for promoting the projects results including a projects website design and content description b official project templates c enewsletters d social media channels e press releases e project film content 35 minutes long subcontracted

D4.3 Creative XR Productions and Evaluation Report 1

We will carry a first experimental XR production to evaluate the first iteration of the achievements of the INVICTUS project

D1.2 First software package for animatable volumetric video generation

This deliverables provides first software components for the creation of interactive volumetric content together with a description of implemented algorithms THis includes modules for reconstruction registration and animation With these components first demonstration of volumetric video that can be animated and modified will be showcased

D4.2 Evaluation Plan

We will have two XR productions to evaluate the achievements of this project This document is the Evaluation Plan of those XR productions We will make sure to design the best scenario to cover all the achievement off the different work packages

Demonstrators, pilots, prototypes (3)

D2.3 First Motion adaptation and style transfer

This first demonstrator will display the implementation of morphology-and-style character adaptation applied to volumetric motion-captured data. Typically we will show how characteristics of the motions need to be adapted to match the morphology and style of the given character (e.g. heaviness, strength, “cartoon animation style” motions, etc.).

D3.3 Volumetric spatio-temporal avatar manipulators

Demonstration of dedicated features for re-animating volumetric motion captured data by extending spatio-temporal animators to specifics of volumetric motion.

D2.6 Final Motion adaptation and style transfer

This final demonstrator will display the implementation of morphology-and-style character adaptation applied to volumetric motion-captured data. Typically we will show how characteristics of the motions need to be adapted to match the morphology and style of the given character (e.g. heaviness, strength, “cartoon animation style” motions, etc.).

Publications

Peer reviewed articles (5)

Preserving Memories of Contemporary Witnesses Using Volumetric Video

Auteurs: Oliver Schreer, Markus Worchel, Rodrigo Diaz, Sylvain Renault, Wieland Morgenstern, Ingo Feldmann, Marcus Zepp, Anna Hilsmann, Peter Eisert
Publié dans: Journal of Interactive Media, Numéro Volume 21 Numéro 1, 2022, Page(s) 71-82, ISSN 2196-6826
Éditeur: Oldenbourg Wissenschaftsverlag
DOI: 10.1515/icom-2022-0015

Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation

Auteurs: A. Zimmer, A. Hilsmann, W. Morgenstern, P. Eisert
Publié dans: Comp. Visual Media, Numéro 9, 2022, Page(s) 123–139, ISSN 2096-0662
Éditeur: Tsinghua University Press and Springer
DOI: 10.48550/arxiv.2202.03074

Study on Automatic 3D Facial Caricaturization: from Rules to Deep Learning

Auteurs: Nicolas Olivier, Glenn Kerbiriou, Ferran Argelaguet, Quentin Avril, Fabien Danieau, Philippe Guillotel, Ludovic Hoyet and Franck Multon
Publié dans: Frontiers in Virtual Reality, 2022, ISSN 2673-4192
Éditeur: Frontiers in Virtual Reality
DOI: 10.3389/frvir.2021.785104

Example-Based Facial Animation of Virtual Reality Avatars using Auto-Regressive Neural Networks

Auteurs: Wolfgang Paier, Anna Hilsmann, Peter Eisert
Publié dans: IEEE Computer Graphics and Applications, 2021, Page(s) 1-1, ISSN 0272-1716
Éditeur: Institute of Electrical and Electronics Engineers
DOI: 10.1109/mcg.2021.3068035

FaceTuneGAN: Face Autoencoder for Convolutional Expression Transfer Using Neural Generative Adversarial Networks

Auteurs: N. Olivier, K. Baert, F. Danieau, F. Multon, Q. Avril
Publié dans: Computers & Graphics Volume, Numéro 110, 2022, Page(s) Pages 69-85, ISSN 0097-8493
Éditeur: Pergamon Press Ltd.
DOI: 10.48550/arxiv.2112.00532

Conference proceedings (8)

Neural Face Models for Example-Based Visual Speech Synthesis

Auteurs: Wolfgang Paier, Anna Hilsmann, Peter Eisert
Publié dans: European Conference on Visual Media Production, 2020, Page(s) 1-10, ISBN 9781450381987
Éditeur: ACM
DOI: 10.1145/3429341.3429356

Temporal Shape Transfer Network for 3D Human Motion

Auteurs: J. Regateiro and E. Boyer
Publié dans: International Conference on 3D Vision (3DV), 2022
Éditeur: 3DV2022

Smart Motion Trails for Animating in VR

Auteurs: J-B. Bordier, A. Mirabile, R. Courant, M. Christie
Publié dans: IEEE Conference on AIVR 2022, 2022, ISBN 978-1-6654-5725-5
Éditeur: IEEE

Recovering Fine Details for Neural Implicit Surface Reconstruction

Auteurs: D. Chen, P. Zhang, I. Feldmann, O. Schreer, P. Eisert
Publié dans: IEEE/CVF Winter Conference on Applications of Computer Vision, 2023
Éditeur: IEEE
DOI: 10.48550/arxiv.2211.11320

Model-Based Deep Portrait Relighting

Auteurs: F. Schreiber, A. Hilsmann, P. Eisert
Publié dans: European Conference on Visual Media Production, 2022
Éditeur: ACM
DOI: 10.1145/3565516.3565526

Detailed Eye Region Capture and Animation

Auteurs: G. Kerbiriou, M. Marchal, Q. Avril
Publié dans: CM SIGGRAPH / Eurographics Symposium on Computer Animation 2022, Numéro Volume 41 (2022), Number 8, 2022
Éditeur: The Eurographics Association and John Wiley & Sons Ltd.
DOI: 10.1111/cgf.14642

Neural Human Deformation Transfer

Auteurs: Basset, Jean; Boukhayma, Adnane; Wuhrer, Stefanie; Multon, Franck; Boyer, Edmond
Publié dans: 3DV 2021 - 9th International Conference on 3D Vision, Dec 2021, London (on line event), United Kingdom., 2021, Page(s) pp.1-12
Éditeur: IEEE
DOI: 10.1109/3dv53792.2021.00064

Multi-View Mesh Reconstruction with Neural Deferred Shading

Auteurs: M. Worchel, R. Diaz, W. Hu, O. Schreer, I. Feldmann, P. Eisert
Publié dans: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, ISBN 978-1-6654-6946-3
Éditeur: IEEE
DOI: 10.1109/cvpr52688.2022.00609

Other (2)

Measuring the benefits of VR animation tools over traditional interfaces

Auteurs: Jean-Baptiste Bordier, Anthony Mirabile, Robin Courant, Marc Christie
Publié dans: ACM Motion in Games, 2022
Éditeur: ACM Motion in Games

Volograms & V-SENSE Volumetric Video Dataset

Auteurs: Pagés, Rafael; Amplianitis, Konstantinos; Ondrej, Jan; Zerman, Emin; Aljosa Smolic
Publié dans: 2022
Éditeur: ResearchGate
DOI: 10.13140/rg.2.2.24235.31529/1