During first Reporting Period, the consortium:
- Focused on improving the different stages that compose the volumetric capture pipeline to improve both the visual quality of textures and the volumetric mesh reconstruction. Different machine-learning approaches were investigated and successfully applied. Volumetric capture sessions were performed in preparation for the evaluations and made available to the consortium.
- Has developed authoring tools to assist the designers in the capture and design of volumetric-based characters, improving the visual quality and adapting the mesh to the intended context. Facial style transfer techniques were proposed to enhance creativity. Techniques of motion adaptation were also explored to adapt the animations of volumetric characters to scene constraints.
- Has delivered the Open-Source tool VRTist which proposes VR authoring capacities for the design of linear and non-linear narratives. The tool includes: scene layout features, lighting features, camera control features including framing and focus, animation features, and shot editing.
- Has identified two use-cases which will be implemented and against which the tools and technologies will be evaluated in the second reporting period.
During the second Reporting Period, the consortium:
- Explored new modalities for volumetric reconstruction to improve both the quality and the realism of the outputs by more strongly relying on machine learning techniques, also improving robustness of the methods. Test data was acquired in the volumetric capture studio especially dedicated to planned experiments and research. Moreover, new methods for the generation of animatable avatars from the volumetric video data have been developed, including pre-processing of the data. Especially, generative models have been trained from the volumetric video data. Finally, neural implicit representations were tested as an alternative representation.
- Completed the creation pipeline with additional functionalities (hairs, eyes), both for modelling and animation (speech and body motion). An innovative new model for the eye region has been developed. Also, different AI-based algorithms have been designed and tested. In particular, a deep learning method for the synthesis of gestures from speech, a retargeting method for facial expressions and a semantic facial features extraction and deep generative reconstruction for face animation. In addition, the model used during the project (Morgan) has been proposed as a new reference model for humanoid avatars within the MPEG standard.
- Has delivered an updated version of the VRTist authoring tool that first enables the intuitive editing of complex character motions using manipulatable motion trails, and second enables the interactive manipulation of Volumetric Captured characters using Deep Learning generative models. The tools integrated also the use of artist-driven rigs, together with numerous features that exploit VR capacities to improve artists productivity and creativity.
- Relied on the technologies designed and implemented during the project to conduct a number of experimentations to assess their applicability and usability in use-cases. Technologies were also demonstrated in both professional shows and general audience events. All experiments laid the ground for an assessment of the project KPIs which were successfully reached.
- Identified seven Key Exploitable Results (KERs) which will be exploited as the basis for further education and research, product, services, licence/transfer or open-source software. Two results are already available to the public:
a) VRTist: an open source VR design tool dedicated the creation of 3D environments and animations for rapidly prototyping scenes and shots (
https://github.com/Irisa-Invictus/VRTist(odnośnik otworzy się w nowym oknie)).
b) Volu: an iOS application that allows anyone with a mobile device to capture, play and share volograms (
https://www.volograms.com/volu(odnośnik otworzy się w nowym oknie)).
