WP2:
-defined user groups and user requirements and technical specifications of the PREMIRE software components
-understood the behaviour of the various algorithms in regards to the video quality and we defined specific criteria for dance and theatre videos and a set of baseline video and audio features that should be extracted
-defined performance capture protocols and the main technologies to capture various facets of the performances such as a performer’s body and facial movement, audio, other objects, etc
-identified the technical requirements for VR streaming and defined the technical specifications of the Premiere Technical Architecture (Image).
WP3:
-adopted and adapted to performing arts domain several audio indexing algorithms focusing on speech, music and sound event analysis
-deployed deep learning architectures to capture nuanced elements of human movement guided by Laban Movement Analysis (LMA)
-developed methods on sentiment analysis based on different modalities such as body motion, facial expression recognition, voice analysis and text analysis
-adapted several mature machine learning methods for language technologies (translation, transcription)
-developed methods for cross-linking of archived data through identifying multi-modal relations between segments within performances that integrates annotations and latent embeddings.
WP4:
-developed several algorithms to analyse videos (illumination, segmentation, object tracking, 2D and 3D human body estimation, human tracking, actions parsing, sentiment analysis and body motion analysis)
-deployed a platform for creating human avatars for the virtual theatre and collect requirements related to avatar fidelity, expressiveness
-developed a 3D scene editing and modelling tool used to design the 3D virtual environments that will host their performances or rehearsals for remote audiences
-pursuited of efficient streaming technologies in the realm of best experience for content delivery aiming to minimize latency, reduce bandwidth requirements, and optimize visual fidelity.
WP5:
-developed the 3D Virtual Theatre that is a virtual space where all performance events will be projected to and incorporates most of the WP3 and WP4 outcomes and supports real-time data streaming, Machine Learning integration, and cross-platform accessibility
-developed a visualisation tool that displays various metadata of the performances originating from various modalities (text, audio, video) into the 3D Virtual Theatre;
-developed a cloud-driven Content Management System aiming at managing Premiere content designed to streamline the creation, editing, and distribution of the content
-an AI toolbox oriented to analyse and generate movement, sound and image, all of them oriented to create a tight connection between body movement and media.
WP6:
Use case 1:Development of an application for browsing theatre and dance archives. We selected specific plays and performances, run several AI algorithms for content enrichment, and developed an annotation tool for correcting annotations.
Use case 2:Experimented with the design and selection of appropriate representations of human avatars taking into account fidelity, facial animation and movement from performance capture devices in order to broadcast a real performance in a physical space into VR
Use case 3: Developed a demo version of the 3D Virtual Theatre entrly point of this application and that will allow remote rehearsing and co-creation. We considered two different motion capture systems (Xsens and Rokoko).
Use case 4: Developed the AI toolbox that contain a series of tools were based on Machine Learning and Artificial Intelligence targeted to artistic creations and was integrated into two dance performances that were shown in public.
