Periodic Reporting for period 1 - PREMIERE (Performing arts in a new era: AI and XR tools for better understanding, preservation, enjoyment and accessibility)
Période du rapport: 2022-10-01 au 2023-12-31
PREMIERE seeks to modernize the performing arts (dance,theater) by using digital technologies to support the whole lifecycle of performances. The overall objective is to develop and validate a comprehensive ecosystem of digital applications, powered by leading edge AI, XR and 3D technologies involved in the main stages of the lifecycle of performing arts productions and contribute to the following general goals:
i) Broaden accessibility to audiences that cannot attend live performances due to physical distance, health reasons, or economic reasons.
ii) Enhance the understanding of present and past performances by audiences with diverse cultural, social, historical and educational backgrounds.
iii) Enrich the creative process by adding new elements, dimensions and capabilities, offered by advanced digital technologies, while enabling international (and therefore inter-cultural) collaboration.
iv) Increase the visibility of performances by supporting both spectators in searching for relevant content and producers/curators in communicating their work.
Social sciences and humanities have a key role in PREMIERE. SSH played a very important role in understanding user needs and transforming them in very specific requirements for technologies so that they are beneficial and util to the several target users. Artistic partners of the consortium provide continuous feedback to the technical partners throughout the development of the technologies.
i) Broaden accessibility to audiences that cannot attend live performances due to physical distance, health reasons, or economic reasons.
ii) Enhance the understanding of present and past performances by audiences with diverse cultural, social, historical and educational backgrounds.
iii) Enrich the creative process by adding new elements, dimensions and capabilities, offered by advanced digital technologies, while enabling international (and therefore inter-cultural) collaboration.
iv) Increase the visibility of performances by supporting both spectators in searching for relevant content and producers/curators in communicating their work.
Social sciences and humanities have a key role in PREMIERE. SSH played a very important role in understanding user needs and transforming them in very specific requirements for technologies so that they are beneficial and util to the several target users. Artistic partners of the consortium provide continuous feedback to the technical partners throughout the development of the technologies.
WP2:
-defined user groups and user requirements and technical specifications of the PREMIRE software components
-understood the behaviour of the various algorithms in regards to the video quality and we defined specific criteria for dance and theatre videos and a set of baseline video and audio features that should be extracted
-defined performance capture protocols and the main technologies to capture various facets of the performances such as a performer’s body and facial movement, audio, other objects, etc
-identified the technical requirements for VR streaming and defined the technical specifications of the Premiere Technical Architecture (Image).
WP3:
-adopted and adapted to performing arts domain several audio indexing algorithms focusing on speech, music and sound event analysis
-deployed deep learning architectures to capture nuanced elements of human movement guided by Laban Movement Analysis (LMA)
-developed methods on sentiment analysis based on different modalities such as body motion, facial expression recognition, voice analysis and text analysis
-adapted several mature machine learning methods for language technologies (translation, transcription)
-developed methods for cross-linking of archived data through identifying multi-modal relations between segments within performances that integrates annotations and latent embeddings.
WP4:
-developed several algorithms to analyse videos (illumination, segmentation, object tracking, 2D and 3D human body estimation, human tracking, actions parsing, sentiment analysis and body motion analysis)
-deployed a platform for creating human avatars for the virtual theatre and collect requirements related to avatar fidelity, expressiveness
-developed a 3D scene editing and modelling tool used to design the 3D virtual environments that will host their performances or rehearsals for remote audiences
-pursuited of efficient streaming technologies in the realm of best experience for content delivery aiming to minimize latency, reduce bandwidth requirements, and optimize visual fidelity.
WP5:
-developed the 3D Virtual Theatre that is a virtual space where all performance events will be projected to and incorporates most of the WP3 and WP4 outcomes and supports real-time data streaming, Machine Learning integration, and cross-platform accessibility
-developed a visualisation tool that displays various metadata of the performances originating from various modalities (text, audio, video) into the 3D Virtual Theatre;
-developed a cloud-driven Content Management System aiming at managing Premiere content designed to streamline the creation, editing, and distribution of the content
-an AI toolbox oriented to analyse and generate movement, sound and image, all of them oriented to create a tight connection between body movement and media.
WP6:
Use case 1:Development of an application for browsing theatre and dance archives. We selected specific plays and performances, run several AI algorithms for content enrichment, and developed an annotation tool for correcting annotations.
Use case 2:Experimented with the design and selection of appropriate representations of human avatars taking into account fidelity, facial animation and movement from performance capture devices in order to broadcast a real performance in a physical space into VR
Use case 3: Developed a demo version of the 3D Virtual Theatre entrly point of this application and that will allow remote rehearsing and co-creation. We considered two different motion capture systems (Xsens and Rokoko).
Use case 4: Developed the AI toolbox that contain a series of tools were based on Machine Learning and Artificial Intelligence targeted to artistic creations and was integrated into two dance performances that were shown in public.
-defined user groups and user requirements and technical specifications of the PREMIRE software components
-understood the behaviour of the various algorithms in regards to the video quality and we defined specific criteria for dance and theatre videos and a set of baseline video and audio features that should be extracted
-defined performance capture protocols and the main technologies to capture various facets of the performances such as a performer’s body and facial movement, audio, other objects, etc
-identified the technical requirements for VR streaming and defined the technical specifications of the Premiere Technical Architecture (Image).
WP3:
-adopted and adapted to performing arts domain several audio indexing algorithms focusing on speech, music and sound event analysis
-deployed deep learning architectures to capture nuanced elements of human movement guided by Laban Movement Analysis (LMA)
-developed methods on sentiment analysis based on different modalities such as body motion, facial expression recognition, voice analysis and text analysis
-adapted several mature machine learning methods for language technologies (translation, transcription)
-developed methods for cross-linking of archived data through identifying multi-modal relations between segments within performances that integrates annotations and latent embeddings.
WP4:
-developed several algorithms to analyse videos (illumination, segmentation, object tracking, 2D and 3D human body estimation, human tracking, actions parsing, sentiment analysis and body motion analysis)
-deployed a platform for creating human avatars for the virtual theatre and collect requirements related to avatar fidelity, expressiveness
-developed a 3D scene editing and modelling tool used to design the 3D virtual environments that will host their performances or rehearsals for remote audiences
-pursuited of efficient streaming technologies in the realm of best experience for content delivery aiming to minimize latency, reduce bandwidth requirements, and optimize visual fidelity.
WP5:
-developed the 3D Virtual Theatre that is a virtual space where all performance events will be projected to and incorporates most of the WP3 and WP4 outcomes and supports real-time data streaming, Machine Learning integration, and cross-platform accessibility
-developed a visualisation tool that displays various metadata of the performances originating from various modalities (text, audio, video) into the 3D Virtual Theatre;
-developed a cloud-driven Content Management System aiming at managing Premiere content designed to streamline the creation, editing, and distribution of the content
-an AI toolbox oriented to analyse and generate movement, sound and image, all of them oriented to create a tight connection between body movement and media.
WP6:
Use case 1:Development of an application for browsing theatre and dance archives. We selected specific plays and performances, run several AI algorithms for content enrichment, and developed an annotation tool for correcting annotations.
Use case 2:Experimented with the design and selection of appropriate representations of human avatars taking into account fidelity, facial animation and movement from performance capture devices in order to broadcast a real performance in a physical space into VR
Use case 3: Developed a demo version of the 3D Virtual Theatre entrly point of this application and that will allow remote rehearsing and co-creation. We considered two different motion capture systems (Xsens and Rokoko).
Use case 4: Developed the AI toolbox that contain a series of tools were based on Machine Learning and Artificial Intelligence targeted to artistic creations and was integrated into two dance performances that were shown in public.
PREMIERE has moved towards scientific advancements in three main scientific areas:
AI technologies: Performing arts have the specifities that performers’ behaviour/acting goes beyond ordinary, resulting in circumstances that consequently cannot be tackled effectively by current state-of-art algorithms. For example, skeleton analysis/tracking algorithms from videos or sensor data are trained on populations that do not have the movement capabilities of professional dancers, and this fine information is lost when capturing dancer movements. We evaluated several SOTA methods in many recognition tasks and understood their limitations and the development is progressing towards more refined and domain-specific methods.
VR technologies: they have not been systematically and extensively tested in the context of performing arts, that exhibit some specifities that SOTA algorithms are not effective to tackle. The transmission for example of a human expression (or even intention) is very delicate to be transmitted through VR. PREMIERE has identified these aspects; thanks to its interdisciplinary consortium we focus on delivering VR technologies that are valid and domain-informed in the context of performing arts.
Interdisciplinary research on performing arts and new technologies: there is a scientific gap on understanding the relationship between the new technologies and the performing arts: What are the specific needs with respect to technological developments of several stakeholder categories? What are the implications of using XR technologies in the lifecycle of a dance/theatre piece (creation, production, delivery, preservation)? Which the ethical implications of using such technologies in the performing arts? How technology can be beneficial to the context of performing arts, as for example for education and accessibility?
AI technologies: Performing arts have the specifities that performers’ behaviour/acting goes beyond ordinary, resulting in circumstances that consequently cannot be tackled effectively by current state-of-art algorithms. For example, skeleton analysis/tracking algorithms from videos or sensor data are trained on populations that do not have the movement capabilities of professional dancers, and this fine information is lost when capturing dancer movements. We evaluated several SOTA methods in many recognition tasks and understood their limitations and the development is progressing towards more refined and domain-specific methods.
VR technologies: they have not been systematically and extensively tested in the context of performing arts, that exhibit some specifities that SOTA algorithms are not effective to tackle. The transmission for example of a human expression (or even intention) is very delicate to be transmitted through VR. PREMIERE has identified these aspects; thanks to its interdisciplinary consortium we focus on delivering VR technologies that are valid and domain-informed in the context of performing arts.
Interdisciplinary research on performing arts and new technologies: there is a scientific gap on understanding the relationship between the new technologies and the performing arts: What are the specific needs with respect to technological developments of several stakeholder categories? What are the implications of using XR technologies in the lifecycle of a dance/theatre piece (creation, production, delivery, preservation)? Which the ethical implications of using such technologies in the performing arts? How technology can be beneficial to the context of performing arts, as for example for education and accessibility?