Periodic Reporting for period 2 - Terpsichore (Transforming Intangible Folkloric Performing Arts into Tangible Choreographic Digital Objects)
Berichtszeitraum: 2018-04-01 bis 2020-03-31
This second periodic report presents the overall activities that have been carried out during the lifecycle of the Terpsichore project. Terpsichore project aims to research, analyse, develop, and implement innovative and collaborative research from heterogeneous stakeholders in multi-disciplinary sectors. The project targets at integrating the latest innovative results of photogrammetry, computer vision, semantic technologies, time evolved, 3D modeling, combined with the story telling and folklore choreography. Terpsichore involves collaboration with universities, and companies.
The overall objective of this project is to support a set of services such as virtual/augmented reality, social media, choreography database, interactive maps, presentation and learning of European Folk danced with tremendous impact on the society, culture and tourism.
The purpose of this report is to inform the stakeholders about the Terpsichore project’s achievements that have been carried out during the twenty-four months of its activity. In particular, this periodic report includes the main achievements with respect to the key communication tools, the publications and the scientific framework that the consortium has identified and selected, collaboration opportunities pursued, awareness raising actions and the financial statements prepared and presented. In addition, in this report is presented the overall framework and the transferred knowledge during the secondments and the clustering activities, the transferring knowledge from education activities, the international activities through organization of special sessions and workshops, the Terpsichore summer schools, the secondments training activities and the built-in return mechanism for knowledge sharing and long term collaboration.
The overall objective of this project is to support a set of services such as virtual/augmented reality, social media, choreography database, interactive maps, presentation and learning of European Folk danced with tremendous impact on the society, culture and tourism.
The purpose of this report is to inform the stakeholders about the Terpsichore project’s achievements that have been carried out during the twenty-four months of its activity. In particular, this periodic report includes the main achievements with respect to the key communication tools, the publications and the scientific framework that the consortium has identified and selected, collaboration opportunities pursued, awareness raising actions and the financial statements prepared and presented. In addition, in this report is presented the overall framework and the transferred knowledge during the secondments and the clustering activities, the transferring knowledge from education activities, the international activities through organization of special sessions and workshops, the Terpsichore summer schools, the secondments training activities and the built-in return mechanism for knowledge sharing and long term collaboration.
The work performed from the beginning of the project until M48 is summarized below:
Three datasets have been captured supporting the research activities the Terpsichore project. The second dataset has been captured in Brno and the third dataset in Trikala. More details about these motion capturing process are presented in D6.3 and 6.2.
Research on choreographic analysis (D1.2) was successfully concluded.
Research on fusion techniques (D2.3) motion patterns extraction (D4.1) and temporal processing algorithms have been conducted in this period, with the main purpose of temporally processing and defining the main choreographic patterns of the folklore choreographies captured in the aforementioned datasets.
Research on background and foreground algorithms (D3.1) have been carried out in order to extract the main choreographic silhouette reducing the noise of the RGB images.
Research on surface estimation has been carried out (D3.2).
Research on visualization techniques has been conducted in D5.1.
New algorithms for a) foreground estimation of the dancers, b) motion analysis and automatic extraction of choreographic patterns, c) summarization of the choreographies, d) pose estimation using Deep Learning algorithms, e) surface estimation, f) automatic pose identification.
Several dissemination activities including paper publications, workshop organizations, summer schools have been carried out as described in D7.5 D7.4 D7.6.
Terpsichore consortium during the fourth years of the project carried out successfully three Summer Schools. Terpsichore presence at workshops and conferences is really important since each partner receive training on keys aspects regarding the Terpsichore objectives.
An accurate exploitation plan was created in D7.2 defining the market potential for the reuse of European folklore dances by creative industries with respect to Virtual Reality/Augmented Reality and video games market. Due to the fact that the performing arts community is fragmented and not straightforward connected with the creative industry, as the VR/AR community (XR4ALL), the dance market is quite unexplored.
A connection with the existing digital libraries approached in D6.4. Europeana provides content in image, text, video, sound and 3D format (.c3d, . fbx or obj) will be included soon. All of the aforementioned content will be used by the creative industry and the XR4LL community in order to create interactive applications. The ultimate scope is to enhance those creative industries, cultural heritage organizations, professionals, researchers, performers and occasional dancers to use Europeana’s content in creating applications boosting the Europeans’ Gross Domestic Product.
In total, the consortium has published 63 papers.
Three datasets have been captured supporting the research activities the Terpsichore project. The second dataset has been captured in Brno and the third dataset in Trikala. More details about these motion capturing process are presented in D6.3 and 6.2.
Research on choreographic analysis (D1.2) was successfully concluded.
Research on fusion techniques (D2.3) motion patterns extraction (D4.1) and temporal processing algorithms have been conducted in this period, with the main purpose of temporally processing and defining the main choreographic patterns of the folklore choreographies captured in the aforementioned datasets.
Research on background and foreground algorithms (D3.1) have been carried out in order to extract the main choreographic silhouette reducing the noise of the RGB images.
Research on surface estimation has been carried out (D3.2).
Research on visualization techniques has been conducted in D5.1.
New algorithms for a) foreground estimation of the dancers, b) motion analysis and automatic extraction of choreographic patterns, c) summarization of the choreographies, d) pose estimation using Deep Learning algorithms, e) surface estimation, f) automatic pose identification.
Several dissemination activities including paper publications, workshop organizations, summer schools have been carried out as described in D7.5 D7.4 D7.6.
Terpsichore consortium during the fourth years of the project carried out successfully three Summer Schools. Terpsichore presence at workshops and conferences is really important since each partner receive training on keys aspects regarding the Terpsichore objectives.
An accurate exploitation plan was created in D7.2 defining the market potential for the reuse of European folklore dances by creative industries with respect to Virtual Reality/Augmented Reality and video games market. Due to the fact that the performing arts community is fragmented and not straightforward connected with the creative industry, as the VR/AR community (XR4ALL), the dance market is quite unexplored.
A connection with the existing digital libraries approached in D6.4. Europeana provides content in image, text, video, sound and 3D format (.c3d, . fbx or obj) will be included soon. All of the aforementioned content will be used by the creative industry and the XR4LL community in order to create interactive applications. The ultimate scope is to enhance those creative industries, cultural heritage organizations, professionals, researchers, performers and occasional dancers to use Europeana’s content in creating applications boosting the Europeans’ Gross Domestic Product.
In total, the consortium has published 63 papers.
Within Terpsichore framework new algorithms have been developed focusing on (i) dance summarization of choreographic data that include point joints, (ii) dance classification and (iii) posture identification algorithms including deep machine learning, (iv) deployment of a new game that can be used for dance educational purposes enhancing the social impact of the project, (v) new calibration tools for depth sensors, (vii) a semantic metadata description framework for dances, (viii) Labanotation of the dance movements, (ix) and the creation of three innovative dance dataset.
Terpsichore consortium during the lifecycle of the project carried out successfully three Summer Schools and many dissemination events. Terpsichore presence at workshops and conferences is really important since each partner receive training on keys aspects regarding the Terpsichore objectives. The transdisciplinary environment and the diversity of the participant’s research backgrounds is a source of scientific inspiration for new approaches, collaborations and research activities. At this point is important to mention that the presence of the Terpsichore consortium at various workshops and conferences disseminating the importance and the rehabilitation of the intangible cultural heritage. Moreover, the transdisciplinary and inter-sectorial knowledge via the presence at conferences and workshops providing Terpsichore partners the high-quality learning.
To overcome the challenges of the innovative project, we carried out research in the area of 3D capturing and imaging, computer vision, machine learning, 3D modeling, symbolic representation and finally virtual scene generation. In particular, the main research objectives in 3D computer vision research focused on 3D modeling process, while maintain high resolution accuracy. Additionally, the research is focused on complex background environments and of moving objects. To address these difficulties in the Terpsichore project, we have introduced a scalable capturing framework by incorporating state of the art devices able to acquire depth information in real-time constraints. Moreover, the current 3D modeling algorithms are not appropriate of complex human movements and complex background regions. Additionally, in case where multiple dancers interact with each other and with the environment several research challenges are emerged. To address these aspects, we need, on the one hand, computer vision tools, able to detect the foreground/background content under a highly dynamic framework, to track geometrically enriched points of interest through time and finally to estimate 3D skeletons from the 3D voxels. In addition, we need technologies able to fit the 3D skeletons into pre-defined deformable models. Furthermore, the general advantage of the project is that a complete pipeline from capturing to digital visualization.
Terpsichore consortium during the lifecycle of the project carried out successfully three Summer Schools and many dissemination events. Terpsichore presence at workshops and conferences is really important since each partner receive training on keys aspects regarding the Terpsichore objectives. The transdisciplinary environment and the diversity of the participant’s research backgrounds is a source of scientific inspiration for new approaches, collaborations and research activities. At this point is important to mention that the presence of the Terpsichore consortium at various workshops and conferences disseminating the importance and the rehabilitation of the intangible cultural heritage. Moreover, the transdisciplinary and inter-sectorial knowledge via the presence at conferences and workshops providing Terpsichore partners the high-quality learning.
To overcome the challenges of the innovative project, we carried out research in the area of 3D capturing and imaging, computer vision, machine learning, 3D modeling, symbolic representation and finally virtual scene generation. In particular, the main research objectives in 3D computer vision research focused on 3D modeling process, while maintain high resolution accuracy. Additionally, the research is focused on complex background environments and of moving objects. To address these difficulties in the Terpsichore project, we have introduced a scalable capturing framework by incorporating state of the art devices able to acquire depth information in real-time constraints. Moreover, the current 3D modeling algorithms are not appropriate of complex human movements and complex background regions. Additionally, in case where multiple dancers interact with each other and with the environment several research challenges are emerged. To address these aspects, we need, on the one hand, computer vision tools, able to detect the foreground/background content under a highly dynamic framework, to track geometrically enriched points of interest through time and finally to estimate 3D skeletons from the 3D voxels. In addition, we need technologies able to fit the 3D skeletons into pre-defined deformable models. Furthermore, the general advantage of the project is that a complete pipeline from capturing to digital visualization.