EnTimeMent - ENtrainment and synchronization at multiple TIME scales in the MENTal foundations of expressive gesture

Project Information

EnTimeMent

Grant agreement ID: 824160

DOI

10.3030/824160

Project closed

EC signature date 24 October 2018

Start date 1 January 2019

End date 31 December 2022

Funded under

• EXCELLENT SCIENCE - Future and Emerging Technologies (FET)

Total cost

€ 4 206 385,86

EU contribution

€ 4 188 885,86

Coordinated by

UNIVERSITA DEGLI STUDI DI GENOVA
Italy

Periodic Reporting for period 3 - EnTimeMent (EnTimeMent - ENtrainment and synchronization at multiple TIME scales in the MENTal foundations of expressive gesture)

Reporting period: 2022-01-01 to 2022-12-31

Summary of the context and overall objectives of the project

In 1973, the Swedish psychologist Gunnar Johansson and his colleagues showed that the human visual system can perceive the movement of a human body from a limited number of moving points. This landmark study grounded the scientific basis of current motion capture technologies. However, the state of the art of motion capture technology only considers low-level temporal and spatial information, limited to computation of kinematic measures whose time-frame is usually too short for an effective perception and prediction of complex phenomena. While a lot of effort is being spent improving such technologies in the direction of more accurate and more portable systems (e.g. wearable and wireless), such developments are incremental with respect to a conceptual and technological paradigm that remains unchanged. The Future and Emerging Technologies (FET PROACTIVE) project EnTimeMent proposes a radical change of paradigm and technology in human movement analysis. Under this new perspective, the time-frame for analysis is grounded on novel neuroscientific, biomechanical, psychological, and computational evidence, inspired by the arts, and dynamically adapted to the human time governing the phenomena under investigation, the multiple temporal scales concurrently interacting in the brain.

EnTimeMent includes measurable scientific, technological, and community-building objectives.
Scientific objectives include (i) the definition and empirical validation of neuro-cognitive models of the multiple and mutually interactive time scales that contribute to human perception of gesture qualities and action prediction, (ii) the development of computational models grounded on such neuro-cognitive models, (iii) the investigation of music and dance performers movements synchronised at different temporal scales, and a broader set of controlled as well as ecological experiments.
Technological objectives consist of the development of computational methods and software modules integrated in the project platform, to support experiments and proof-of-concepts in three different real-world scenarios focusing on healing and support of everyday life in disabled persons, sensory-motor entrainment and commitment in sport, dance and music.
Community-building objectives include the building of a novel scientific interdisciplinary community leading the transformation of the current generation of motion capture systems into a novel generation of time-aware motion perception and prediction technologies.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

A number of joint scientific experiments were started by scientific partners; some are concluded and results are available in the project web portal and some of them are already published in scientific journals and conference proceedings (see https://entimement.dibris.unige.it/documents). Scientific results include the development of computational models and software modules based on machine learning, on graph and game theory, on synchronization theories, and on theories on individual motor signature.
Technological results consist in the consolidation of the project platform, including novel Qualisys software on motion capture, integrated with other sensing technologies, and used in scientific experiments and public events and in videos.
Details on dissemination activities are available at https://entimement.dibris.unige.it/events and https://entimement.dibris.unige.it/press
Project deliverables and open-access scientific papers available on the project web page.

The dissemination activities of EnTimeMent include a video submission to FETFX 2020, a number of interviews and media (e.g. https://ec.europa.eu/digital-single-market/en/news/improving-human-movement-analysis-interview-antonio-camurri).
Furthermore, EnTimeMent organizes the ICMI 2020 Intl Workshop on Multi-Scale Movement Technologies (October 2020), and participates to
several other dissemination initiatives.

Results in the second period:
- Models on predictions of multiscale temporally aggregated socialmotor synchronization models (D1.5) models and algorithms (D1.7)
- Experiments: Results on prediction in action execution and observation - Phase 2 (D2.2); Results on prediction in dyadic action execution and observation – Phase 2 (D2.4); Results on prediction in Complex Action execution and observation - Phase 2 (D2.6)
- Data Acquisition and Multi-Time Signal Analysis and Processing: EnTimeMent platform and software libraries for multi-time analysis, entrainment, and prediction - Phase 3. A new release of the project platform and libraries is available on Github (D3.4); Data acquisition analytic tools for complex actions (D3.6 D3.7); Annotated open-source datasets (D3.8);
- Use Case Scenarios: Scenario 1: Proof-of-Concept Testing and Validation in healing and everyday life support of disabled - Phase 2 (D4.2); Scenario 2: Proof-of-Concept Testing and Validation in chronic pain management in everyday life - Phase 2 (D4.5); Scenario 3: Proof-of-Concept Testing and Validation in dance, living architectures, sports and entertainment - Phase 2 (D4.8);
- Dissemination, Communication and Exploitation: see D5.8.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

EnTimeMent includes measurable scientific, technological, and community-building objectives.
Scientific objectives include (i) the definition and empirical validation of neuro-cognitive models of the multiple and mutually interactive time scales that contribute to human perception of gesture qualities and action prediction, (ii) the development of computational models grounded on such neuro-cognitive models, (iii) the investigation of music and dance performers movements synchronised at different temporal scales, and a broader set of controlled as well as ecological experiments.
Technological objectives consist of the development of computational methods and software modules integrated in the project platform, to support experiments and proof-of-concepts in three different real-world scenarios focusing on healing and support of everyday life in disabled persons, sensory-motor entrainment and commitment in sport, dance and music. Community-building objectives include the building of a novel scientific interdisciplinary community leading the transformation of the current generation of motion capture systems into a novel generation of time-aware motion perception and prediction technologies.
The EnTimeMent project will move away from traditional motion capture technology, emphasizing movement analysis, that will not only record movements, but also compute on-the-fly their origins and their emotional and social qualities; This will allow novel scientific approaches for individualized predictions, from the identification of relations between time-scales, prediction of longer time-scales from shorter ones, with impact on individualisation of learning and the design of learning accelerators for healthy and patients. EnTimeMent will also shape new research directions addressing multiple temporal scales, capturing "individual motor signature" and "group motor signature", from biology to social interaction, from support to everyday life of disabled, to therapy and rehabilitation, to dance and music, and in general impact in the Creative and Cultural Sector.

Demo on Individual Motor Signature (from a video available at http://www.casapaganini.org/atempo/)