Enhancing Motion Interaction through Music Performance

Project Information

MIM

Grant agreement ID: 659232

DOI

10.3030/659232

Project closed

EC signature date 9 April 2015

Start date 4 January 2016

End date 3 January 2019

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 232 160,40

EU contribution

€ 232 160,40

Coordinated by

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE CNRS
France

Periodic Reporting for period 2 - MIM (Enhancing Motion Interaction through Music Performance)

Reporting period: 2018-01-04 to 2019-01-03

Summary of the context and overall objectives of the project

As Information Communications Technology (ICT) is now spreading outside of workspaces towards our everyday life, new elements of human life such as culture, emotion or experience must be embraced in technology for Human-Computer Interaction (HCI). Part of this new objective in HCI is to build technologies that are “closer”, or more “natural”, to human, promoting the use of body movement in HCI. Exploring this challenge has been the cornerstone of digital arts and especially performing arts involving digital media. For example, music technologists have leveraged on motion sensing technology to explore new types of interaction between the body and digital medias. Nevertheless, there is a general agreement that these new interactive systems remain less satisfying in terms of expressive control compared to traditional instruments, and, importantly, do not allow to acquire skills through them.

The general objective of the project MIM is to investigate motor learning mechanisms in expert motor activity, especially instrumental music practice, in order to enhance movement computational models in movement-based musical interaction systems. The project MIM addresses three specific scientific objectives:
- Objective 1: Study the role of movement task variation in music skill acquisition,
- Objective 2: Study the nature of movement variability and learning schedules in the development of motor learning,
- Objective 3: Build interactive systems facilitating motor learning.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project started by an exhaustive literature review on sensorimotor learning and music performance.

A first experimental study has been conducted showing that the motor task variability has a dissociable effect on timing and motor skill acquisition in piano sequence learning. The results were published in PLoS ONE:
- Caramiaux, B., Bevilacqua, F., Wanderley, M., and Palmer, C. Dissociable effects of practice variability on learning motor and timing skills. PLOS ONE. 13(3), 1-18. 2018.

A second experimental study has been initiated (pilot study has been run) investigating the individual-specificity of motor variability in piano sequence learning and the role of cognitive and biomechanical processes on motor variability. A conference proceedings showing preliminary results has been published in MOCO:
- Caramiaux, B., Bevilacqua, F., Palmer, C., and Wanderley, M. Individuality in Piano Performance Depends on Skill Learning. Proceedings of the 4th International Conference on Movement Computing. 2017

Design methodology to transfer this knowledge to the realisation of movement-based music systems have been explored. A collaborative paper reports on design highlights:
- Höök, K., Caramiaux, B., Erkut, C., Forlizzi, J., Hajinejad, N., et al. Embracing First-Person Perspectives in Soma-Based Design. Informatics. 5(1), 8. 2018

New design methodologies have been applied to better understand complex motor learning and learning schedule:
- Rivière, JP, Fdili Alaoui, S, Caramiaux, B., and Mackay WE. How Do Dancers Learn to Dance? A first-person perspective of dance acquisition by expert contemporary dancers. Proceedings of the 5th International Conference on Movement and Computing, 6. 2018

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

While often considered as noise stemming from the motor or the sensory system, motor variability has recently been shown to have an important role in sensorimotor learning. Underlying processes vary the motor output in order to learn faster and better certain changes in environment dynamics (motor adaptation) or a new control policy (skill acquisition). This new perspective of motor variability has gained interest since the seminal Nature article in 2014. In music performance, variability has been barely explored. Few studies have shown that motor variability can be used to highlight chunking. However, motor variability as a parameter for skill acquisition in music remains unexplored.

Our first experimental study brings a new contribution to the state of the art in music performance and sensorimotor learning showing the impact of motor variability on motor and timing skill learning.

Based on our first experiments, we extended the methodology to include learner-centred persepctive of learning. These qualitative data helped the understanding of complex motor learning and drove the design of an interactive prototype aiming at documenting learning schedule and facilitating the learner along the process.

The impact is important for music pedagogy and more generally body-based activity, with implications in the design of innovative skill-facilitating interactive designs. These systems aim to provide adapted learning schedules to the learner. Such impact has recently been discussed during the Piano & gesture capture workshop at GREAM, at University of Strasbourg, and during a workshop dedicated to the topic at the Conference NIME (New Interfaces for Musical Expression) in Copenhagen, and finally a wider-scoped research workshop on Human-Machine Collaboration in Embodied Interaction, at IRCAM, Paris.

Piano skill acquisition