Periodic Reporting for period 2 - TELMI (Technology Enhanced Learning of Musical Instrument Performance)
Reporting period: 2017-08-01 to 2019-01-31
Music pedagogy is mostly based on a master-apprentice model in which the student observes and imitates the teacher, the teacher provides verbal feedback and the student engages in long periods of self-study. The time lag between the student‘s performance and the teacher‘s feedback makes the feedback to be dissociated from the online proprioceptive and auditory sensations accompanying the performance. The resulting long periods of private-study by the student frequently make the learning of musical instruments a rather harsh and solitary experience. In addition, approaches to teaching the biomechanics of musical performance may be based on subjective and vague perception, rather than on accurate understanding of the principles of human movement. Other factors such as the belief in the need for long hours of repetitive practice to acquire technical skills or the competitive nature of music performance often result in inefficient practice. For the student, consequences of such an uninformed pedagogy can range from a frustrating lack of progress, and consequent abandonment, to health problems.
TELMI addresses the deficiencies of traditional performance learning by contributing to design and implement new multi-modal interaction paradigms for music learning and to develop assistive, self-learning, and social-aware prototypes complementary to traditional teaching. Concretely, as a result of a tightly coupled interaction between technical and pedagogical partners, TELMI aims (1) to design and implement new multi-modal interaction paradigms for music learning and training based on state-of-the-art audio processing, and motion capture technologies, (2) to evaluate from a pedagogical point of view the effectiveness of such new paradigms, (3) based on the evaluation results, to develop new multi-modal interactive music learning prototypes, (4) to create a publicly available reference database of multimodal recordings with data analytics tools for assessing student knowledge and progress and (5) to contribute to the engagement of musicians in the community and making more appealing music practise and instrument training.
TELMI addresses the deficiencies of traditional performance learning by contributing to design and implement new multi-modal interaction paradigms for music learning and to develop assistive, self-learning, and social-aware prototypes complementary to traditional teaching. Concretely, as a result of a tightly coupled interaction between technical and pedagogical partners, TELMI aims (1) to design and implement new multi-modal interaction paradigms for music learning and training based on state-of-the-art audio processing, and motion capture technologies, (2) to evaluate from a pedagogical point of view the effectiveness of such new paradigms, (3) based on the evaluation results, to develop new multi-modal interactive music learning prototypes, (4) to create a publicly available reference database of multimodal recordings with data analytics tools for assessing student knowledge and progress and (5) to contribute to the engagement of musicians in the community and making more appealing music practise and instrument training.
Design of new multi-modal interaction paradigms for music learning: based on the fundamental violin techniques, a series of 41 TELMI exercises was assembled and four professional violinists recorded expert models of these exercises. Using these exercises as a basis, the the partners worked to develop score rendering software so that the exercises can be displayed with incorporated feedback overlays within the final TELMI system. This is exemplified by the development of the PLANNER App and the SKYNOTE App that not only incorporate evaluation and feedback from the audio and motion capture systems but encourage and develop the full range of practice habits and skills described by self-regulated learning and deliberate practice principles, including planning and review strategies.
Evaluation from a pedagogical point of view the effectiveness of such new paradigms: a core group of high-end violinists and teachers were recruited, based at the RCM and internationally, with whom there has been engagement at several levels of data collection through the tasks and who were called upon Phase 2 project evaluations. Evaluation studies were designed and implemented examining technology use to aid bowing technique, tuning and timing accuracy, and tone quality, the evaluation of performers’ motion under stress conditions, and workshops examining reception to the TELMI App. A study involving violin participants at the RCM tested the accuracy of violin exercises of different complexity with and without the use of technology, obtaining very positive and interesting results.
Development of new multi-modal interactive music learning prototypes: systems for the acquisition of multimodal data from musical performances were developed. The systems use complementary measurement methods such as high quality motion capture systems, RGB-D sensors, 2D video, multichannel audio, biometric signals, inertial sensors and electromagnetic field sensors. The main modalities acquired from the musical performances are the sound, violin and bow motion, human body motion and biometric signals.
Creation of a publicly available reference database of multimodal recordings with data analytics tools: a database with the reference recordings from masters has been built using the repoVizz framework. The framework allows remote storage and visualization of multimodal signals. In particular, the TELMI recordings can be accessed through different visualizations. A customized high level 3D visualizations of body and instrument gestures has been implemented.
Dissemination: the project has produced 30+ publications, 14 conferences, and 7 workshops, and it has raised awareness on the role of technology in music learning (2000+ practitioners reached).
Evaluation from a pedagogical point of view the effectiveness of such new paradigms: a core group of high-end violinists and teachers were recruited, based at the RCM and internationally, with whom there has been engagement at several levels of data collection through the tasks and who were called upon Phase 2 project evaluations. Evaluation studies were designed and implemented examining technology use to aid bowing technique, tuning and timing accuracy, and tone quality, the evaluation of performers’ motion under stress conditions, and workshops examining reception to the TELMI App. A study involving violin participants at the RCM tested the accuracy of violin exercises of different complexity with and without the use of technology, obtaining very positive and interesting results.
Development of new multi-modal interactive music learning prototypes: systems for the acquisition of multimodal data from musical performances were developed. The systems use complementary measurement methods such as high quality motion capture systems, RGB-D sensors, 2D video, multichannel audio, biometric signals, inertial sensors and electromagnetic field sensors. The main modalities acquired from the musical performances are the sound, violin and bow motion, human body motion and biometric signals.
Creation of a publicly available reference database of multimodal recordings with data analytics tools: a database with the reference recordings from masters has been built using the repoVizz framework. The framework allows remote storage and visualization of multimodal signals. In particular, the TELMI recordings can be accessed through different visualizations. A customized high level 3D visualizations of body and instrument gestures has been implemented.
Dissemination: the project has produced 30+ publications, 14 conferences, and 7 workshops, and it has raised awareness on the role of technology in music learning (2000+ practitioners reached).
Systems for the acquisition of multimodal data from musical performances has been developed using complementary measurement methods such as high quality motion capture systems, RGB sensors, 2D video, multichannel audio, biometric signals, and inertial sensors. The main modalities acquired from the musical performances are the sound, violin and bow motion, human body motion and biometric signals. The system provides Multi-instrument support (Violin, Voice, Viola, Cello, Trumpet, Trombone and Clarinet).
Algorithms for the analysis of audio, motion and physiological signals acquired by the measuring systems were developed. We defined expert movement profiles to improve the students’ motor control for learning and the development of specific skills and analytics of music performance learning. We defined a set of measures to automatically evaluate performance skill and to assess the progress of the students, which are aligned to the human evaluations. We developed two engagement and social applications, the final versions of the TELMI Planner and the TELMI Studio ensure a good user experience to encourage engagement and adoption by students, an online service that permits students and teachers to set up, coordinate and manage studio classes where students are encouraged to share their performance and learn to provide constructive feedback to one another.
The impact of ICT research on pedagogy is expected to increase in the long term. TELMI targeted effort addressed future education needs with broader impact. By producing novel technologies for learning, the project aims at sparking creativity at both professional and amateur level, while stimulating new teaching methodologies. Further societal benefits include new ways of interacting with technologies and media content and new communication systems in and outside of education. In addition, the research outcomes of the project have scientific impacts through a multidisciplinary research agenda at the intersection of pedagogy, motor learning, kinematics, signal processing, computer vision, visualisation and music performance.
As a result of the work and prototypes developed in the project, the constitution of two different spin-off companies are under way, with the potential to create new jobs and thus contributing to the economic ecosystem in the society as a whole.
Algorithms for the analysis of audio, motion and physiological signals acquired by the measuring systems were developed. We defined expert movement profiles to improve the students’ motor control for learning and the development of specific skills and analytics of music performance learning. We defined a set of measures to automatically evaluate performance skill and to assess the progress of the students, which are aligned to the human evaluations. We developed two engagement and social applications, the final versions of the TELMI Planner and the TELMI Studio ensure a good user experience to encourage engagement and adoption by students, an online service that permits students and teachers to set up, coordinate and manage studio classes where students are encouraged to share their performance and learn to provide constructive feedback to one another.
The impact of ICT research on pedagogy is expected to increase in the long term. TELMI targeted effort addressed future education needs with broader impact. By producing novel technologies for learning, the project aims at sparking creativity at both professional and amateur level, while stimulating new teaching methodologies. Further societal benefits include new ways of interacting with technologies and media content and new communication systems in and outside of education. In addition, the research outcomes of the project have scientific impacts through a multidisciplinary research agenda at the intersection of pedagogy, motor learning, kinematics, signal processing, computer vision, visualisation and music performance.
As a result of the work and prototypes developed in the project, the constitution of two different spin-off companies are under way, with the potential to create new jobs and thus contributing to the economic ecosystem in the society as a whole.