Final Report Summary - METAGESTUREMUSIC (Meta-Gesture Music: Social, Embodied, Interactive Musical Instruments)
The MetaGesture Music project enabled Atau Tanaka to consolidate and expand the Embodied Audiovisual Interaction unit at Goldsmiths to become an internationally recognised research cluster in embodied human-computer interaction (HCI), sonic interaction coupled with User Centered Design (UCD) and machine learning, and the study of body physiology in new musical performance.
The project conducted fundamental research in humans’ corporeal relationship to sound. We studied this in everyday settings, investigating the affordances that sound have to invite embodied interaction, and with the human body as the source of musical gesture. The Form Follows Sound workshop series developed qualitative research methods and deployed advanced gesture-sound interaction software allowing everyday users to remember sonic incidents, storyboard them, and create functioning gesture/sound prototypes to act out these scenarios. We studied the source of musical gesture inside the body by using physiological signals, notably the electromyogram to report on the muscle tension exerted to articulate arm gesture. Sonification and musical transformation of data from the body were used in the composition of concert and performance works by team members and toured internationally. The Gesture Variation Follower (GVF) was developed to carry out machine learning analysis of musical gesture, classifying on a single example, reporting early recognition, and tracking expressive variation. We attempted to apply the GVF on pianist and violinist gesture, using multimodal sensing of EMG alongside motion capture and inertial measurement. These studies pointed out the complexities of musical gesture and physiological signals and point to challenges for future research in feature extraction and machine learning of real world signals in creative applications.
Project outputs have been published in scientific journals, international conferences, and chapters of books. Musical output has been released on CD in partnership with The Wire music magazine, published by Goldsmiths Press, and distributed by NX Records internationally in physical form and in digital download. Our research and creative output have been the subject of reports and documentaries on BBC and Arté. Performances and presentations have taken place in music festivals such as WOMAD, Club Transmediale, and Sonar/MHD.
The core MGM research team was led by Professor Atau Tanaka, bringing to the project his vision of visceral interaction in computer music performance. Dr Baptiste Caramiaux brought expertise in machine learning of musical gesture; and students Alessandro Altavilla and Marco Donnarumma, in their doctoral research, made fundamental contributions to knowledge in sonic incidents, retroactive listening, and body/technology configurations. The team also included Drs Adam Parkinson, Miguel Ortiz, and Andrew Harding, carrying out research on participation, concerts as sites of research, muscle interfaces and statistical analysis of musical gesture.
Intellectual, creative, and public engagement formed important parts of MGM research. The MGM team hosted an international conference, New Interfaces for Musical Expression (NIME) in London in 2014, bringing together international scientific and artistic communities of practice. A series of visiting researcher/visiting artists came to Goldsmiths to engage with the MGM project, sharing knowledge and best practice in the science and art of tangible interfaces, new instrument building and live performance. Notable guests included Prof Hiroshi Ishii from the MIT Media Lab, performer Laetitia Sonami from the San Francisco Art Institute, and sound art scholars Caleb Kelly and Brandon Labelle. The MGM project included an ambitious public events programme to communicate research results to the general audiences in compelling contexts, ranging from hosting school visits, performing and exhibiting at festivals, museums, and galleries to presenting installations and concerts in public parks and pubs, including the ZKM Karlsruhe and the ICA London.
Gender representation in music technology is an important issue. Female representation across the MGM project was excellent, at 34%. We have contributed to discussions on the topic at the MusDig conference, organised a special session during the NIME2014 conference, and have been active in the Women in Sound/Women on Sound (WISWOS) network.
The project conducted fundamental research in humans’ corporeal relationship to sound. We studied this in everyday settings, investigating the affordances that sound have to invite embodied interaction, and with the human body as the source of musical gesture. The Form Follows Sound workshop series developed qualitative research methods and deployed advanced gesture-sound interaction software allowing everyday users to remember sonic incidents, storyboard them, and create functioning gesture/sound prototypes to act out these scenarios. We studied the source of musical gesture inside the body by using physiological signals, notably the electromyogram to report on the muscle tension exerted to articulate arm gesture. Sonification and musical transformation of data from the body were used in the composition of concert and performance works by team members and toured internationally. The Gesture Variation Follower (GVF) was developed to carry out machine learning analysis of musical gesture, classifying on a single example, reporting early recognition, and tracking expressive variation. We attempted to apply the GVF on pianist and violinist gesture, using multimodal sensing of EMG alongside motion capture and inertial measurement. These studies pointed out the complexities of musical gesture and physiological signals and point to challenges for future research in feature extraction and machine learning of real world signals in creative applications.
Project outputs have been published in scientific journals, international conferences, and chapters of books. Musical output has been released on CD in partnership with The Wire music magazine, published by Goldsmiths Press, and distributed by NX Records internationally in physical form and in digital download. Our research and creative output have been the subject of reports and documentaries on BBC and Arté. Performances and presentations have taken place in music festivals such as WOMAD, Club Transmediale, and Sonar/MHD.
The core MGM research team was led by Professor Atau Tanaka, bringing to the project his vision of visceral interaction in computer music performance. Dr Baptiste Caramiaux brought expertise in machine learning of musical gesture; and students Alessandro Altavilla and Marco Donnarumma, in their doctoral research, made fundamental contributions to knowledge in sonic incidents, retroactive listening, and body/technology configurations. The team also included Drs Adam Parkinson, Miguel Ortiz, and Andrew Harding, carrying out research on participation, concerts as sites of research, muscle interfaces and statistical analysis of musical gesture.
Intellectual, creative, and public engagement formed important parts of MGM research. The MGM team hosted an international conference, New Interfaces for Musical Expression (NIME) in London in 2014, bringing together international scientific and artistic communities of practice. A series of visiting researcher/visiting artists came to Goldsmiths to engage with the MGM project, sharing knowledge and best practice in the science and art of tangible interfaces, new instrument building and live performance. Notable guests included Prof Hiroshi Ishii from the MIT Media Lab, performer Laetitia Sonami from the San Francisco Art Institute, and sound art scholars Caleb Kelly and Brandon Labelle. The MGM project included an ambitious public events programme to communicate research results to the general audiences in compelling contexts, ranging from hosting school visits, performing and exhibiting at festivals, museums, and galleries to presenting installations and concerts in public parks and pubs, including the ZKM Karlsruhe and the ICA London.
Gender representation in music technology is an important issue. Female representation across the MGM project was excellent, at 34%. We have contributed to discussions on the topic at the MusDig conference, organised a special session during the NIME2014 conference, and have been active in the Women in Sound/Women on Sound (WISWOS) network.