Periodic Reporting for period 1 - EAR Stretch (Enhancing aesthetic reception of contemporary music through active tempo control with physiological monitoring)
Période du rapport: 2021-08-15 au 2023-08-14
The EAR Stretch Action was about enhancing aesthetic reception of Contemporary Classical Music (CCM) using active tempo control. This action was proposed because CCM is often disliked by the general public, yet it is the lifeblood of current classical music development and contributes to the creation and evolution of culture which enlivens society.
EAR Stretch aimed to enhance the reception of CCM by adding embodied interactions to newly composed music to simultaneously boost and prolong engagement with the music and to guide the formation of temporal predictions while listening to the music. The embodied interaction explored in the action was direct tapping on a bespoke conductor interface.
Several objectives were enumerated to help determine whether embodied interactions could improve appreciation for Contemporary Classical Music. The specific objectives for the action were [O1] to alter aesthetic evaluations of complex CCM, [O2] establish baseline physiological responses to CCM, [O3] optimize experience through interactive tapping, and [O4] characterize the changes resulting from tapping.
Through composition, interface design, and experimentation on CCM, the objectives of the action were achieved.
The Fellow found that the question of engagement was highly relevant to CCM composers. Indeed, non-specialized listeners consistently reported disliking CCM samples compared to non-CCM samples. The empirical approaches used by EAR Stretch to examine physiological responses to CCM revealed that listener experiences are highly varied for CCM. Creating an intervention with tapping showed promise in the lab by altering appraisals of the music. Since CCM engagement is positively influenced by embodiment, composers of new CCM works are encouraged to consider and incorporate embodied elements into new compositions to engage the human physiological system and improve the reception of CCM.
EAR Stretch aimed to enhance the reception of CCM by adding embodied interactions to newly composed music to simultaneously boost and prolong engagement with the music and to guide the formation of temporal predictions while listening to the music. The embodied interaction explored in the action was direct tapping on a bespoke conductor interface.
Several objectives were enumerated to help determine whether embodied interactions could improve appreciation for Contemporary Classical Music. The specific objectives for the action were [O1] to alter aesthetic evaluations of complex CCM, [O2] establish baseline physiological responses to CCM, [O3] optimize experience through interactive tapping, and [O4] characterize the changes resulting from tapping.
Through composition, interface design, and experimentation on CCM, the objectives of the action were achieved.
The Fellow found that the question of engagement was highly relevant to CCM composers. Indeed, non-specialized listeners consistently reported disliking CCM samples compared to non-CCM samples. The empirical approaches used by EAR Stretch to examine physiological responses to CCM revealed that listener experiences are highly varied for CCM. Creating an intervention with tapping showed promise in the lab by altering appraisals of the music. Since CCM engagement is positively influenced by embodiment, composers of new CCM works are encouraged to consider and incorporate embodied elements into new compositions to engage the human physiological system and improve the reception of CCM.
The fellowship period consisted of Design, Music Creation, Interface Creation, Experimentation, and Dissemination phases.
Design
First, the Fellow completed preliminary tasks required prior to undertaking human subjects research. The tasks included acquiring mobile sensors for measuring the electrical activities of the brain and heart, designing experiments, and obtaining ethics approvals for the experiments.
Music Creation
The Fellow composed two original works. The first, SALO for solo piano, was based on a novel by Kurt Vonnegut. The pitches were largely determined algorithmically from the text in two different styles – perpetual motion and improvisation. This work was performed and recorded on a Disklavier. The recording was subsequently used in experimentation.
An additional work called Follow Me was composed by the Fellow for two percussionists. Follow Me exemplifies a composition using embodied principles – it is performed by overt and synchronized physical actions that go beyond what is necessary to play the sounds indicated in the score. Those actions engage the audience and drive movement- and timing-based predictions.
Both SALO and Follow Me were performed live in public concerts in Italy.
Interface Creation
A tappable MIDI interface was created for embodied, interactive control of CCM. For every tap, the playback of the music adjusted to the tap rate in real time. The tapper was therefore like a conductor controlling the nuances of music as it is being played.
The tapping interface was designed to work with recordings in MIDI format, such as those produced by a Disklavier. The interface controlled music playback, recorded inter-tap interval for offline analysis of tapping patterns and music-tap relationships, and streamed data to a global clock for synchronization.
Experimentation
Experiments were carried out at multiple points throughout the creation-research phases described above. The topics under study were [1] appreciation of CCM, [2] physiological responses to CCM, [3] the impact of tapping on temporal processing, and [4] music appreciation after interactive or non-interactive tapping.
Results
[1] The data showed that CCM was found to be significantly more disliked than non-CCM by non-specialized listeners. Even CCM-derived clips 100ms long were reportedly less liked compared to equivalent non-CCM-derived clips.
[2] Brain and heart data from CCM experts (composers and performers) successfully modelled CCM and non-CCM. But non-experts had inconsistent responses to CCM and poor modelling results. Only non-CCM could be modelled from the brain and heart data from non-experts.
[3] Tapping was found to neither improve nor impair temporal processing for CCM-like sound examples. Tapping to tonal sound examples did however impair temporal processing; tapping seemed to facilitate the formation of a groove which diminished temporal error detection.
[4] Finally, interactive tapping with real-time control tended to improve the tappers liking ratings given to CCM compared to non-interactive tapping. Nevertheless, non-CCM remained preferred overall.
Dissemination
The composed music, interface, and experimental results described above were disseminated in a number of formats and venues. The Fellow organized one public concert, performed in 5 additional public concerts, gave 1 public research demo, 2 internal research presentations, 3 external research presentations, 6 international research presentations or demos, ran 1 workshop, and gave an invited talk about DEI in music technology.
Design
First, the Fellow completed preliminary tasks required prior to undertaking human subjects research. The tasks included acquiring mobile sensors for measuring the electrical activities of the brain and heart, designing experiments, and obtaining ethics approvals for the experiments.
Music Creation
The Fellow composed two original works. The first, SALO for solo piano, was based on a novel by Kurt Vonnegut. The pitches were largely determined algorithmically from the text in two different styles – perpetual motion and improvisation. This work was performed and recorded on a Disklavier. The recording was subsequently used in experimentation.
An additional work called Follow Me was composed by the Fellow for two percussionists. Follow Me exemplifies a composition using embodied principles – it is performed by overt and synchronized physical actions that go beyond what is necessary to play the sounds indicated in the score. Those actions engage the audience and drive movement- and timing-based predictions.
Both SALO and Follow Me were performed live in public concerts in Italy.
Interface Creation
A tappable MIDI interface was created for embodied, interactive control of CCM. For every tap, the playback of the music adjusted to the tap rate in real time. The tapper was therefore like a conductor controlling the nuances of music as it is being played.
The tapping interface was designed to work with recordings in MIDI format, such as those produced by a Disklavier. The interface controlled music playback, recorded inter-tap interval for offline analysis of tapping patterns and music-tap relationships, and streamed data to a global clock for synchronization.
Experimentation
Experiments were carried out at multiple points throughout the creation-research phases described above. The topics under study were [1] appreciation of CCM, [2] physiological responses to CCM, [3] the impact of tapping on temporal processing, and [4] music appreciation after interactive or non-interactive tapping.
Results
[1] The data showed that CCM was found to be significantly more disliked than non-CCM by non-specialized listeners. Even CCM-derived clips 100ms long were reportedly less liked compared to equivalent non-CCM-derived clips.
[2] Brain and heart data from CCM experts (composers and performers) successfully modelled CCM and non-CCM. But non-experts had inconsistent responses to CCM and poor modelling results. Only non-CCM could be modelled from the brain and heart data from non-experts.
[3] Tapping was found to neither improve nor impair temporal processing for CCM-like sound examples. Tapping to tonal sound examples did however impair temporal processing; tapping seemed to facilitate the formation of a groove which diminished temporal error detection.
[4] Finally, interactive tapping with real-time control tended to improve the tappers liking ratings given to CCM compared to non-interactive tapping. Nevertheless, non-CCM remained preferred overall.
Dissemination
The composed music, interface, and experimental results described above were disseminated in a number of formats and venues. The Fellow organized one public concert, performed in 5 additional public concerts, gave 1 public research demo, 2 internal research presentations, 3 external research presentations, 6 international research presentations or demos, ran 1 workshop, and gave an invited talk about DEI in music technology.
In the scientific domain, this project has made multiple unique methodological contributions. Specifically, a data modeling method using heart data was developed and successfully used to show that heart data can add non-redundant information to EEG recordings. Previously this method was applied only to brain data, but EAR Stretch expanded the method by incorporating heart data. Importantly, this methodological technique can be used in simple lab setups that do not require expensive EEG equipment. Another methodological advancement made by the project was in stimulus presentation. The Fellow presented musical stimuli through a Disklavier, uniquely turning the lab into a more ecologically valid musical setting. Using real instruments is normally not feasible in a lab because instrumentalists are required to be present and those instrumentalists cannot necessarily play stimuli repeatedly in an identical fashion. The Disklavier circumvented these issues because it replicated piano performances on a real piano in a repeatable way with no performer.
The scientific results themselves contribute to the domains of music cognition, music perception, temporal processing, and embodied interaction design as described in the results. To the best of the Fellow’s knowledge, systematic behavioral testing of interactive musical interfaces for tapping has not been done before.
The New European Agenda for Culture and Music Moves Europe Action have recognized music and music experiences as important pillars of culture capable of positively shaping society and contributing to social cohesion. In support of the policies, EAR Stretch was able to create and present new live musical experiences and demos, and it has made more CCM available online for engaging the public.
The scientific results themselves contribute to the domains of music cognition, music perception, temporal processing, and embodied interaction design as described in the results. To the best of the Fellow’s knowledge, systematic behavioral testing of interactive musical interfaces for tapping has not been done before.
The New European Agenda for Culture and Music Moves Europe Action have recognized music and music experiences as important pillars of culture capable of positively shaping society and contributing to social cohesion. In support of the policies, EAR Stretch was able to create and present new live musical experiences and demos, and it has made more CCM available online for engaging the public.