Periodic Reporting for period 3 - COSMOS (COSMOS: Computational Shaping and Modeling of Musical Structures)
Periodo di rendicontazione: 2022-06-01 al 2023-11-30
Performed music is the form of music we most often hear, music that is shaped by musicians and communicated to listeners. This is the music that we experience, it is also the music that affects us. But we have few tools with which to describe the fleeting experience of music, or the work of performers who create these experiences. The project COSMOS (https://cosmos.ircam.fr|http://bit.ly/COSMOS-YouTube) is about finding ways to represent the ephemeral qualities of music, namely, its expressivity and the musical structures that result from performers manipulating this expressivity.
The scientific approach leverages developments in citizen science, data science/analytics, and performance studies to represent performers’ know-how in efficient ways that are scalable to large datasets. The research themes are: i) to build software tools to enable people to find, represent, explore, and talk about performed structures; ii) to harness volunteer thinking (citizen science) to dissect the musical structures experienced in performance and in performed music; iii) to create sandbox environments that allow users to experiment with making performed structures; iv) to develop theoretical frameworks to discover the reasoning behind musical structures perceived and made; and, v) to foster community engagement by training experts to provide feedback on structure solutions so as to increase public understanding of the creative work in music performance.
This research has broad implications for society because engaging with music is a near universal experience. Music listeners respond not only to its content but are also deeply influenced by how performers express that content. Being able to characterise the nature of musical communication not only offers new ways to explore and talk about music performance, it also serves as a bedrock for connecting the essentials of the musical experience to human response. These developments will open up new paths to explaining physiological reactions to music, with implications for therapeutic uses of music.
The scientific approach leverages developments in citizen science, data science/analytics, and performance studies to represent performers’ know-how in efficient ways that are scalable to large datasets. The research themes are: i) to build software tools to enable people to find, represent, explore, and talk about performed structures; ii) to harness volunteer thinking (citizen science) to dissect the musical structures experienced in performance and in performed music; iii) to create sandbox environments that allow users to experiment with making performed structures; iv) to develop theoretical frameworks to discover the reasoning behind musical structures perceived and made; and, v) to foster community engagement by training experts to provide feedback on structure solutions so as to increase public understanding of the creative work in music performance.
This research has broad implications for society because engaging with music is a near universal experience. Music listeners respond not only to its content but are also deeply influenced by how performers express that content. Being able to characterise the nature of musical communication not only offers new ways to explore and talk about music performance, it also serves as a bedrock for connecting the essentials of the musical experience to human response. These developments will open up new paths to explaining physiological reactions to music, with implications for therapeutic uses of music.
In order to design accurate computational models for representing and explaining expressive choices through musicians’ performed structures, we first gather information about the performed structures. For this, we will enlist the help of citizen scientists. To this end, we have built CosmoNote (https://cosmonote.ircam.fr|https://bit.ly/CosmoNote-YouTube) a web-based interface that allows for the synchronized presentation of discrete and continuous information layers, which include note and pedal data, audio features such as loudness and tempo, score features such as harmonic tension, and physiological data in a visual and audio environment. Citizen scientists can toggle between the information layers and view training modules to learn to recognize and mark performed structures such as segmentation and prominence using boundaries of varying strengths, regions, and note groupings, and insert comments. Performers may also use CosmoNote to make explicit the structures conveyed in their interpretations. CosmoNote is currently the only software interface that offers this full set of features and makes them available on the web. Substantial effort has been put into not only the design of the interface, but also the development of a modular and extensible software framework to support the automatic extraction and computation of data for CosmoNote’s information layers, with attention to efficiency and scalability. The CosmoNote web portal was launched publicly in December 2021 with Glenn Gould’s 1955 recording of the Goldberg Variations. Subsequent annotation campaigns will feature other performers and classical piano music genres.
The premise of COSMOS and the use of a reproducing piano to gather data for performance research has been explained in a short film, Le Piano Virtuose (https://youtu.be/yXkwusNyte4) disseminated on Le Monde and CNRS Le Journal. Expressive manipulations in musical communication was explored in a widely viewed ARTE.TV documentary on hearing, Entendons-nous tous pareil ? (https://youtu.be/JcxhaGdapE0). The COSMOS project was one of 15 featured in the ERC 10k Grantees Celebration – https://erc.europa.eu/how-ERC-transformed-science/#Chew
The therapeutic potential of linking performed structures with physiological response was covered in Klänge Fürs Herz (https://bit.ly/ec-Spiegel20) in Der Spiegel. Discussions started at an exploratory seminar at Harvard’s Radcliffe Institute for Advanced Study bringing together computational music information research and cardiac therapeutics led to a Scientific American Opinion article on How Music Can Literally Heal the Heart (https://bit.ly/sciam-musicheart). The potential for exploiting computational music performance research for heart health is laid out in a presentation for clinicians and patients, Music: An Underutilised Tool in Neurocardiology? (https://youtu.be/7oSdvgv3JMQ).
Applying techniques for transcribing rhythms of performance to abnormal heart rhythms produced arrhythmia music like the Little Etudes for Piano (https://youtu.be/42TRIL9MeOA) powerful inscriptions of heart rhythm anomalies. Empirical justification for notating cardiac arrhythmias using music representations is described in a European Heart Journal CardioPulse article, Putting (One’s) Heart Into Music (https://doi.org/10.1093/eurheartj/ehab108) which was among the top 10 downloaded CardioPulse articles in 2021.
The premise of COSMOS and the use of a reproducing piano to gather data for performance research has been explained in a short film, Le Piano Virtuose (https://youtu.be/yXkwusNyte4) disseminated on Le Monde and CNRS Le Journal. Expressive manipulations in musical communication was explored in a widely viewed ARTE.TV documentary on hearing, Entendons-nous tous pareil ? (https://youtu.be/JcxhaGdapE0). The COSMOS project was one of 15 featured in the ERC 10k Grantees Celebration – https://erc.europa.eu/how-ERC-transformed-science/#Chew
The therapeutic potential of linking performed structures with physiological response was covered in Klänge Fürs Herz (https://bit.ly/ec-Spiegel20) in Der Spiegel. Discussions started at an exploratory seminar at Harvard’s Radcliffe Institute for Advanced Study bringing together computational music information research and cardiac therapeutics led to a Scientific American Opinion article on How Music Can Literally Heal the Heart (https://bit.ly/sciam-musicheart). The potential for exploiting computational music performance research for heart health is laid out in a presentation for clinicians and patients, Music: An Underutilised Tool in Neurocardiology? (https://youtu.be/7oSdvgv3JMQ).
Applying techniques for transcribing rhythms of performance to abnormal heart rhythms produced arrhythmia music like the Little Etudes for Piano (https://youtu.be/42TRIL9MeOA) powerful inscriptions of heart rhythm anomalies. Empirical justification for notating cardiac arrhythmias using music representations is described in a European Heart Journal CardioPulse article, Putting (One’s) Heart Into Music (https://doi.org/10.1093/eurheartj/ehab108) which was among the top 10 downloaded CardioPulse articles in 2021.
The expected results until the end of the project include the release of numerous collections for the CosmoNote interface for citizen science engagement, with some containing structural annotations by the performers or composers for comparison against listeners’ experienced structures. Annotations from these collections will drive computational algorithms for representing and explaining performers’ musical decisions and listeners’ perceptions. An extension to the CosmoNote interface will allow for the capture of the expanded communication between performer and listener, to include not only music structural annotations but also physiological signals collected by the Heart.FM mobile app created as part of a spin-off proof-of-concept project. Public showcasing of the efficacy of musical communication through technology demonstrations with scientific visualisations in live performance are planned.