Periodic Reporting for period 2 - Whither Music? (Exploring Musical Possibilities via Machine Simulation)
Periodo di rendicontazione: 2023-07-01 al 2024-12-31
"Whither Music?" was the motto of Leonard Bernstein's 1973 Norton Lectures at Harvard ("The Unanswered Question"), where he analysed the musical developments that led to what he called the 20th Century Crisis of Music: the gradual decline of tonality, driven by a takeover of tonal ambiguity in the late 19th and early 20th centuries, eventually leading to complete abandonment of tonality in Schönberg's dodecaphony - a historical process that Bernstein portrays as equally inevitable and problematic.
WHITHER MUSIC? is a project that aims to establish model-based computer simulation (via methods of AI, (deep) Machine Learning and probabilistic modelling) as a viable methodology for asking questions about musical processes, developments, possibilities and alternatives - for music research, for didactic purposes, for creative music exploration scenarios. Computer simulation here means the design of predictive or generative computational models of music (of certain styles), learned from large corpora, and their purposeful and skilful application to answer, e.g. "what if" questions, make testable predictions, or generate musical material for further analysis, musicological or aesthetic. We believe that this would open new possibilities for music research, education, and for creative engagement with music, some of which will be further explored in the project.
This vision of purposeful application of computational models dictates the central methodological principles for our research: veridical modeling and simulation require stylistically faithful, tightly controllable, transparent, and explainable models. These requirements, in turn, motivate us to develop and pursue a musically informed approach to computational modeling, as an alternative to the currently prevailing trend of end-to-end learning with huge, opaque neural networks. In terms of modeling domains, we will be concerned with three types of computational models: models of music generation, of expressive performance, and of musical expectancy, mirroring the three major components in the system of music: the composer, the performer, and the listener. In addition to developing fundamental machine learning and modeling methods, we will explore concrete simulation and application scenarios for our computer models, in the form of musicological studies, creative and didactic tools and exhibits, and public educational events, in cooperation with musicologists, music educators, and institutions from the creative arts and sciences sector.
At a fundamental level, the goal of his project is thus really two-fold: beyond developing the technology for, and demonstrating, controlled musical simulation for serious purposes, we wish to develop and propagate an alternative approach to AI-based music modeling, hoping to contribute to a re-orientation of the field of Music Information Research (MIR) towards more musically informed modeling - a mission we already started in our previous ERC project Con Espressione.
WHITHER MUSIC? is a project that aims to establish model-based computer simulation (via methods of AI, (deep) Machine Learning and probabilistic modelling) as a viable methodology for asking questions about musical processes, developments, possibilities and alternatives - for music research, for didactic purposes, for creative music exploration scenarios. Computer simulation here means the design of predictive or generative computational models of music (of certain styles), learned from large corpora, and their purposeful and skilful application to answer, e.g. "what if" questions, make testable predictions, or generate musical material for further analysis, musicological or aesthetic. We believe that this would open new possibilities for music research, education, and for creative engagement with music, some of which will be further explored in the project.
This vision of purposeful application of computational models dictates the central methodological principles for our research: veridical modeling and simulation require stylistically faithful, tightly controllable, transparent, and explainable models. These requirements, in turn, motivate us to develop and pursue a musically informed approach to computational modeling, as an alternative to the currently prevailing trend of end-to-end learning with huge, opaque neural networks. In terms of modeling domains, we will be concerned with three types of computational models: models of music generation, of expressive performance, and of musical expectancy, mirroring the three major components in the system of music: the composer, the performer, and the listener. In addition to developing fundamental machine learning and modeling methods, we will explore concrete simulation and application scenarios for our computer models, in the form of musicological studies, creative and didactic tools and exhibits, and public educational events, in cooperation with musicologists, music educators, and institutions from the creative arts and sciences sector.
At a fundamental level, the goal of his project is thus really two-fold: beyond developing the technology for, and demonstrating, controlled musical simulation for serious purposes, we wish to develop and propagate an alternative approach to AI-based music modeling, hoping to contribute to a re-orientation of the field of Music Information Research (MIR) towards more musically informed modeling - a mission we already started in our previous ERC project Con Espressione.
The project has progressed along all major fronts described above. In terms of *new technologies and methods* of a general nature, we developed new methods for obtaining musically relevant and interpretable explanations from deep learning models, as well as strategies for critically testing the plausibility of such explanations; we looked into new ways of learning effective general representations for music and audio tasks from data, and combined these with complexity reduction methods; and we are have been developing, and demonstrating on a range of different musical structure recognition tasks, a coherent, generic, and perception-inspired methodology for modeling musical scores and pieces as graph structures.
In terms of musical *modeling domains*, we are addressing all three domains mentioned above: (1) music perception and prediction ("the listener"), (2) music performance and interaction ("the performer"), and (3) music generation ("the composer"). Regarding (1), we developed a general way of modeling musical pieces as graphs, which now makes it possible to address all sorts of music perception and structure recognition tasks with graph neural networks while offering a very natural structured representation for music and scores; our first published demonstrations and results concern tasks like voice separation, cadence detection, and harmonic structure analysis. Also, we took first steps towards probabilistic musical expectancy models based on differentiable short-term models. Regarding (2), our central research and demonstration object is the ACCompanion, an autonomous piano accompaniment and expressive co-performance system; a paper describing this system and analysing its core components was presented at the IJCAI 2023 conference. Regarding (3), we extended the notion of deep diffusion models to discrete representations, opening up new avenues for controllable music generation at a symbolic level. In latest work, we have demonstrated a novel method for controlling and shaping the "surprisal" profile of generated music via information-theoretic models.
The project also develops and openly distributes *research resources*, in the form of open source datasets and software tools, for the research community. That includes note-level-aligned piano performance corpora, where precise performance information is connected to score information and to higher-level structural and musicological annotations for analysis purposes; the Partitura software library for musical score and alignment handling; a generic graph processing library for many different musical tasks; and a general framework for extracting interpretable explanations from such graph neural networks.
Finally, we continually engange in dissemination activities that address both scientific audiences (such as keynotes at the IJCAI-ECAI 2022 conference) and also more general audiences (e.g. a public presentation in Vienna City Hall, broadcast live on public TV). In June 2023, we started our own video channel on YouTube, which offers a live video stream focusing on specific experiments directly from our music research lab (https://www.youtube.com/@paowcpjku(si apre in una nuova finestra)).
In terms of musical *modeling domains*, we are addressing all three domains mentioned above: (1) music perception and prediction ("the listener"), (2) music performance and interaction ("the performer"), and (3) music generation ("the composer"). Regarding (1), we developed a general way of modeling musical pieces as graphs, which now makes it possible to address all sorts of music perception and structure recognition tasks with graph neural networks while offering a very natural structured representation for music and scores; our first published demonstrations and results concern tasks like voice separation, cadence detection, and harmonic structure analysis. Also, we took first steps towards probabilistic musical expectancy models based on differentiable short-term models. Regarding (2), our central research and demonstration object is the ACCompanion, an autonomous piano accompaniment and expressive co-performance system; a paper describing this system and analysing its core components was presented at the IJCAI 2023 conference. Regarding (3), we extended the notion of deep diffusion models to discrete representations, opening up new avenues for controllable music generation at a symbolic level. In latest work, we have demonstrated a novel method for controlling and shaping the "surprisal" profile of generated music via information-theoretic models.
The project also develops and openly distributes *research resources*, in the form of open source datasets and software tools, for the research community. That includes note-level-aligned piano performance corpora, where precise performance information is connected to score information and to higher-level structural and musicological annotations for analysis purposes; the Partitura software library for musical score and alignment handling; a generic graph processing library for many different musical tasks; and a general framework for extracting interpretable explanations from such graph neural networks.
Finally, we continually engange in dissemination activities that address both scientific audiences (such as keynotes at the IJCAI-ECAI 2022 conference) and also more general audiences (e.g. a public presentation in Vienna City Hall, broadcast live on public TV). In June 2023, we started our own video channel on YouTube, which offers a live video stream focusing on specific experiments directly from our music research lab (https://www.youtube.com/@paowcpjku(si apre in una nuova finestra)).
The project has already produced a substantial number of new technical results for fields like Music Information Research (MIR) and Computational Musicology. That these go beyond the state of the art is documented by corresponding publications in some of the top scientific media of our fields, such as the IJCAI conferences, or the Transactions of the International Society for Music Information Research (TISMIR). We are confident that by continuing along these lines and taking advantage of new developments in fields like AI and machine learning, we will be able to contribute to a re-orientation of the field of Music Information Research (MIR) towards more structured and efficient modeling of music and musical processes.