Periodic Reporting for period 2 - MORPH (From Timbre Perception to the Creative Exploration of Musical Instrument Sound Morphing)
Berichtszeitraum: 2022-10-01 bis 2023-09-30
The MORPH project proposes to use sound morphing to investigate musical instrument timbre. In music, timbre is traditionally associated with the musical instrument producing the sound. Thus, it is common to refer to the timbre of the violin, for example. However, musicians talk about timbral variations between different makers or even different pitch registers of the same instrument. In practice, not all violins sound the same due to many factors, ranging from the shape of the instrument to the materials used. For instance, the strings are known to influence the sound quality of the violin, and different materials will produce perceptually different results, such as a “warm” or “dark” sound quality.
Research into timbre perception investigates the ways in which sounds are perceived to differ. The term timbre covers many perceptual parameters that are not accounted for by pitch, loudness, spatial position, and duration. Due to its complexity, timbre is often considered as one of the “last frontiers” in auditory science. Today, we understand timbre from two distinct viewpoints, a categorical contributor to source identification and a sensory quality. Timbre is the primary perceptual vehicle for the recognition of a sound source, and thus involves the absolute categorization of a sound. Categorical timbre is what allows us to recognize that a violin is playing a melody instead of a piano, for example. However, variations such as warmer or darker violin sounds are associated with the sensory quality of timbre perception.
Recent advances in timbre research have investigated the relationship between the sensory and categorical facets of timbre perception. The categorical view assumes that there are “timbral gaps” between musical instruments, whereas the sensory view posits that timbre is potentially continuous in nature and the timbral gaps result from the physical limitations of the sound producing objects. The MORPH project uses sound morphing to create intermediate sounds between two different instruments and then investigate how the morphing transformation affects the timbre of these sounds. Musical instrument sound morphing theoretically allows creating continuous timbre spaces by filling the gaps between traditional musical instruments and therefore breaking the categorical perception of musical instrument timbre by the auditory illusion of hybrid musical instruments.
There are three overall research objectives (RO). RO1 aims to develop a high-quality and accurate model of musical instrument sounds to enable precise and controlled manipulations of acoustic features. The model should be perceptually transparent for orchestral instruments, and the manipulations should be perceptually natural. RO2 aims to investigate timbre perception using the models from RO1. The morphs create hybrid musical instrument sounds that potentially break the absolute categorization of musical instrument perception. RO2 requires perceptual evaluation to validate if morphing connects the musical instruments, essentially creating continuous timbre spaces. Finally, RO3 aims to assist musicians and composers in gaining creative control over musical instrument sound morphing.
The MORPH project has resulted in powerful tools to advance our fundamental understanding of timbre and potentially led to groundbreaking discoveries in timbre perception, as well as impact music technology by transferring the methodology to digital media used in creative musical applications. The MORPH project has advanced music audio processing by developing new modeling techniques specifically tailored for musical instrument sounds. Additionally, the MORPH project has provided powerful analysis tools to further our understanding of musical timbre perception.
Research into timbre perception investigates the ways in which sounds are perceived to differ. The term timbre covers many perceptual parameters that are not accounted for by pitch, loudness, spatial position, and duration. Due to its complexity, timbre is often considered as one of the “last frontiers” in auditory science. Today, we understand timbre from two distinct viewpoints, a categorical contributor to source identification and a sensory quality. Timbre is the primary perceptual vehicle for the recognition of a sound source, and thus involves the absolute categorization of a sound. Categorical timbre is what allows us to recognize that a violin is playing a melody instead of a piano, for example. However, variations such as warmer or darker violin sounds are associated with the sensory quality of timbre perception.
Recent advances in timbre research have investigated the relationship between the sensory and categorical facets of timbre perception. The categorical view assumes that there are “timbral gaps” between musical instruments, whereas the sensory view posits that timbre is potentially continuous in nature and the timbral gaps result from the physical limitations of the sound producing objects. The MORPH project uses sound morphing to create intermediate sounds between two different instruments and then investigate how the morphing transformation affects the timbre of these sounds. Musical instrument sound morphing theoretically allows creating continuous timbre spaces by filling the gaps between traditional musical instruments and therefore breaking the categorical perception of musical instrument timbre by the auditory illusion of hybrid musical instruments.
There are three overall research objectives (RO). RO1 aims to develop a high-quality and accurate model of musical instrument sounds to enable precise and controlled manipulations of acoustic features. The model should be perceptually transparent for orchestral instruments, and the manipulations should be perceptually natural. RO2 aims to investigate timbre perception using the models from RO1. The morphs create hybrid musical instrument sounds that potentially break the absolute categorization of musical instrument perception. RO2 requires perceptual evaluation to validate if morphing connects the musical instruments, essentially creating continuous timbre spaces. Finally, RO3 aims to assist musicians and composers in gaining creative control over musical instrument sound morphing.
The MORPH project has resulted in powerful tools to advance our fundamental understanding of timbre and potentially led to groundbreaking discoveries in timbre perception, as well as impact music technology by transferring the methodology to digital media used in creative musical applications. The MORPH project has advanced music audio processing by developing new modeling techniques specifically tailored for musical instrument sounds. Additionally, the MORPH project has provided powerful analysis tools to further our understanding of musical timbre perception.
The work performed for WP1 focused on modeling and manipulation. We implemented a baseline sinusoidal model followed by improvements. The main results are a high-quality sinusoidal model that renders a perceptually transparent representation of orchestral musical instrument sounds. Additionally, we implemented manipulations of the sinusoidal model and then improved its transformation strategies. The main results are the ability to independently manipulate the oscillatory modes of musical instrument sounds to obtain perceptually natural transformations. We also implemented a baseline source-filter model and made improvements. The main results are a high-quality representation capable of perceptually natural transformations by independently manipulating the excitation (source) and the resonant cavity (filter). Finally, we implemented the baseline distribution derivative method analysis and are currently improving the phase model. The main results are the ability to represent time-varying features of musical instrument sounds with a better representation of vibrato and tremolo.
The work performed for WP2 focused on the timbre perception of morphing. We did a listening test to investigate the perceptual impact of morphing musical instrument sounds. We selected the sound stimuli for the listening test, created the morphs, ran a pilot to validate the experimental protocol, collected pairwise dissimilarity data, used multidimensional scaling (MDS) of dissimilarities to build a timbre space with the morphs, did a geometric analysis of the placement of the morphs relative to the original sounds, and used statistical analysis on the dissimilarity data to test the perceptual linearity and intermediateness of the morphs. The main results are the geometric and perceptual validation of the linearity and of the intemediateness of morphing.
The work originally planned for WP3 was highly impacted by the COVID19 pandemic. The fellow and the supervisors devised a mitigation strategy that consisted in fully developing the technical work originally planned for WP1 and WP2 at the expense of not working on WP3. Consequently, the technical activities originally scheduled for months 25 to 36 (WP3) were not fully developed.
The work performed for WP2 focused on the timbre perception of morphing. We did a listening test to investigate the perceptual impact of morphing musical instrument sounds. We selected the sound stimuli for the listening test, created the morphs, ran a pilot to validate the experimental protocol, collected pairwise dissimilarity data, used multidimensional scaling (MDS) of dissimilarities to build a timbre space with the morphs, did a geometric analysis of the placement of the morphs relative to the original sounds, and used statistical analysis on the dissimilarity data to test the perceptual linearity and intermediateness of the morphs. The main results are the geometric and perceptual validation of the linearity and of the intemediateness of morphing.
The work originally planned for WP3 was highly impacted by the COVID19 pandemic. The fellow and the supervisors devised a mitigation strategy that consisted in fully developing the technical work originally planned for WP1 and WP2 at the expense of not working on WP3. Consequently, the technical activities originally scheduled for months 25 to 36 (WP3) were not fully developed.
The progress beyond the state of the art in WP1 is the accurate estimation of parameters of sinusoids resulting in a high-quality representation of the oscillatory modes of musical instruments, and the sound analysis technique using the distribution derivative method with the improved phase model, which has potential to push the state of the art for vibrato/tremolo modeling. The progress beyond the state of the art in WP2 is related to the experimental protocol. Specifically, the experiment has potential to become a proof of concept for the investigation of perceptual aspects of musical instrument timbre. Additionally, we have shown that a relatively simple morphing strategy is capable of generating perceptually linear and intermediate morphs. The results indicate that sound morphing is capable of potentially breaking the categorical perception of musical instrument timbre and generate continuous timbre spaces. The work originally planned for WP3 was highly impacted by the COVID19 pandemic. The expected results included real-time morphing transformations, control of the morphs via a touch user interface (TUI), and integration of real-time morphing transformations controlled by the TUI into the prototype for the VHMI.