We briefly describe below the main work conducted in each objective.
Objective O1: Hybrid deep learning with differentiable models
• Music synthesis and transformation: We proposed GLA-Grad, a model which integrates a classical signal processing iterative technique into the deep learning framework. In a related effort, we introduced SpecDiff-GAN which enhances the performance of an existing GAN model by incorporating a forward diffusion process during training with a spectrally-shaped adaptive noise injection, tailored to the audio signal knowledge. In music transformation, we introduced WaveTransfer, a flexible end-to-end diffusion model that trains a single model to handle multiple musical timbre transfer pairs.
• Dereverberation under diverse supervision paradigms: We developed a deep-learning-based dereverberation method using a novel physical coherence loss which encourages the room impulse response (RIR) induced by the dereverberated output of the model to match the acoustic properties of the room in which the signal was recorded. We further demonstrated the ability of such hybrid models to be trained in an unsupervised manner using minimal acoustic information and reverberant speech.
• Model-based deep learning approaches for source separation: We proposed a novel unsupervised model-based deep learning approach to musical source separation. Each source is modelled with a vocal production model, here implemented as a differentiable parametric source-filter model. The work has been further extended into a fully differentiable model by integrating a multipitch estimator and a novel differentiable assignment module within the core model.
Objective O2 : Attention-based and model-driven neural architectures :
• Structure-informed Positional Encoding (PE) for Music Generation: We proposed a PE technique to incorporate human knowledge into attention mechanisms in order to provide the model with insights about the input signals through the multi-resolution structure of music.
• Representation learning: We proposed a weakly-supervised framework for cross-modality representation learning in the context of a newly proposed task: melody-lyrics matching (MLM). It integrates neural network encoders with a differentiable alignment algorithm in a contrastive setting which learns the representations of melody and lyrics and provide their alignment simultaneously. Specific work was also dedicated to discrete representation learning. First, we investigated a radically new approach for quantization by exploiting a sequence of random dictionaries which allow the model to use larger quantization dictionaries with unchanged complexity. Second, we proposed several strategies to obtain disentangled representations for instance in frequency bands (for bandwidth extension) or for each source in a mix-recording (for joint audio coding and source separation). We also introduced a novel method to quantify the sensitivity of pre-trained audio embeddings to common effects .
Objective O3: Statistical Reverberation and musicological models:
• A novel representation for lyrics: For MLM, we need representations that can capture the inherent relationships between melody and lyrics around conceptual pairs such as rhythm and rhyme, note duration and syllabic stress, and structure correspondence. To that aim, we introduced sylphone, a novel representation for lyrics at syllable-level activated by phoneme identity and vowel stress.
• Reverberation models: We refined several reverberation models to express them efficiently in a differentiable manner, in order to embed them in hybrid neural architectures. For instance, we embedded Polack’s reverberation model in a differentiable hybrid architecture, and expressed backpropagation from an RIR synthesized from Polack’s model to the model’s parameters.
Objective O4: Distributed music crowdsourcing database:
The main goal of this objective is to build the core software tool to gather a large, varied, multi-genre, multi-track, multi-instruments annotated music database suitable for MIR applications. The developed HI-AUDIO online platform relies on a distributed and iterative music recording paradigm to asynchronously record musicians localised at different remote individual sites. The system architecture is based on the client-server paradigm. Specific care was dedicated to data handling and usage, specifically regarding ethics, privacy, and regulations such as GDPR, as well as intellectual property and copyright. The source code will be publicly available at:
https://github.com/hi-paris/hiaudio(opens in new window) and the application is published at:
https://hiaudio.fr(opens in new window). The platform has been demonstrated publicly.
