New Frontiers in Music Information Processing

Project Information

MIP-Frontiers

Grant agreement ID: 765068

DOI

10.3030/765068

Project closed

EC signature date 17 August 2017

Start date 1 April 2018

End date 31 March 2022

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 3 926 675,88

EU contribution

€ 3 926 675,88

Coordinated by

QUEEN MARY UNIVERSITY OF LONDON
United Kingdom

Periodic Reporting for period 2 - MIP-Frontiers (New Frontiers in Music Information Processing)

Reporting period: 2020-04-01 to 2022-03-31

Summary of the context and overall objectives of the project

New Frontiers in Music Information Processing (MIP-Frontiers) was an EU Horizon 2020 funded project, a Marie Skłodowska-Curie Action (MSCA) European Training Network (ETN). from April 2018 to March 2022.
Music Information Processing (MIP), also known as Music Information Retrieval (MIR) involves the use of information processing methodologies to understand and model music, and to develop products and services for music and music-related industries.

The project brought together 4 academic and 3 non-academic Beneficiaries and 9 Partner Organisations with the aim to trained 15 early career stage researchers (ESRs) in a range of university-based and industry-based PhDs.
The work was structured along three research frontiers identified as requiring intensive attention and integration (data-driven, knowledge-driven, and user-driven approaches). Scientifically speaking, it addressed three main weaknesses in MIR:
• not being robust to differences in musical style
• not generalizing across different use contexts
• not being scalable to industrial scale datasets.

The 15 researchers were trained to address MIR skills needs and gaps. The fellows have learned to think entrepreneurially and to exploit their research in new ways that will benefit European industry and society leading MIR transformation in academia and industry. And will contribute to Europe's leading role in this field.
The ESRs have also been exposed to industry and academic environments in their daily work and via secondments and seminars led by industry and academic partners. The training have given them the technical and research skills they needed, and a range of transferable skills training to enhance the ESRs’ employability whether in industry or academia.

The 15 ESRs' PhD topics reflect the importance that knowledge, data and users-driven methodologies have in the field and provide a good example of the diverse approaches that are currently being proposed. However, in the case of most music related problems, they have also shown that by themselves these methodologies cannot successfully solve most problems. It is clear that an interdisciplinary approach has to be taken to tackle music problems.
Also, the fellows research has showed the need for more informed, deliberately structured, knowledge-directed approaches to solve the big open problems in music and sound modelling. And that systems need to be built for and with users, the user-driven approach has to be incorporated and considered in all projects.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The aim of MIP-Frontiers was to train 15 Researchers to the level of a PhD in Music Information Processing. These PhD projects were designed to address MIR needs and make advances beyond the current state of the art. All 15 fellows enrolled for a doctorate (PhD) at one of the 4 academic partners.

In the first half of the project, two training weeks and a collaborative sandbox event were held. All training events focused on technical skills needed for them to start and grow as researchers, from communication to software carpentry, deep learning to music theory and specific MIR knowledge.
The trainings in the second half of the project focused mainly on transferable skills that the fellows will need for their careers after the PhD and complemented their research skills. To take advantage of the online conferences all fellows attended in MIR field conferences: ICASSP 2020 and 2021 and ISMIR 2020.
The 3rd Training Week was held online as a series of short training sessions. The seminars are available on the project website, and range from presenting your research, communication and impact to commercialisation and IP rights and, industry vs academic careers.
Industry and academic careers demand different skills and working habits. All fellows had a cross-sectoral secondment to an organisation different to their host: those hosted at an academic entity had an industry partner as a secondment host, and vice versa. It was advantageous for the fellows to experience work in a different sector and have a research focus on real industry problems.

The communication and dissemination activities focused on developing and promoting a video to explain the MIR field to undergraduates. While we all listen to music in online formats or use music apps, not many students, whether at high school or undergraduate, know the field, its research and its impact on society. Our video introduces the field to newcomers.
Furthermore, the ESRs have described in blog posts their experience and view of the MIP research. It shows that a research career can be a rewarding way to link technical ability with a passion for music.

The scientific communication activities included two scientific workshops organised by MIP-Frontiers. The fellows’ presentations were recorded and are available the project website. Individually the ESRs have already published more than 70 open access publications available from the project's web page, and show their progress and research beyond the state of the art.

The research work done by the 15 ESRs, their scientific results and progress beyond the state of the art is reflected in their open access publications and the progress beyond the state of the art.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

15 ESRs were trained in the field of MIR and their research address the main challenges that face the field. These 3 challenges were identified as relating to data-driven, knowledge driven and user-driven aspects.
Each aspect has been addressed on an individual basis, by each researcher via their research project. All 15 ESRs' PhD topics reflected the importance that data, knowledge, and user driven methodologies have in the field and are a good example of the diverse approaches that are currently being proposed.

Many of the PhD research projects involved the development and use of large datasets, the development and use of deep-learning methodologies, and they all faced the challenge that music-specific data and problems bring to data-driven methodologies. Each ESR had to find a particular solution to address this and the individual research plan of each ESR has had to adapt to these developments.

The fellows’ contributions have demonstrated the variety of ways in which careful consideration of domain knowledge can help in devising new methods, representations, and models in the field of MIR, for a wide range of tasks. Knowledge-related questions are as important in MIR as they were years ago, when MIP-Frontiers proposal was written.

Clear advances have also been made in user-driven research. A fundamental requirement for modern Artificial Intelligence is that it is built for and with users, and MIP-Frontiers has also highlighted the necessity to further involve users in all possible states of algorithm development (design, interaction, evaluation,). This aspect is particularly important in this field since music is originally created and performed for humans.

All publications are open access and available in repositories and on the project website. The project has also included the publication of 15 open datasets.

At an industry level, the outcomes of one of the projects have already been licensed and will be available from DoReMIR in a new product from May 2022; in addition, Another ESR's work will be incorporated in a new product from Sony CSL. And the collaboration with Jamendo led to a new dataset that was used at 3 workshops (MediaEval).

MIP-Frontiers, Mid term meeting. BCN May 2019

MIP-Frontiers group photo at ISMIR 2018 training

MIP-Frontiers Sandbox event at QMUL London 2020

MIP-Frontiers ESRs at ISMIR 2019 Delft