Periodic Reporting for period 2 - INTENT (Intelligent Instruments: Understanding 21st-Century AI Through Creative Music Technologies)
Reporting period: 2023-03-01 to 2024-03-31
Creative artificial intelligence is emerging as a powerful force in our culture, but its consequences are poorly understood and societal infrastructure lags behind. A public understanding of AI is needed. The Intelligent Instruments project signifies a pioneering leap in research about AI through the systematic study of how creative AI is transforming our culture. We take a novel approach to the study of creative AI, where the impact of intelligent technologies on culture is investigated for the first time, through interdisciplinary practice-based research. Using music technologies for this research is ideal as they represent some of the most sophisticated technologies in all periods of human history, imposing a heightened design challenge compared with common industry standards, but critically because of people’s affection with music and its instruments.
The Intelligent Instruments project seeks to foster a public understanding of AI. The pressing question in European AI research is: how do we build public trust in AI technologies? But trust builds on experience, familiarity and understanding. It is urgent to develop a new public discourse on creative AI but this cannot originate in isolated marketing departments of commercial software houses or individual university labs: it has to emerge from people’s actual experience! Through a collaborative methodology involving a range of other scientific fields we develop a new humanities-based theory of creative AI, build the foundations of a shared discourse and ethos around human-centred AI design, and define principles of ethical design of services and products.
In order to achieve this goal, the objectives of the project are to create musical instruments imbued with new artificial intelligence and explore its effect on musicians, audiences and the general public. We do this through various methods, for example participatory design with composers and musicians, workshops with instrument makers and musicians, user studies of new instruments, concerts and participation in public events, and public presentations. We have built a technical infrastructure for the experimental humanities that enables interdisciplinary scientific collaborations and in this area we collaborate with visiting artists and scientists to further our project.
Our aim is to generate a discourse around AI, promoting new understanding and embodied experience of AI amongst the general public. By so doing, we are able to discern and analyse the emerging discourse around creative AI, embodied AI and AI as part of our cultural sphere.
The Intelligent Instruments project seeks to foster a public understanding of AI. The pressing question in European AI research is: how do we build public trust in AI technologies? But trust builds on experience, familiarity and understanding. It is urgent to develop a new public discourse on creative AI but this cannot originate in isolated marketing departments of commercial software houses or individual university labs: it has to emerge from people’s actual experience! Through a collaborative methodology involving a range of other scientific fields we develop a new humanities-based theory of creative AI, build the foundations of a shared discourse and ethos around human-centred AI design, and define principles of ethical design of services and products.
In order to achieve this goal, the objectives of the project are to create musical instruments imbued with new artificial intelligence and explore its effect on musicians, audiences and the general public. We do this through various methods, for example participatory design with composers and musicians, workshops with instrument makers and musicians, user studies of new instruments, concerts and participation in public events, and public presentations. We have built a technical infrastructure for the experimental humanities that enables interdisciplinary scientific collaborations and in this area we collaborate with visiting artists and scientists to further our project.
Our aim is to generate a discourse around AI, promoting new understanding and embodied experience of AI amongst the general public. By so doing, we are able to discern and analyse the emerging discourse around creative AI, embodied AI and AI as part of our cultural sphere.
The Intelligent Instruments project has been operating for 30 months. Our team consisting of PI, project manager, technician, postdocs, and PhD researchers, together with visiting artists and scientists, have produced various types of practical and scientific work:
18 research projects – these are typically instruments that embody various types of AI as part of their design. These projects are used as boundary objects to test the ideas we are working with. The projects are the results of actual practice-based research where each inventor uses the projects for their own creative art and scientific exploration.
21 peer reviewed and published conference papers – We have participated in many of the key conferences in our field and have been successful in having our work accepted.
1 book – The PI co-wrote a book published as open access with MIT Press.
1 edited journal special issue – The PI co-edited a special issue for the Organised Sound journal.
7 chaired conferences and symposia – We organised the International Conference on Artificial Intelligence and Music Creativity at the University of Sussex. We have also organised various symposia with our work.
18 workshops – Workshops have been an ideal platform to explore the reception of our technologies. We have run workshops internationally and in Iceland.
23 presentations – Presentations of work at conferences or festivals.
19 appearances in public media – The research we conduct is particularly timely in the time of new AI and we have been active in discussing our work in the media.
34 performances – Members of the lab have performed with their instruments on the musical stage internationally.
4 installations – Some of our technologies have been used in sound installations, for example one where artificial life was controlling a self-playing piano.
10 open source software projects – All our projects are open source and available to download.
1 database of technical elements – Our Organium library of technical elements is represented online through a database. It enables people to build their own organium.
80 open lab sessions – Here we invite people into our lab and engage in project presentations, experimentation, exploration of ideas and general discussions.
In summary, we have made some key technologies that we use for performance, in user studies and run workshops around. We reflect critically on these occasions and we publish our findings at key conferences. The members of the lab perform regularly with their instruments and share their science, art and thoughts as much as possible on social media, in our open labs and in the public.
18 research projects – these are typically instruments that embody various types of AI as part of their design. These projects are used as boundary objects to test the ideas we are working with. The projects are the results of actual practice-based research where each inventor uses the projects for their own creative art and scientific exploration.
21 peer reviewed and published conference papers – We have participated in many of the key conferences in our field and have been successful in having our work accepted.
1 book – The PI co-wrote a book published as open access with MIT Press.
1 edited journal special issue – The PI co-edited a special issue for the Organised Sound journal.
7 chaired conferences and symposia – We organised the International Conference on Artificial Intelligence and Music Creativity at the University of Sussex. We have also organised various symposia with our work.
18 workshops – Workshops have been an ideal platform to explore the reception of our technologies. We have run workshops internationally and in Iceland.
23 presentations – Presentations of work at conferences or festivals.
19 appearances in public media – The research we conduct is particularly timely in the time of new AI and we have been active in discussing our work in the media.
34 performances – Members of the lab have performed with their instruments on the musical stage internationally.
4 installations – Some of our technologies have been used in sound installations, for example one where artificial life was controlling a self-playing piano.
10 open source software projects – All our projects are open source and available to download.
1 database of technical elements – Our Organium library of technical elements is represented online through a database. It enables people to build their own organium.
80 open lab sessions – Here we invite people into our lab and engage in project presentations, experimentation, exploration of ideas and general discussions.
In summary, we have made some key technologies that we use for performance, in user studies and run workshops around. We reflect critically on these occasions and we publish our findings at key conferences. The members of the lab perform regularly with their instruments and share their science, art and thoughts as much as possible on social media, in our open labs and in the public.
Recent advents in artificial intelligence, in particular with the public use of LLM (Large Language Models) for text and GANs (Generative Adversarial Networks) for image and sound generation have made AI very prominent in the public discourse of late. Our project takes a different approach in that we develop technologies that enable people to train their own machine learning networks and develop their own instruments and musical practices with those. We are providing people with experience of working with AI as part of physical musical instruments, thus grasping AI between their hands and not through textual prompting or other such digital interfaces. This direct relationship with AI through embodied interaction helps us to understand how people experience and build mental models of AI in their lives. We analyse the discourse around these new emergent technologies.
The second cycle of the project will produce:
- Published technologies that people will apply in their artistic and scientific work: Living Looper, Notochord, Stacco, Thales, Dark Sonification, Tölvera, halldorophone, Threnoscope.
- A complete Python software stack that will integrate many of our technologies with other music software and audio programming languages.
- A set of key theories that engage with the topic of artificial intelligence in creativity and embodied interfaces. We seek to contribute to new understandings of intelligence, agency and intent in non-human systems such as intelligent instruments.
- Journal articles by the PI, the postdocs and doctoral researchers.
- Book publication by the PI on the Instruments of the Mind.
- An established Centre for the Experimental Humanities.
The second cycle of the project will produce:
- Published technologies that people will apply in their artistic and scientific work: Living Looper, Notochord, Stacco, Thales, Dark Sonification, Tölvera, halldorophone, Threnoscope.
- A complete Python software stack that will integrate many of our technologies with other music software and audio programming languages.
- A set of key theories that engage with the topic of artificial intelligence in creativity and embodied interfaces. We seek to contribute to new understandings of intelligence, agency and intent in non-human systems such as intelligent instruments.
- Journal articles by the PI, the postdocs and doctoral researchers.
- Book publication by the PI on the Instruments of the Mind.
- An established Centre for the Experimental Humanities.