Periodic Reporting for period 1 - IoMUT (The Internet of Musical Things - An ecosystem of interoperable devices connecting performers and audiences)
Periodo di rendicontazione: 2017-05-01 al 2019-04-30
This project dealt with the design, development, and evaluation of novel interfaces for technologically-mediated interaction between performers and between performers audience members in order to achieve novel forms of musical expression and foster audience engagement in live performing arts. Current systems for networked music performance (that allow performers to collaboratively create music over a network) and for participatory live music performance (that enable audiences to actively engage in the music creation process) present several limitations in terms of interfaces for audience-performer interaction and interoperability. To progress the state-of-the-art, the approach followed in this project has been to extend the concepts of the Internet of Things to the musical domain. Specifically, this project investigated the foundations of a novel research area, the Internet of Musical Things (IoMusT). The IoMusT has been defined as the network of physical/virtual objects (Musical Things) dedicated to the production and consumption of multimodal musical content. This emerging research field is positioned at the intersection of different lines of existing research, including the Internet of Things, networked music performances, artificial intelligence, and technology-mediated audience participation.
The project was structured around two interdependent objectives. The first objective consisted in investigating and developing new fundamental methods for co-located and remote interconnection of performers and audiences. The second objective concerned the exploration and empirical studies of new networked and participatory musical forms exploiting the developed methods.
The results obtained during the project demonstrate the achievement of such objectives: various audience-performer musical interfaces were created; methods to interconnect different Musical Things were devised; ecosystems were formed around the developed technologies, which led to novel forms of musical interactions between performers and between performers and audience members.
The adopted approach aspired to effect a step-change in the design of interfaces for musical expression, which has the potential to result in a high economic impact on the whole music industry. The project also contributed to the understanding of the mechanisms underlying music perception and music performance in interactive and multimodal contexts.
The project was structured around two interdependent objectives. The first objective consisted in investigating and developing new fundamental methods for co-located and remote interconnection of performers and audiences. The second objective concerned the exploration and empirical studies of new networked and participatory musical forms exploiting the developed methods.
The results obtained during the project demonstrate the achievement of such objectives: various audience-performer musical interfaces were created; methods to interconnect different Musical Things were devised; ecosystems were formed around the developed technologies, which led to novel forms of musical interactions between performers and between performers and audience members.
The adopted approach aspired to effect a step-change in the design of interfaces for musical expression, which has the potential to result in a high economic impact on the whole music industry. The project also contributed to the understanding of the mechanisms underlying music perception and music performance in interactive and multimodal contexts.
Novel musical interfaces targeting both performers and audience members were designed, developed, and evaluated. These interfaces interconnect performers and audience members and support new technology-mediated musical interactions which were investigated in live music performance contexts. This resulted in piloting new approaches for music creation and consumption involving both co-located and remote interactions enabled by wireless networks.
The project focused on two types of musical interfaces: smart musical instruments and musical haptic wearables. The interfaces and their applications were designed by using participatory design methodologies, in most cases.
The smart musical instruments are a family of Internet of Musical Things devices for music performance. They are characterized by sensors, actuators, embedded intelligence, and wireless connectivity to local networks and to the Internet. During the project, prototypes of smart mandolin and smart cajón were developed, by augmenting conventional acoustic mandolin (stringed instrument) and cajón (percussive instrument) with sensing technology, intelligence and network connectivity. Applications for these instruments included the direct connection of the instrument to cloud-based music repositories to create musical accompaniments (avoiding the use of external devices such as personal computers), and the use of apps for smartphones to enable novel kinds of collaborative music making (using the instrument itself as a hub interpreting and rendering the musical messages coming from multiple devices). Such instruments were also used in actual performance contexts.
The musical haptic wearables are a novel class of wearable devices targeting both musicians and audience members. These devices can be designed to provide haptic stimulation, user gesture and physiological factor tracking, and wireless connectivity features. Musical haptic wearables were conceived to enhance creative communication between musicians as well as between musicians and audience members by leveraging the sense of touch, in both co-located and remote settings. They were also devised to enrich musical experiences of audiences of music performances by integrating haptic stimulations, as well as provide new capabilities for creative participation thanks to embedded sensor interfaces. Various kinds of musical haptic wearables and applications were devised during the project, including armbands with embedded motors for the communication between electronic music performers, and jackets with embedded motors that were controlled from the smart mandolin during live music performances in order to enrich the listeners’ experience by adding a tactile sensory layer to the musical content.
Multiple dissemination activities were conducted, the outcomes of which targeted the academic, artistic, and industrial communities along with the general public. In terms of scientific publications, 23 peer-reviewed articles were published (5 journal papers, 2 book chapters, and 16 conference papers), and 3 articles are currently under review. One paper received an award. The fellow presented the project results at 9 international conferences and he gave demonstrations of the developed technologies at 4 national and international events. The fellow was also invited to give a talk at the Re:publica media convention in Berlin as well as to contribute to different blog posts. Finally, the project also resulted in live music performances at national and international festivals and conferences.
The project focused on two types of musical interfaces: smart musical instruments and musical haptic wearables. The interfaces and their applications were designed by using participatory design methodologies, in most cases.
The smart musical instruments are a family of Internet of Musical Things devices for music performance. They are characterized by sensors, actuators, embedded intelligence, and wireless connectivity to local networks and to the Internet. During the project, prototypes of smart mandolin and smart cajón were developed, by augmenting conventional acoustic mandolin (stringed instrument) and cajón (percussive instrument) with sensing technology, intelligence and network connectivity. Applications for these instruments included the direct connection of the instrument to cloud-based music repositories to create musical accompaniments (avoiding the use of external devices such as personal computers), and the use of apps for smartphones to enable novel kinds of collaborative music making (using the instrument itself as a hub interpreting and rendering the musical messages coming from multiple devices). Such instruments were also used in actual performance contexts.
The musical haptic wearables are a novel class of wearable devices targeting both musicians and audience members. These devices can be designed to provide haptic stimulation, user gesture and physiological factor tracking, and wireless connectivity features. Musical haptic wearables were conceived to enhance creative communication between musicians as well as between musicians and audience members by leveraging the sense of touch, in both co-located and remote settings. They were also devised to enrich musical experiences of audiences of music performances by integrating haptic stimulations, as well as provide new capabilities for creative participation thanks to embedded sensor interfaces. Various kinds of musical haptic wearables and applications were devised during the project, including armbands with embedded motors for the communication between electronic music performers, and jackets with embedded motors that were controlled from the smart mandolin during live music performances in order to enrich the listeners’ experience by adding a tactile sensory layer to the musical content.
Multiple dissemination activities were conducted, the outcomes of which targeted the academic, artistic, and industrial communities along with the general public. In terms of scientific publications, 23 peer-reviewed articles were published (5 journal papers, 2 book chapters, and 16 conference papers), and 3 articles are currently under review. One paper received an award. The fellow presented the project results at 9 international conferences and he gave demonstrations of the developed technologies at 4 national and international events. The fellow was also invited to give a talk at the Re:publica media convention in Berlin as well as to contribute to different blog posts. Finally, the project also resulted in live music performances at national and international festivals and conferences.
The umbrella term “Internet of Things” (IoT) broadly refers to the extension of the Internet into the physical realm, by means of everyday physical objects that are spatially distributed and augmented using information and communication technologies. In 2017, when the project begun, the application of IoT technologies in musical contexts had received remarkably little attention compared to other domains such as consumer electronics, healthcare, cities, and geospatial analysis.
This project has conducted pioneering work towards the implementation of the Internet of Musical Things (IoMusT) paradigm by creating Musical Things in the form of new musical instruments and new wearable devices, and by delivering a preliminary technological infrastructure that allows for their bidirectional wireless communication, both locally and remotely. Such an infrastructure has enabled ecosystems of interoperable devices connecting performers as well as performers and audiences to support new performer-performer and audience-performers interactions, not explored beforehand. The project has defined and implemented some use cases demonstrating the proposed IoMusT paradigm and evaluated them.
New smart musical instrument and haptic musical wearable interfaces have progressed the state-of-the-art, and the results produced by the project are expected to impact the music industry in the medium-term, as well as the practices of musicians and the experience of audience members.
This project has conducted pioneering work towards the implementation of the Internet of Musical Things (IoMusT) paradigm by creating Musical Things in the form of new musical instruments and new wearable devices, and by delivering a preliminary technological infrastructure that allows for their bidirectional wireless communication, both locally and remotely. Such an infrastructure has enabled ecosystems of interoperable devices connecting performers as well as performers and audiences to support new performer-performer and audience-performers interactions, not explored beforehand. The project has defined and implemented some use cases demonstrating the proposed IoMusT paradigm and evaluated them.
New smart musical instrument and haptic musical wearable interfaces have progressed the state-of-the-art, and the results produced by the project are expected to impact the music industry in the medium-term, as well as the practices of musicians and the experience of audience members.