Novel musical interfaces targeting both performers and audience members were designed, developed, and evaluated. These interfaces interconnect performers and audience members and support new technology-mediated musical interactions which were investigated in live music performance contexts. This resulted in piloting new approaches for music creation and consumption involving both co-located and remote interactions enabled by wireless networks.
The project focused on two types of musical interfaces: smart musical instruments and musical haptic wearables. The interfaces and their applications were designed by using participatory design methodologies, in most cases.
The smart musical instruments are a family of Internet of Musical Things devices for music performance. They are characterized by sensors, actuators, embedded intelligence, and wireless connectivity to local networks and to the Internet. During the project, prototypes of smart mandolin and smart cajón were developed, by augmenting conventional acoustic mandolin (stringed instrument) and cajón (percussive instrument) with sensing technology, intelligence and network connectivity. Applications for these instruments included the direct connection of the instrument to cloud-based music repositories to create musical accompaniments (avoiding the use of external devices such as personal computers), and the use of apps for smartphones to enable novel kinds of collaborative music making (using the instrument itself as a hub interpreting and rendering the musical messages coming from multiple devices). Such instruments were also used in actual performance contexts.
The musical haptic wearables are a novel class of wearable devices targeting both musicians and audience members. These devices can be designed to provide haptic stimulation, user gesture and physiological factor tracking, and wireless connectivity features. Musical haptic wearables were conceived to enhance creative communication between musicians as well as between musicians and audience members by leveraging the sense of touch, in both co-located and remote settings. They were also devised to enrich musical experiences of audiences of music performances by integrating haptic stimulations, as well as provide new capabilities for creative participation thanks to embedded sensor interfaces. Various kinds of musical haptic wearables and applications were devised during the project, including armbands with embedded motors for the communication between electronic music performers, and jackets with embedded motors that were controlled from the smart mandolin during live music performances in order to enrich the listeners’ experience by adding a tactile sensory layer to the musical content.
Multiple dissemination activities were conducted, the outcomes of which targeted the academic, artistic, and industrial communities along with the general public. In terms of scientific publications, 23 peer-reviewed articles were published (5 journal papers, 2 book chapters, and 16 conference papers), and 3 articles are currently under review. One paper received an award. The fellow presented the project results at 9 international conferences and he gave demonstrations of the developed technologies at 4 national and international events. The fellow was also invited to give a talk at the Re:publica media convention in Berlin as well as to contribute to different blog posts. Finally, the project also resulted in live music performances at national and international festivals and conferences.
