Periodic Reporting for period 2 - ReSilence (Retune the Soundscape of future cities through art and science collaboration)
Período documentado: 2024-05-01 hasta 2025-12-31
Cities and metropolitan areas are often acting as social condensers, through the dense cohabitation of people, services and cultures. By 2050, cities will host 2.5 billion more urban dwellers, making the world almost 70% urban. The main objective of the ReSilence is to accept this big challenge of the Urbanism/Mobility sector and tries to address one of the less commonly discussed but crucial aspects of urban life: the soundscape.
This is not anymore to reduce what we call noise but to qualify the soundscape, to shape the vibrations of urban spaces, and to design the ambiance of the city in a way that makes it enjoyable, secure, and familiar. ReSilence is acting on the sound of the city as an issue engaging the arts in a transdisciplinary manner, and sound ecologies as an immersive “matter” regarding our understanding of urban life and our possibilities to improve it. To meet the general objective, ReSilence has defined a set of specific objectives, through which it aims to:
• develop AI based methods to increase the active experience of citizens and to integrate individual and collective experience of music and sound.
• achieve a novel understanding of the human perception in relation to sound and his environment
• support the collaboration between artists, users and the technology industry in human- centered design projects through open calls.
The results of ReSilence (artworks, prototype(s) of digital technology or tech component(s), methodologies) are co-created through the collaboration between artists, creatives, scientists and technologists, ensuring that the development of artworks, services and prototypes will follow a human-centered design philosophy.
Additionally, the project focuses on disseminating its results within the architecture, music, and artistic industries through exhibitions, emphasizing the transformative impact of thoughtfully designed soundscapes in these fields.
ReSilence overall objectives are:
RO1 – AI-based technologies for real-time interaction
RO2 – Technologies for soundscape experience assessment and emotional/cognitive sensing
TO1 – Art-driven experimentation toolkit and XR development for multisensory environments
AO1 – Art-driven experiments and societal needs for human-centred design
AO2 – Use case scenarios, societal evaluation, and impact assessment
This is not anymore to reduce what we call noise but to qualify the soundscape, to shape the vibrations of urban spaces, and to design the ambiance of the city in a way that makes it enjoyable, secure, and familiar. ReSilence is acting on the sound of the city as an issue engaging the arts in a transdisciplinary manner, and sound ecologies as an immersive “matter” regarding our understanding of urban life and our possibilities to improve it. To meet the general objective, ReSilence has defined a set of specific objectives, through which it aims to:
• develop AI based methods to increase the active experience of citizens and to integrate individual and collective experience of music and sound.
• achieve a novel understanding of the human perception in relation to sound and his environment
• support the collaboration between artists, users and the technology industry in human- centered design projects through open calls.
The results of ReSilence (artworks, prototype(s) of digital technology or tech component(s), methodologies) are co-created through the collaboration between artists, creatives, scientists and technologists, ensuring that the development of artworks, services and prototypes will follow a human-centered design philosophy.
Additionally, the project focuses on disseminating its results within the architecture, music, and artistic industries through exhibitions, emphasizing the transformative impact of thoughtfully designed soundscapes in these fields.
ReSilence overall objectives are:
RO1 – AI-based technologies for real-time interaction
RO2 – Technologies for soundscape experience assessment and emotional/cognitive sensing
TO1 – Art-driven experimentation toolkit and XR development for multisensory environments
AO1 – Art-driven experiments and societal needs for human-centred design
AO2 – Use case scenarios, societal evaluation, and impact assessment
ReSilence carried out coordinated technical, scientific, and art-driven activities across multiple work packages, enabling experimentation with artists and real-world validation of AI-based sound technologies. Key achievements include the development of multimodal AI systems for real-time interaction with sound, combining movement analysis, eco-acoustic sensing, computer vision, and sound synthesis.
The project delivered toolkits for expressive movement analysis, enabling real-time interpretation of individual and collective bodily features, as well as sound and space sensing systems for three-dimensional urban soundscape capture. These systems support ambisonic recording, real-time analysis, and privacy-aware sound source separation, ensuring GDPR compliance.
In parallel, ReSilence developed technologies for soundscape experience assessment, including physiological sensing, behavioural analysis, sentiment analysis of audience feedback, and multilingual self-report generation. These tools were deployed in laboratories, performances, exhibitions, urban pilot sites, and XR environments.
An integrated Art-Driven Experimentation Toolkit combining AI, XR, and multisensory interaction technologies was developed and validated through multiple artistic prototypes. The project also established a structured art-driven methodology, a framework for developing art-driven experiments that explore the social potential of sound in urban environments. It brings together artists, researchers, engineers, communities, and urban actors to experiment with sound, perception, and technology
The project delivered toolkits for expressive movement analysis, enabling real-time interpretation of individual and collective bodily features, as well as sound and space sensing systems for three-dimensional urban soundscape capture. These systems support ambisonic recording, real-time analysis, and privacy-aware sound source separation, ensuring GDPR compliance.
In parallel, ReSilence developed technologies for soundscape experience assessment, including physiological sensing, behavioural analysis, sentiment analysis of audience feedback, and multilingual self-report generation. These tools were deployed in laboratories, performances, exhibitions, urban pilot sites, and XR environments.
An integrated Art-Driven Experimentation Toolkit combining AI, XR, and multisensory interaction technologies was developed and validated through multiple artistic prototypes. The project also established a structured art-driven methodology, a framework for developing art-driven experiments that explore the social potential of sound in urban environments. It brings together artists, researchers, engineers, communities, and urban actors to experiment with sound, perception, and technology
ReSilence developed AI-based technologies for real-time interaction with soundscapes. This includes multimodal algorithms for activity and emotion recognition based on non-verbal behaviour, privacy-aware eco-acoustic soundscape sensing and processing systems, modular AI systems for urban and vehicular soundscapes, and AI-based sound–image correlation and synthesis models. These results go beyond the state of the art by enabling real-time, embodied, and spatial interaction with sound, integrating movement, sound, and visual modalities while respecting GDPR and privacy-by-design principles.
ReSilence delivered technologies for soundscape experience assessment and emotional/cognitive sensing. These include bio-signal-based emotion detection and physiological analysis , multilingual and entity-based sentiment analysis of audience feedback , multimodal behavioural analysis platforms for audience participation, and AI-based multilingual report generation for self-assessment and redesign cycles. Together, these results exceed current soundscape evaluation practices by combining physiological, behavioural, linguistic, and self-reported data into integrated assessment pipelines applicable in real-world artistic and urban environments.
The results of ReSilence include immersive XR installations, EM-based sensing instruments, gesture-based and hands-free interaction devices, soundscape design and algorithmic composition tools, and the integrated ReSilence Toolkit. This goes beyond the state of the art by providing a reusable, artist-centred technological ecosystem that bridges AI, XR, sound, and embodied interaction.
The project produced results, including the definition of requirements and challenges for bridging art, science, and technology, and the ReSilence Art-Driven Methodology. These results formalise artistic practice as a core driver of technology development, ethical reflection, and social trust, going beyond conventional R&D models where art plays only a demonstrative role.
ReSilence validated its results through 15 real-world use case scenarios tested and exhibited in museums, festivals, research institutions, urban environments, and XR settings. This level of public, real-context validation exceeds the state of the art in soundscape research and AI-based interaction, which is typically limited to controlled experimental settings.
ReSilence delivered technologies for soundscape experience assessment and emotional/cognitive sensing. These include bio-signal-based emotion detection and physiological analysis , multilingual and entity-based sentiment analysis of audience feedback , multimodal behavioural analysis platforms for audience participation, and AI-based multilingual report generation for self-assessment and redesign cycles. Together, these results exceed current soundscape evaluation practices by combining physiological, behavioural, linguistic, and self-reported data into integrated assessment pipelines applicable in real-world artistic and urban environments.
The results of ReSilence include immersive XR installations, EM-based sensing instruments, gesture-based and hands-free interaction devices, soundscape design and algorithmic composition tools, and the integrated ReSilence Toolkit. This goes beyond the state of the art by providing a reusable, artist-centred technological ecosystem that bridges AI, XR, sound, and embodied interaction.
The project produced results, including the definition of requirements and challenges for bridging art, science, and technology, and the ReSilence Art-Driven Methodology. These results formalise artistic practice as a core driver of technology development, ethical reflection, and social trust, going beyond conventional R&D models where art plays only a demonstrative role.
ReSilence validated its results through 15 real-world use case scenarios tested and exhibited in museums, festivals, research institutions, urban environments, and XR settings. This level of public, real-context validation exceeds the state of the art in soundscape research and AI-based interaction, which is typically limited to controlled experimental settings.