Periodic Reporting for period 1 - ReSilence (Retune the Soundscape of future cities through art and science collaboration)
Période du rapport: 2022-09-01 au 2024-04-30
Cities and metropolitan areas are often acting as social condensers, through the dense cohabitation of people, services and cultures. By 2050, cities will host 2.5 billion more urban dwellers, making the world almost 70% urban. The main objective of the ReSilence is to accept this big challenge of the Urbanism/Mobility sector and tries to address one of the less commonly discussed but crucial aspects of urban life: the soundscape.
This is not anymore to reduce what we call noise but to qualify the soundscape, to shape the vibrations of urban spaces, and to design the ambiance of the city in a way that makes it enjoyable, secure, and familiar. ReSilence is acting on the sound of the city as an issue engaging the arts in a transdisciplinary manner, and sound ecologies as an immersive “matter” regarding our understanding of urban life and our possibilities to improve it. To meet the general objective, ReSilence has defined a set of specific objectives, through which it aims to:
• develop AI based methods to increase the active experience of citizens and to integrate individual and collective experience of music and sound.
• achieve a novel understanding of the human perception in relation to sound and his environment
• support the collaboration between artists, users and the technology industry in human- centered design projects through open calls.
The results of ReSilence (artworks, prototype(s) of digital technology or tech component(s), methodologies) are co-created through the collaboration between artists, creatives, scientists and technologists, ensuring that the development of artworks, services and prototypes will follow a human-centered design philosophy.
Additionally, the project focuses on disseminating its results within the architecture, music, and artistic industries through exhibitions, emphasizing the transformative impact of thoughtfully designed soundscapes in these fields.
This is not anymore to reduce what we call noise but to qualify the soundscape, to shape the vibrations of urban spaces, and to design the ambiance of the city in a way that makes it enjoyable, secure, and familiar. ReSilence is acting on the sound of the city as an issue engaging the arts in a transdisciplinary manner, and sound ecologies as an immersive “matter” regarding our understanding of urban life and our possibilities to improve it. To meet the general objective, ReSilence has defined a set of specific objectives, through which it aims to:
• develop AI based methods to increase the active experience of citizens and to integrate individual and collective experience of music and sound.
• achieve a novel understanding of the human perception in relation to sound and his environment
• support the collaboration between artists, users and the technology industry in human- centered design projects through open calls.
The results of ReSilence (artworks, prototype(s) of digital technology or tech component(s), methodologies) are co-created through the collaboration between artists, creatives, scientists and technologists, ensuring that the development of artworks, services and prototypes will follow a human-centered design philosophy.
Additionally, the project focuses on disseminating its results within the architecture, music, and artistic industries through exhibitions, emphasizing the transformative impact of thoughtfully designed soundscapes in these fields.
ReSilence conducted several advanced technical and scientific activities to support experiments with artists. Regarding the work on AI-based technologies for real-time interaction, our work primarily focused on multimodal movement analysis, sonification, soundscape analysis, and AI-based soundscape synthesis.
Multimodal movement analysis and sonification: We developed a framework for multimodal movement analysis and sonification to measure features related to non-verbal social signals of users moving in city spaces. This involved using sensors and computer vision techniques to capture and analyse movements, gestures, and interactions. The data collected was then sonified, translating movement patterns into sound, providing artists with an innovative tool to explore and interpret human dynamics in urban environments.
Soundscape analysis: we implemented a sophisticated soundscape analysis module that included voice activity detection and blurring. This module was essential for experiments that required distinguishing between different sound sources and modulating them for artistic purposes. Additionally, we integrated an emotion recognition system using deep learning techniques. This system could identify and classify the emotional content of soundscapes, offering artists nuanced insights into the auditory environment's affective dimensions.
AI -based soundscapes: we explored AI-based soundscapes through cross-modal learning. Our goal was to develop models capable of synthesizing images from audio inputs and vice versa. This cross-modal approach enabled the creation of visual representations of soundscapes and auditory interpretations of visual scenes, facilitating a rich, immersive experience for the artists. The models were trained using extensive datasets, ensuring high fidelity and relevance to the artistic context.
Regarding the assessment of soundscape experiences our work focused on conducting hyper scanning VR experiments, generating functional tests for sentiment analysis models, setting up a multimodal measurement environment, and developing a unified taxonomy for text comparison.
Emotional sensing and analysis: We conducted a hyper scanning VR experiment to assess performance in noisy environments, examining neutral and stressful noise types (auditory materials provided by artists). We aimed to determine if environmental noise disrupts performance and whether social presence mitigates this impact. Using multimodal measurements, including behavioural analysis, EEG, heart rate monitoring, eye tracking, and gesture analysis, we captured detailed data on participants' reactions. Our findings provided insights into the cognitive and physiological effects of noise and the potential buffering role of social presence in maintaining performance.
Sentiment analysis models: Sentiment analysis involved semantic clustering of spectators’ comments on musical performances and open-class aspect and emotion detection using a large language model. Robustness in hate speech detection was enhanced, and functionalities were adapted for sentiment analysis in the ReSilence domain. Our datasets include monolingual and multilingual data for entity disambiguation and fine-grained emotion recognition.
Multimodal analysis of sound related behaviour: The ArtLab of MPIEA was adapted for ReSilence by identifying the specific needs of artists and expanding relevant technologies, such as behavioural measures for duration of stay and custom-made questionnaires. Components were made suitable for individual and mobile use, and an integrated platform was developed for multimodal data display and analysis, enhancing the functionality and accessibility of the lab for artistic research and experimentation.
Self-report generation: We developed a pipeline for extracting abstract dependency templates for data-to-text NLG and a neural NLG module combining knowledge-to-text and text-to-text models for multimodal report generation. We also created a reporting approach using a unified taxonomy with RAG and LLMs. Preliminary experiments with artists integrated specific lexicons into generated statements.
Multimodal movement analysis and sonification: We developed a framework for multimodal movement analysis and sonification to measure features related to non-verbal social signals of users moving in city spaces. This involved using sensors and computer vision techniques to capture and analyse movements, gestures, and interactions. The data collected was then sonified, translating movement patterns into sound, providing artists with an innovative tool to explore and interpret human dynamics in urban environments.
Soundscape analysis: we implemented a sophisticated soundscape analysis module that included voice activity detection and blurring. This module was essential for experiments that required distinguishing between different sound sources and modulating them for artistic purposes. Additionally, we integrated an emotion recognition system using deep learning techniques. This system could identify and classify the emotional content of soundscapes, offering artists nuanced insights into the auditory environment's affective dimensions.
AI -based soundscapes: we explored AI-based soundscapes through cross-modal learning. Our goal was to develop models capable of synthesizing images from audio inputs and vice versa. This cross-modal approach enabled the creation of visual representations of soundscapes and auditory interpretations of visual scenes, facilitating a rich, immersive experience for the artists. The models were trained using extensive datasets, ensuring high fidelity and relevance to the artistic context.
Regarding the assessment of soundscape experiences our work focused on conducting hyper scanning VR experiments, generating functional tests for sentiment analysis models, setting up a multimodal measurement environment, and developing a unified taxonomy for text comparison.
Emotional sensing and analysis: We conducted a hyper scanning VR experiment to assess performance in noisy environments, examining neutral and stressful noise types (auditory materials provided by artists). We aimed to determine if environmental noise disrupts performance and whether social presence mitigates this impact. Using multimodal measurements, including behavioural analysis, EEG, heart rate monitoring, eye tracking, and gesture analysis, we captured detailed data on participants' reactions. Our findings provided insights into the cognitive and physiological effects of noise and the potential buffering role of social presence in maintaining performance.
Sentiment analysis models: Sentiment analysis involved semantic clustering of spectators’ comments on musical performances and open-class aspect and emotion detection using a large language model. Robustness in hate speech detection was enhanced, and functionalities were adapted for sentiment analysis in the ReSilence domain. Our datasets include monolingual and multilingual data for entity disambiguation and fine-grained emotion recognition.
Multimodal analysis of sound related behaviour: The ArtLab of MPIEA was adapted for ReSilence by identifying the specific needs of artists and expanding relevant technologies, such as behavioural measures for duration of stay and custom-made questionnaires. Components were made suitable for individual and mobile use, and an integrated platform was developed for multimodal data display and analysis, enhancing the functionality and accessibility of the lab for artistic research and experimentation.
Self-report generation: We developed a pipeline for extracting abstract dependency templates for data-to-text NLG and a neural NLG module combining knowledge-to-text and text-to-text models for multimodal report generation. We also created a reporting approach using a unified taxonomy with RAG and LLMs. Preliminary experiments with artists integrated specific lexicons into generated statements.
In the context of the ReSilence project, we have developed prototypes like the Audio Recording Toolbox, ReSilent app, and Theatre of Memory, showcasing innovative methods for real-time, GDPR-compliant sound recording, low-intrusiveness sound design, and interactive sound networks. The potential impacts of these prototypes include enhanced privacy through GDPR-compliant anonymization, innovative loudspeakers transformed to "audio neurons", and improved urban soundscape analysis for better noise management. They foster interdisciplinary research in psychology, sociology, and environmental science. Additionally, they offer commercial and educational opportunities, creating new market avenues and advancing knowledge in sound analysis and manipulation.