Periodic Reporting for period 2 - I-SPOT (Intelligent Ultra Low-Power Signal Processing for Automotive)
Berichtszeitraum: 2022-11-01 bis 2024-10-31
While traditional cars relied solely on the senses of their drivers – mostly their eyes and ears –, the car of today is equipped with more and more sensors. Self-driving cars come within reach by equipping cars with their own set of eyes, in the form of LIDARs and cameras. Yet, to become truly aware of their environment, such smart cars are still missing an invaluable input on which we, as humans, rely strongly: the acoustic information. I-SPOT targets the addition of acoustic sensing technology to cars, to bring enhanced environmental awareness.
This acoustic information complements the information from other sensory technologies. In active (drive) mode, this will give information about nearby emergency vehicles, accidents, passing cars, etc., which are currently causing major disruptions of autonomous and computer assisted driving. Moreover, information on weather conditions, or mechanical car wear or failure is present in the acoustic signal. More than just detection of the nature of the sound, also the direction can be derived, and this even for visually occluded sources. In passive (park) mode, information can be obtained on car damaging, theft, or nearby critical events (e.g. cry for help). The acoustic sensor can as such form the low-cost wake-up trigger to direct the more power-hungry camera system to activate and point in a specific direction.
This acoustic information complements the information from other sensory technologies. In active (drive) mode, this will give information about nearby emergency vehicles, accidents, passing cars, etc., which are currently causing major disruptions of autonomous and computer assisted driving. Moreover, information on weather conditions, or mechanical car wear or failure is present in the acoustic signal. More than just detection of the nature of the sound, also the direction can be derived, and this even for visually occluded sources. In passive (park) mode, information can be obtained on car damaging, theft, or nearby critical events (e.g. cry for help). The acoustic sensor can as such form the low-cost wake-up trigger to direct the more power-hungry camera system to activate and point in a specific direction.
The I-SPOT project resulted in a wide range of scientific achievements, many of which have also influence the product roadmaps of our industrial project partner Bosch.
1. Synthetic Data Generation Techniques for Training Deep Acoustic Siren Identification Networks (published 2024): In this study ESR1 explored methods to enhance the training of deep learning models for siren identification by generating synthetic siren signals. This work was started on the explicit suggestion from Bosch, who requires this feature for their future smart cars.
2. A Framework for the Acoustic Simulation of Passing Vehicles Using Traffic Flow Data (published in 2024): ESR1 introduced TrafficSoundSim, an open-source framework designed to simulate the acoustics of vehicles passing on a road. This work was done in tight collaboration with Bosch, during the research stay of ESR1 at Bosch
3. Frequency Tracking Features for Data-Efficient Deep Siren Identification (published in 2023): ESR1 developed frequency tracking features to improve the efficiency of deep learning models in identifying sirens.
4. Pyroadacoustics: A Road Acoustics Simulator Based on Variable Length Delay Lines (published in 2022): ESR1 introduced Pyroadacoustics, a simulator that models road acoustics using variable length delay lines. This tool provides a framework for simulating the acoustic environment of roads, which can be useful for studying traffic noise and its impact on surrounding areas.
5. A CNN-based Robust Sound Source Localization with SRP-PHAT for the Extreme Edge (published in 2023): ESR2 enhanced the SRP-PHAT algorithm using CNNs to improve robustness and efficiency, making it suitable for deployment on edge devices with limited computational resources. With the help from industrial partner Bosch, this algorithm was also effectively mapped on a microcontroller platform.
6. ACCO: Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators (published in 2023): ESR2 developed the open-source framework ACCO, an automated optimizer designed to enhance the efficiency of Spatio-Temporal Convolutional Neural Networks (ST-CNNs) on edge hardware accelerators. By exploring efficient causal CNN transformations and depth-first scheduling, ACCO can optimize computation and data movement, thereby improving the performance and energy efficiency of real-time applications on edge devices.
7. Coarse-Grained Reconfigurable Array implementation (2024 and ongoing): ESR2 developed a complete processing fabric making use of the principles of CGRA's (coarse Grained Reconfigurable Arrays) compatible with both the developed acoustic neural networks, as well as the required preprocessing (auto-correlation, FFT,...) and post-processing (normalizations, filtering). This development include the hardware design (Verilog), compiler design, and chip tape out. The chip is currently in production.
Finally, knowledge has been vividly shared by all partners, with several spillovers to the Bosch departments, such as the traffic and siren detection algorithms and the microcontroller mappings. These collaborations also results in shared papers (several already mentioned earlier), as well as a shared vision/overview paper (published in 2023).
1. Synthetic Data Generation Techniques for Training Deep Acoustic Siren Identification Networks (published 2024): In this study ESR1 explored methods to enhance the training of deep learning models for siren identification by generating synthetic siren signals. This work was started on the explicit suggestion from Bosch, who requires this feature for their future smart cars.
2. A Framework for the Acoustic Simulation of Passing Vehicles Using Traffic Flow Data (published in 2024): ESR1 introduced TrafficSoundSim, an open-source framework designed to simulate the acoustics of vehicles passing on a road. This work was done in tight collaboration with Bosch, during the research stay of ESR1 at Bosch
3. Frequency Tracking Features for Data-Efficient Deep Siren Identification (published in 2023): ESR1 developed frequency tracking features to improve the efficiency of deep learning models in identifying sirens.
4. Pyroadacoustics: A Road Acoustics Simulator Based on Variable Length Delay Lines (published in 2022): ESR1 introduced Pyroadacoustics, a simulator that models road acoustics using variable length delay lines. This tool provides a framework for simulating the acoustic environment of roads, which can be useful for studying traffic noise and its impact on surrounding areas.
5. A CNN-based Robust Sound Source Localization with SRP-PHAT for the Extreme Edge (published in 2023): ESR2 enhanced the SRP-PHAT algorithm using CNNs to improve robustness and efficiency, making it suitable for deployment on edge devices with limited computational resources. With the help from industrial partner Bosch, this algorithm was also effectively mapped on a microcontroller platform.
6. ACCO: Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators (published in 2023): ESR2 developed the open-source framework ACCO, an automated optimizer designed to enhance the efficiency of Spatio-Temporal Convolutional Neural Networks (ST-CNNs) on edge hardware accelerators. By exploring efficient causal CNN transformations and depth-first scheduling, ACCO can optimize computation and data movement, thereby improving the performance and energy efficiency of real-time applications on edge devices.
7. Coarse-Grained Reconfigurable Array implementation (2024 and ongoing): ESR2 developed a complete processing fabric making use of the principles of CGRA's (coarse Grained Reconfigurable Arrays) compatible with both the developed acoustic neural networks, as well as the required preprocessing (auto-correlation, FFT,...) and post-processing (normalizations, filtering). This development include the hardware design (Verilog), compiler design, and chip tape out. The chip is currently in production.
Finally, knowledge has been vividly shared by all partners, with several spillovers to the Bosch departments, such as the traffic and siren detection algorithms and the microcontroller mappings. These collaborations also results in shared papers (several already mentioned earlier), as well as a shared vision/overview paper (published in 2023).
The I-SPOT project has enabled large steps forward for the state-of-the-art in helping cars “listen” to their surroundings more effectively. The ESRs have focused on two key areas: a.) developing smarter ways for cars to recognize sounds (including sound generation for faster network training) and b.) creating efficient hardware to sense and process this information in real time. These breakthroughs are already influencing Bosch's portfolio and shaping how future vehicles will detect emergency sirens, predict traffic noise, and process sounds with minimal energy use.
One of the biggest achievements is improving how cars recognize emergency vehicle sirens. Normally, deep learning models need a lot of real-world data to train, but collecting and labeling thousands of siren sounds isn’t always practical. To solve this, the team developed a technique to generate synthetic siren sounds, allowing AI models to train even when real-world recordings are scarce. This means that even in cities where sirens sound slightly different, smart cars will still recognize them and react quickly, helping ambulances and fire trucks get through traffic more efficiently.
Another major innovation is the TrafficSoundSim package, a tool designed to predict what traffic will sound like based on how many cars are on the road. This can not only be used to train new neural network, but could in the future also be is a big step forward for urban planning because it allows cities to simulate noise levels and find ways to reduce traffic noise before roads are even built. ESR1 also developed Pyroadacoustics, a system that models how different road surfaces affect sound, helping engineers design roads that minimize noise pollution.
But it’s not just about software—hardware plays a huge role too. One of the key challenges in sound processing is that deep learning models require a lot of computing power, which isn’t ideal for cars that need to make quick decisions without draining too much battery. To address this, the researchers optimized a sound source localization system that runs on a tiny, low-power microcontroller, together with Bosch engineers. On top of that, ESR2 developed a completely new computer chip designed specifically for acoustic signal processing. It’s currently being built and will soon be tested in real-world applications.
The impact of the I-SPOT project hence goes beyond just smarter cars. With better siren detection, emergency vehicles can get to their destinations faster, potentially saving lives. Cities can use these tools to design quieter, more livable urban spaces. By making sound processing more efficient and accessible, the I-SPOT project is paving the way for a world where cars, roads, and cities are not just connected—but truly aware of their surroundings.
One of the biggest achievements is improving how cars recognize emergency vehicle sirens. Normally, deep learning models need a lot of real-world data to train, but collecting and labeling thousands of siren sounds isn’t always practical. To solve this, the team developed a technique to generate synthetic siren sounds, allowing AI models to train even when real-world recordings are scarce. This means that even in cities where sirens sound slightly different, smart cars will still recognize them and react quickly, helping ambulances and fire trucks get through traffic more efficiently.
Another major innovation is the TrafficSoundSim package, a tool designed to predict what traffic will sound like based on how many cars are on the road. This can not only be used to train new neural network, but could in the future also be is a big step forward for urban planning because it allows cities to simulate noise levels and find ways to reduce traffic noise before roads are even built. ESR1 also developed Pyroadacoustics, a system that models how different road surfaces affect sound, helping engineers design roads that minimize noise pollution.
But it’s not just about software—hardware plays a huge role too. One of the key challenges in sound processing is that deep learning models require a lot of computing power, which isn’t ideal for cars that need to make quick decisions without draining too much battery. To address this, the researchers optimized a sound source localization system that runs on a tiny, low-power microcontroller, together with Bosch engineers. On top of that, ESR2 developed a completely new computer chip designed specifically for acoustic signal processing. It’s currently being built and will soon be tested in real-world applications.
The impact of the I-SPOT project hence goes beyond just smarter cars. With better siren detection, emergency vehicles can get to their destinations faster, potentially saving lives. Cities can use these tools to design quieter, more livable urban spaces. By making sound processing more efficient and accessible, the I-SPOT project is paving the way for a world where cars, roads, and cities are not just connected—but truly aware of their surroundings.