Periodic Reporting for period 2 - ULTRACEPT (Ultra-layered perception with brain-inspired information processing for vehicle collision avoidance)
Berichtszeitraum: 2020-12-01 bis 2024-09-30
Although in their early stages, autonomous vehicles, have demonstrated huge potential in shaping future lifestyles. However, to be accepted by ordinary users, autonomous vehicles have a critical issue to solve – being trustworthy at collision detection. Autonomous vehicles that experience accidents once every few months or years would be unacceptable to the general public. In the real world, human driven vehicles collide at every second. More than 1.3 million people are killed by road accidents every single year. The current approaches for vehicle collision detection such as vehicle to vehicle communication, radar, laser-based Lidar, and GPS are far from acceptable in terms of reliability, cost, energy consumption, and size. For example, radar is too sensitive to metallic material, Lidar is too expensive and does not work well on absorbing/reflective surfaces, GPS based methods are difficult in cities with tall buildings, vehicle to vehicle communication cannot detect pedestrians or any objects unconnected, segmentation based vision methods are too computing power-thirsty to be miniaturized, and normal vision sensors cannot cope with fog, rain and dim environment at night. To save people’s lives and make autonomous vehicles safe to serve human society, a new type of trustworthy, robust, low-cost, and low energy consumption vehicle collision detection and avoidance systems are needed.
This ULTRACEPT consortium proposes an innovative solution with brain-inspired multiple layered and multiple modalities information processing for trustworthy vehicle collision detection. Connecting multidisciplinary teams from different countries together via staff exchange and collaboration, it takes the advantages of low-cost spatial-temporal and parallel computing capacity of bio-inspired visual neural systems and multiple modalities data inputs in extracting potential collision cues at complex weather and lighting conditions.
This ULTRACEPT consortium proposes an innovative solution with brain-inspired multiple layered and multiple modalities information processing for trustworthy vehicle collision detection. Connecting multidisciplinary teams from different countries together via staff exchange and collaboration, it takes the advantages of low-cost spatial-temporal and parallel computing capacity of bio-inspired visual neural systems and multiple modalities data inputs in extracting potential collision cues at complex weather and lighting conditions.
Multi-disciplinary research has been carried out by the consortium according to the plan during the entire project period including the final extended period from Dec 2018 to Sept 2024.
The key challenge in the modelling work is to combine new findings from neurobiologists with existing knowledge. This is the continuous exploration of collision detection neural systems and their underlying mechanisms for mobile intelligent machines such as autonomous vehicles. Researchers in the consortium have proposed and modified the lobular giant movement detector (LGMD) models and tested their performance systematically with various experiments and also at extreme light conditions. These models have now been disseminated in conference and shared within the project consortium via secondments, workshops and networking events. The modelling work has also been extended to bio-inspired locomotion inspired by fish in collaboration with experienced and early stage researchers from partners in Asia and Europe. Secondments between partners have consolidated the collaboration between modelers, system integration researchers and neurobiologists.
To enhance collision detection by integrating multiple visual neural systems, researchers from the consortium proposed novel LGMD model and directional selective visual neural system models with separated on/off channels. On multiple visual neural system integration and coordination in real time systems, researchers have integrated multiple visual neural networks including e-LGMD1, LGMD2, and DS (directional sensitive neural network), into one autonomous robotic system for verification. A swarm robots’ platform has been developed by the partners for testing the vision sensors.
On the system for long distance hazard perception, the consortium proposed new small target movement detector (STMD) models. The proposed model can detect small targets only a few pixels in size. The STMD models are the first step for knowing objects are approaching in distance that may develop into hazards in seconds. The collaborators in Germany have proposed complex-valued neural networks for real-valued classification problems as a white-box model. The proposed models can select the most important features from the complete feature space through a self-organizing modelling process. They also proposed hybrid classification framework based on clustering. These proposed models and methods contributed to both hazard perception and text and road marking recognition.
In order to capture other modalities rather than normal colour vision data, researchers compared thermal sensors from major suppliers and identified the right type of thermal image sensor for data acquisition and selected the right thermal sensors for the project. The feasibility study of using bio-inspired neural systems models such as LGMDs for processing thermal images for collision detection at extreme light conditions has been completed. It demonstrated that the LGMD model works well with temperature-map-based images sensors.
The researchers on secondments to partner universities and SME’s explored the other modality input – sounds - to enhance the safety aspect for driving. Their sound analysis for road condition recognition, such as wet or dry, has been disseminated at an international conference. In collaboration with consortium partners from universities in China and the EU, a road collision database has been created and published for open access in Github.
As part of a new spin off from the research and development activities inspired by the research outcomes of this consortium, researchers from a German partner university have been focusing on how to implement collision avoidance in the robotic scenarios. They have presented a Human-Robot collaboration pipeline that generates efficient and collision-free robot trajectories based on the early motion trajectory and intended target predictions of human arm with optimization capability. The results show that the generated robot trajectory is safe and efficient to complete the whole task together with humans.
During the whole project period (2018-2024), despite the difficulties arising from the pandemic, the consortium has completed: all 20 deliverables; organized all 5 joint workshops and 1 training seminar; completed about 362.12 researcher-month secondments in total; published 79 journal articles and conference papers; achieved all milestones as planned.
The key challenge in the modelling work is to combine new findings from neurobiologists with existing knowledge. This is the continuous exploration of collision detection neural systems and their underlying mechanisms for mobile intelligent machines such as autonomous vehicles. Researchers in the consortium have proposed and modified the lobular giant movement detector (LGMD) models and tested their performance systematically with various experiments and also at extreme light conditions. These models have now been disseminated in conference and shared within the project consortium via secondments, workshops and networking events. The modelling work has also been extended to bio-inspired locomotion inspired by fish in collaboration with experienced and early stage researchers from partners in Asia and Europe. Secondments between partners have consolidated the collaboration between modelers, system integration researchers and neurobiologists.
To enhance collision detection by integrating multiple visual neural systems, researchers from the consortium proposed novel LGMD model and directional selective visual neural system models with separated on/off channels. On multiple visual neural system integration and coordination in real time systems, researchers have integrated multiple visual neural networks including e-LGMD1, LGMD2, and DS (directional sensitive neural network), into one autonomous robotic system for verification. A swarm robots’ platform has been developed by the partners for testing the vision sensors.
On the system for long distance hazard perception, the consortium proposed new small target movement detector (STMD) models. The proposed model can detect small targets only a few pixels in size. The STMD models are the first step for knowing objects are approaching in distance that may develop into hazards in seconds. The collaborators in Germany have proposed complex-valued neural networks for real-valued classification problems as a white-box model. The proposed models can select the most important features from the complete feature space through a self-organizing modelling process. They also proposed hybrid classification framework based on clustering. These proposed models and methods contributed to both hazard perception and text and road marking recognition.
In order to capture other modalities rather than normal colour vision data, researchers compared thermal sensors from major suppliers and identified the right type of thermal image sensor for data acquisition and selected the right thermal sensors for the project. The feasibility study of using bio-inspired neural systems models such as LGMDs for processing thermal images for collision detection at extreme light conditions has been completed. It demonstrated that the LGMD model works well with temperature-map-based images sensors.
The researchers on secondments to partner universities and SME’s explored the other modality input – sounds - to enhance the safety aspect for driving. Their sound analysis for road condition recognition, such as wet or dry, has been disseminated at an international conference. In collaboration with consortium partners from universities in China and the EU, a road collision database has been created and published for open access in Github.
As part of a new spin off from the research and development activities inspired by the research outcomes of this consortium, researchers from a German partner university have been focusing on how to implement collision avoidance in the robotic scenarios. They have presented a Human-Robot collaboration pipeline that generates efficient and collision-free robot trajectories based on the early motion trajectory and intended target predictions of human arm with optimization capability. The results show that the generated robot trajectory is safe and efficient to complete the whole task together with humans.
During the whole project period (2018-2024), despite the difficulties arising from the pandemic, the consortium has completed: all 20 deliverables; organized all 5 joint workshops and 1 training seminar; completed about 362.12 researcher-month secondments in total; published 79 journal articles and conference papers; achieved all milestones as planned.
As detailed in the published report, talks, and research papers uploaded on the project websites:
(http://ultracept.blogs.lincoln.ac.uk/(öffnet in neuem Fenster)) that documents Dec 2018- June 2023 project activities
(https://le.ac.uk/computing-and-mathematical-sciences/research/groups/trustworthy-autonomous-systems/ultracept(öffnet in neuem Fenster)) that documents July 2023- Sept 2024 project activities,
New research outputs beyond the current state of the art have been proposed and verified for improving driving safety by detecting collision early and accurately. In addition, the consortium has created a publicly available collision scene database to aid the development of algorithms. Consortium partners have also secured funding from local funding bodies to contribute to the staff exchange and collaboration in the coming months. The SMEs involved are exploring the potential market of the collision detection visual systems which have led to a new consortium on exploring vehicle sensor systems for road testing and massive production.
(http://ultracept.blogs.lincoln.ac.uk/(öffnet in neuem Fenster)) that documents Dec 2018- June 2023 project activities
(https://le.ac.uk/computing-and-mathematical-sciences/research/groups/trustworthy-autonomous-systems/ultracept(öffnet in neuem Fenster)) that documents July 2023- Sept 2024 project activities,
New research outputs beyond the current state of the art have been proposed and verified for improving driving safety by detecting collision early and accurately. In addition, the consortium has created a publicly available collision scene database to aid the development of algorithms. Consortium partners have also secured funding from local funding bodies to contribute to the staff exchange and collaboration in the coming months. The SMEs involved are exploring the potential market of the collision detection visual systems which have led to a new consortium on exploring vehicle sensor systems for road testing and massive production.