Periodic Reporting for period 1 - SSVPI (Safety in Smart Vehicle - Pedestrian Interaction)
Reporting period: 2022-04-01 to 2024-03-31
The SSVPI project addresses the challenge of enhancing the safety of smart vehicles (SV) in less structured environments, such as pedestrian junctions and mixed-traffic areas. While SVs are highly reliable on structured motorways, their performance in environments with pedestrian interactions needs improvement. Predicting pedestrian intentions is crucial for preventing accidents and ensuring smooth operation in these complex settings.
Improving smart vehicle safety in pedestrian-rich environments is critical for:
1. Safety: Reducing the risk of accidents between pedestrians and vehicles.
2. Public Trust: Building public confidence and acceptance of smart vehicle technologies.
3. Regulatory Support: Supporting governmental and commercial initiatives with robust data and advanced algorithms to improve pedestrian safety and transportation efficiency.
The SSVPI project aims to achieve the following objectives:
1. Develop algorithms that can predict the behavior and intentions of pedestrians in less structured environments to enhance the safety level of pedestrians
2. Capture multimodal driving data, offering valuable resources for pedestrian safety research
3. Promote the public's acceptance of SV as well as trust in SV
The SSVPI project has made meaningful advancements in predicting pedestrian behavior/intention, protecting privacy, and improving pose estimation performance under various conditions. These contributions enhance the safety of pedestrians and the reliability of smart vehicles in less structured environments. The project's outcomes provide valuable resources and insights that support ongoing efforts in autonomous driving technology, contributing to safer and more efficient transportation systems.
Improving smart vehicle safety in pedestrian-rich environments is critical for:
1. Safety: Reducing the risk of accidents between pedestrians and vehicles.
2. Public Trust: Building public confidence and acceptance of smart vehicle technologies.
3. Regulatory Support: Supporting governmental and commercial initiatives with robust data and advanced algorithms to improve pedestrian safety and transportation efficiency.
The SSVPI project aims to achieve the following objectives:
1. Develop algorithms that can predict the behavior and intentions of pedestrians in less structured environments to enhance the safety level of pedestrians
2. Capture multimodal driving data, offering valuable resources for pedestrian safety research
3. Promote the public's acceptance of SV as well as trust in SV
The SSVPI project has made meaningful advancements in predicting pedestrian behavior/intention, protecting privacy, and improving pose estimation performance under various conditions. These contributions enhance the safety of pedestrians and the reliability of smart vehicles in less structured environments. The project's outcomes provide valuable resources and insights that support ongoing efforts in autonomous driving technology, contributing to safer and more efficient transportation systems.
With the support of this funding, we have performed the following work from the beginning of the project:
1. Conducted a thorough ethics approval process at Imperial College London to ensure that our work and dataset collection were ethical. Our project was reviewed by the college panel, and we attended a panel meeting organized by the Science, Engineering, and Technology Research Ethics Committee (SETREC) of Imperial College London to answer questions from the committee members regarding ethical issues. We fully addressed all the concerns of the committee and obtained ethics approval.
2. Developed a pedestrian crossing intention prediction algorithm based on a spatiotemporal graph neural network. This method takes spatial-temporal information as input and predicts whether pedestrians are crossing the street in a future time window. This algorithm achieved state-of-the-art performance on a public benchmark.
3. Developed a pedestrian trajectory prediction algorithm based on a dual-branch spatiotemporal graph neural network. This algorithm takes image sequences as input and predicts the future positions of pedestrians. Moreover, we designed modules to consider social interactions in our graph neural network. This algorithm achieved very competitive results on public datasets.
4. Published a comprehensive survey on deep learning-based visible and infrared image fusion methods. We also compared the performance of 25 methods, providing the most comprehensive benchmark for the field of visible and infrared image fusion.
5. Captured a multimodal dataset in London. We rented a car, mounted our multimodal sensors on the top, and drove around London to capture data. We collected more than 30 hours of driving data under different lighting conditions in various locations around London.
6. Developed two deep learning-based algorithms to protect pedestrian privacy in videos captured using cameras mounted on vehicles or mobile robots. These algorithms outperform existing methods in terms of pedestrian anonymization while maintaining the utility of anonymized data for downstream computer vision tasks. In particular, these two algorithms provide better temporal consistency in anonymized videos compared with existing methods. Additionally, we created a benchmark for the pedestrian anonymization task.
7. Designed a self-supervised method to train a deep learning-based RGB-thermal object tracker. Unlike existing deep learning-based RGB-thermal object trackers that require a large number of annotated data for training, our method does not need human annotations, saving considerable time on labeling.
8. Developed a deep learning-based human pose estimation method using RGB and thermal images as input. This method provides good human pose estimation results regardless of lighting and weather conditions, offering robust and reliable input for downstream tasks such as pedestrian crossing intention prediction.
We have utilized different methods to communicate and disseminate our results:
1. Public engagement
In June 2023, I participated in the Great Exhibition Road Festival to showcase my research. My demo attracted more than 2000 visitors over the weekend, including children, parents, researchers, and some investors. Tens of children tried our robotic wheelchair.
In July 2023, I participated in the Faculty of Engineering’s “Bring Your Child to Work Day” at Imperial College London, showcasing some unusual cameras (e.g. the thermal camera) used in this project to kids in our department.
2. Website and newsletters at Imperial College London
My projects and I have been featured several times on the Imperial College London website and in newsletters during this project. This has significantly helped communicate and disseminate our results.
3. Social media
I have introduced my project, including results, publications, and public engagement, several times on social media platforms such as Twitter, LinkedIn, and some Chinese social media platforms.
1. Conducted a thorough ethics approval process at Imperial College London to ensure that our work and dataset collection were ethical. Our project was reviewed by the college panel, and we attended a panel meeting organized by the Science, Engineering, and Technology Research Ethics Committee (SETREC) of Imperial College London to answer questions from the committee members regarding ethical issues. We fully addressed all the concerns of the committee and obtained ethics approval.
2. Developed a pedestrian crossing intention prediction algorithm based on a spatiotemporal graph neural network. This method takes spatial-temporal information as input and predicts whether pedestrians are crossing the street in a future time window. This algorithm achieved state-of-the-art performance on a public benchmark.
3. Developed a pedestrian trajectory prediction algorithm based on a dual-branch spatiotemporal graph neural network. This algorithm takes image sequences as input and predicts the future positions of pedestrians. Moreover, we designed modules to consider social interactions in our graph neural network. This algorithm achieved very competitive results on public datasets.
4. Published a comprehensive survey on deep learning-based visible and infrared image fusion methods. We also compared the performance of 25 methods, providing the most comprehensive benchmark for the field of visible and infrared image fusion.
5. Captured a multimodal dataset in London. We rented a car, mounted our multimodal sensors on the top, and drove around London to capture data. We collected more than 30 hours of driving data under different lighting conditions in various locations around London.
6. Developed two deep learning-based algorithms to protect pedestrian privacy in videos captured using cameras mounted on vehicles or mobile robots. These algorithms outperform existing methods in terms of pedestrian anonymization while maintaining the utility of anonymized data for downstream computer vision tasks. In particular, these two algorithms provide better temporal consistency in anonymized videos compared with existing methods. Additionally, we created a benchmark for the pedestrian anonymization task.
7. Designed a self-supervised method to train a deep learning-based RGB-thermal object tracker. Unlike existing deep learning-based RGB-thermal object trackers that require a large number of annotated data for training, our method does not need human annotations, saving considerable time on labeling.
8. Developed a deep learning-based human pose estimation method using RGB and thermal images as input. This method provides good human pose estimation results regardless of lighting and weather conditions, offering robust and reliable input for downstream tasks such as pedestrian crossing intention prediction.
We have utilized different methods to communicate and disseminate our results:
1. Public engagement
In June 2023, I participated in the Great Exhibition Road Festival to showcase my research. My demo attracted more than 2000 visitors over the weekend, including children, parents, researchers, and some investors. Tens of children tried our robotic wheelchair.
In July 2023, I participated in the Faculty of Engineering’s “Bring Your Child to Work Day” at Imperial College London, showcasing some unusual cameras (e.g. the thermal camera) used in this project to kids in our department.
2. Website and newsletters at Imperial College London
My projects and I have been featured several times on the Imperial College London website and in newsletters during this project. This has significantly helped communicate and disseminate our results.
3. Social media
I have introduced my project, including results, publications, and public engagement, several times on social media platforms such as Twitter, LinkedIn, and some Chinese social media platforms.
The SSVPI project has achieved significant progress beyond the state of the art in the following areas:
1. Pedestrian Behavior and Intention Prediction
2. Pedestrian Privacy Protection
3. Multimodal Dataset Collection
4. Pedestrian Pose Estimation Under Different Lighting Conditions
5. Self-Supervised Training of Multimodal Object Trackers
The SSVPI project has established a strong foundation for advancing technologies aimed at improving the safety of pedestrians and smart vehicles. Our work has made significant contributions to pedestrian safety, particularly through the development of pedestrian trajectory prediction and pedestrian crossing intention prediction algorithms, as well as the collection of a multimodal dataset. Also, our pedestrian privacy protection algorithms foster public trust in smart technologies in the age of artificial intelligence. Additionally, the project's advancements support the creation of more efficient, safer, and environmentally friendly autonomous systems, contributing to the responsible progression of AI technologies for societal benefit.
1. Pedestrian Behavior and Intention Prediction
2. Pedestrian Privacy Protection
3. Multimodal Dataset Collection
4. Pedestrian Pose Estimation Under Different Lighting Conditions
5. Self-Supervised Training of Multimodal Object Trackers
The SSVPI project has established a strong foundation for advancing technologies aimed at improving the safety of pedestrians and smart vehicles. Our work has made significant contributions to pedestrian safety, particularly through the development of pedestrian trajectory prediction and pedestrian crossing intention prediction algorithms, as well as the collection of a multimodal dataset. Also, our pedestrian privacy protection algorithms foster public trust in smart technologies in the age of artificial intelligence. Additionally, the project's advancements support the creation of more efficient, safer, and environmentally friendly autonomous systems, contributing to the responsible progression of AI technologies for societal benefit.