With the support of this funding, we have performed the following work from the beginning of the project:
1. Conducted a thorough ethics approval process at Imperial College London to ensure that our work and dataset collection were ethical. Our project was reviewed by the college panel, and we attended a panel meeting organized by the Science, Engineering, and Technology Research Ethics Committee (SETREC) of Imperial College London to answer questions from the committee members regarding ethical issues. We fully addressed all the concerns of the committee and obtained ethics approval.
2. Developed a pedestrian crossing intention prediction algorithm based on a spatiotemporal graph neural network. This method takes spatial-temporal information as input and predicts whether pedestrians are crossing the street in a future time window. This algorithm achieved state-of-the-art performance on a public benchmark.
3. Developed a pedestrian trajectory prediction algorithm based on a dual-branch spatiotemporal graph neural network. This algorithm takes image sequences as input and predicts the future positions of pedestrians. Moreover, we designed modules to consider social interactions in our graph neural network. This algorithm achieved very competitive results on public datasets.
4. Published a comprehensive survey on deep learning-based visible and infrared image fusion methods. We also compared the performance of 25 methods, providing the most comprehensive benchmark for the field of visible and infrared image fusion.
5. Captured a multimodal dataset in London. We rented a car, mounted our multimodal sensors on the top, and drove around London to capture data. We collected more than 30 hours of driving data under different lighting conditions in various locations around London.
6. Developed two deep learning-based algorithms to protect pedestrian privacy in videos captured using cameras mounted on vehicles or mobile robots. These algorithms outperform existing methods in terms of pedestrian anonymization while maintaining the utility of anonymized data for downstream computer vision tasks. In particular, these two algorithms provide better temporal consistency in anonymized videos compared with existing methods. Additionally, we created a benchmark for the pedestrian anonymization task.
7. Designed a self-supervised method to train a deep learning-based RGB-thermal object tracker. Unlike existing deep learning-based RGB-thermal object trackers that require a large number of annotated data for training, our method does not need human annotations, saving considerable time on labeling.
8. Developed a deep learning-based human pose estimation method using RGB and thermal images as input. This method provides good human pose estimation results regardless of lighting and weather conditions, offering robust and reliable input for downstream tasks such as pedestrian crossing intention prediction.
We have utilized different methods to communicate and disseminate our results:
1. Public engagement
In June 2023, I participated in the Great Exhibition Road Festival to showcase my research. My demo attracted more than 2000 visitors over the weekend, including children, parents, researchers, and some investors. Tens of children tried our robotic wheelchair.
In July 2023, I participated in the Faculty of Engineering’s “Bring Your Child to Work Day” at Imperial College London, showcasing some unusual cameras (e.g. the thermal camera) used in this project to kids in our department.
2. Website and newsletters at Imperial College London
My projects and I have been featured several times on the Imperial College London website and in newsletters during this project. This has significantly helped communicate and disseminate our results.
3. Social media
I have introduced my project, including results, publications, and public engagement, several times on social media platforms such as Twitter, LinkedIn, and some Chinese social media platforms.
