Neural Video Processing and Streaming for Real-time Traffic Monitoring

Project Information

VISIONS

Grant agreement ID: 898588

DOI

10.3030/898588

Project closed

EC signature date 7 April 2020

Start date 1 August 2021

End date 31 July 2023

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 224 933,76

EU contribution

€ 224 933,76

Coordinated by

THE UNIVERSITY OF EXETER
United Kingdom

Periodic Reporting for period 1 - VISIONS (Neural Video Processing and Streaming for Real-time Traffic Monitoring)

Reporting period: 2021-08-01 to 2023-07-31

Summary of the context and overall objectives of the project

The problem addressed:
This project focuses on how to achieve real-time traffic monitoring in smart cities, through leveraging the emerging machine learning-based methods in video processing and streaming to reduce the required network bandwidth and guarantee the Quality of Experience (QoE) perceived by users in the dynamic network.


The importance for society:
The expected outcome can promote safer and more efficient travel for millions of users in Europe and billions of users all over the world. Moreover, the results of this project can be used in other multimedia applications, such as cloud virtual reality, distance education, smart transportation, and healthcare where video processing and video streaming are needed.

The overall objectives:
This project aims to achieve real-time traffic monitoring with high-quality video transmission in smart cities, leveraging the emerging Artificial Intelligence methods in video processing and video streaming. Firstly, the features of human visual systems will be referred on video quality allocation to reduce the required bandwidth for video transmission. Next, an innovative method for end-to-end video processing based on Deep Neural Networks will be developed to allow the video rendering and streaming at a lower resolution and also restore/improve the quality at the user ends. Finally, a new bitrate adaption scheme based on Reinforcement Learning will be designed to accommodate the unexpected network dynamics, guaranteeing the Quality-of-Experience to be perceived by users.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project has fully achieved its objectives and milestones for the period. In details,

• Task 1 proposed optimal video quality allocation to reduce the required bandwidth of traffic video transmission without noticeable visual quality degradation. Specifically, the fellow first conducted a comprehensive survey on the existing Quality of Experience (QoE) model for video services, which can be classified into subjective test, objective quality model, and data-driven quality model. Then the fellow proposed to allocate dif-ferent qualities to different macroblocks within the traffic monitoring video to reduce the required bandwidth without noticeable visual quality degradation, which conforms to the feature of the human visual system and the network bandwidth constraints.
Related research results have been published in IUCC 2021.
R. Zhang, X. Zhang*, P. Guo, Q. Fan, H. Yin, and Z. Ma, "QoE Models for Online Video Streaming," 20th International Conference on Ubiquitous Computing and Communications (IUCC/CIT/DSCI/SmartCNS), London, United Kingdom, pp. 32-39, 2021.

• Task 2 proposed neural end-to-end video processing to enhance video quality under bandwidth-limited networks. Specifically, the fellow proposed a residual neural network-based collaborative video processing (CVP) system to trade the computational capability at client-end for QoE improvement via a learned resolution scaling system. Then the fellow proposed a novel raw image-based video analysis framework to perform high-level semantic understanding and low-level compression using RAW images, further reducing the required bandwidth for RGB-based video streaming.
Related research results have been published in IEEE TMC 2023.
W. Miao, G. Min, X. Zhang*, Z. Zhao, J. Hu. Performance Modelling and Quantitative Analysis of Vehicular Edge Computing With Bursty Task Arrivals. IEEE Transactions on Mobile Computing 22(2): 1129-1142 (2023)

• Task 3 designed adaptive video bitrate by using a reinforcement learning-based method, thus accommodating the unexpected network dynamics and guaranteeing users’ QoE. Specifically, the fellow first introduced a lightweight and scalable framework called DMS, for accurate statistical latency measurement on the Internet. Based on the accurate network awareness, the fellow proposed a lightweight Deep Reinforcement Learning (DRL)-based approach to enhance the quality of the video ingestion process of a traffic monitoring system by adjusting the video bitrate adaptively in a real-time manner.
Related research results have been published in IEEE TII 2022 and ACM TOSN 2022.
X. Zhang, G. Min, Q. Fan, H. Yin, D. O. Wu, Z. Ma. A Light-Weight Statistical Latency Measurement Platform at Scale. IEEE Trans. Ind. Informatics 18(2): 1186-1196, 2022.
X. Zhang, Y. Zhao, G. Min, W. Miao, H. Huang, Z. Ma. Intelligent Video Ingestion for Real-time Traffic Monitoring. ACM Trans. Sens. Networks 18(3): 47:1-47:13, 2022.

• Task 4 mainly adopted the results obtained in Task 1 to Task 3 and developed an AI-empowered metaverse prototype for smart cities and beyond. The prototype can provide immersive and personalized experiences to users, thus can accommodate various prospective applications, ranging from smart cities, online education, healthcare, and manufacturing, to real estate.
Related research results have been published in IEEE Communication Magazine 2023.
X. Zhang, G. Min, T. Li, Z. Ma, X. Cao, S. Wang. AI and Blockchain Empowered Metaverse for Web 3.0: Vision, Architecture, and Future Directions. IEEE Communication Magazine 61(8): 60-66, 2023.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The progress beyond the state-of-the-art:
Through the four work packages, the fellow systematically studied machine learning based methods in video processing and streaming to enable the real-time traffic monitoring in smart cities, thus reducing the required network bandwidth and guaranteeing the Quality of Experience (QoE) perceived by users in the dynamic network. Specifically, the fellow first proposed optimal video quality allocation to reduce the required bandwidth of traffic video transmission without noticeable visual quality degradation. Then the fellow proposed neural end-to-end video processing to enhance video quality under bandwidth-limited networks and designed adaptive video bitrate by using a reinforcement learning based method, thus accommodating the unexpected network dynamics and guaranteeing users’ QoE. Finally, the fellow developed an AI-empowered metaverse prototype for smart cities and beyond.

Potential Impact:
These advancements can yield profound positive impacts on society, encompassing enhanced safety, reduced congestion, and optimized emergency response strategies.
• Improved Traffic Safety: Real-time monitoring allows for immediate detection and response to accidents or road hazards. Emergency services can be notified promptly, potentially saving lives and reducing the severity of injuries in accidents.
• Reduced Traffic Congestion: With real-time video on traffic flow and incidents, authorities can implement real-time traffic management strategies. This can help redirect traffic, optimize traffic signal timings, and manage congestion more effectively.
• Emergency Response Optimization: Real-time traffic monitoring can significantly improve emergency response times during critical situations, enabling quicker access to areas in need and potentially saving lives during emergencies.

Metaverse for Smart Cities