Periodic Reporting for period 2 - HERON (Improved Robotic Platform to perform Maintenance and Upgrading Roadworks: The ΗΕRΟΝ Approach)
Okres sprawozdawczy: 2022-12-01 do 2024-05-31
HERON develops an automated system to perform maintenance and upgrading in road infrastructure tasks, such as sealing cracks, patching potholes, asphalt rejuvenation, autonomous replacement of removable urban pavement elements and painting road markings. The system will support the pre- and post-intervention phase including visual inspections, dispensing and removing traffic cones in an automated and controlled manner.
The main goal is the development of an integrated robotic platform with increased navigation and positioning capabilities for maintenance and upgrading tasks and concurrent assessment of the RIs. To progressively achieve this, we cosider multiple research objectives.We will provide an optimized control framework for developing and refining robotic manipulation skills required to perform RI interventions.
Then follows the integration of improved sensing and communication capabilities to the robotic platform to extract the required measurements in the identified areas of interest (ROIs) within an acceptable degree accuracy. ROIs identification calls for implementing an AI toolkit enriched with image analysis modules to optimally coordinate the whole maintenance process and simultaneously process in a smart way the data from the sensing interface to take accurate and prompt decisions which can guarantee an unhindered execution of the traffic flow and a safe execution of routine maintenance works by the personnel.
An integrated Decision Support System (DSS) and an advanced Incident Management System (IMS) with interactive AR/VR visualization tools, will complement the HERON ecosystem. The UI steps over a proper communication architecture, capable to support seamless and ubiquitous services among the various actors.
The main goal is the development of an integrated robotic platform with increased navigation and positioning capabilities for maintenance and upgrading tasks and concurrent assessment of the RIs. To progressively achieve this, we cosider multiple research objectives.We will provide an optimized control framework for developing and refining robotic manipulation skills required to perform RI interventions.
Then follows the integration of improved sensing and communication capabilities to the robotic platform to extract the required measurements in the identified areas of interest (ROIs) within an acceptable degree accuracy. ROIs identification calls for implementing an AI toolkit enriched with image analysis modules to optimally coordinate the whole maintenance process and simultaneously process in a smart way the data from the sensing interface to take accurate and prompt decisions which can guarantee an unhindered execution of the traffic flow and a safe execution of routine maintenance works by the personnel.
An integrated Decision Support System (DSS) and an advanced Incident Management System (IMS) with interactive AR/VR visualization tools, will complement the HERON ecosystem. The UI steps over a proper communication architecture, capable to support seamless and ubiquitous services among the various actors.
This report involves Actions and achievements, and the work carried out from 1/5/2021 to 30/11/2022. During this period, HERON focused on paving the way, for a smooth implementation of an appropriate robotic platform, by considering multiple factors, including end user needs, location, road defects, budget constraints, robotic capabilities, and computer vison limitations.
The use cases and the requirement of the end-users have been defined. Their needs and “usual” way of managing and repairing the road infrastructure has been documented. Moreover, existing tools and methodologies, currently available, have been reviewed and considered for integration to the HERON robotic platform.
Multiple AI-based algorithms and computer vision tools have been investigated, for recognition, classification, and localization of points of interest (PoIs). The work is related to the transformation of 2D optical inputs, to high-level feature maps, through deep learning algorithms that can recognize, classify, and precisely localize the defects of the RI. PoIs include cracks, potholes, faded road markings, and traffic cones.
One of the research fields involves the development of the robotic control and planning approaches, for aiding with RI intervention tasks. Conventional robotic manipulation has limited applicability, when involving fluid-like materials (sand, bitumen), dynamic environments with obstacles and imperfect sensory information. The work to this day was about the design of a low-level control pipeline, including different sensing devices and strategies for end-effectors and tool handling with the arm, motor drivers and low-level actuators to perform the planned motion.
HERON utilizes a heterogeneous fleet of aerial and ground vehicles that aim at making a reconstruction of the environment and assess the area by multi-sensor data analysis. Thus, the hardware/software of the vehicle should follow a modular design. The main outcomes, on this field, are the selection of SLAM algorithms to obtain the 3D reconstruction of the area while the ground vehicle is navigating. For the UAVs, we initiated planning on the flight paths, capturing methods and drone means for deployment and data exchange.
The establishment of a secure data communication framework, for the HERON ecosystem is important. An incident management system is being developed to support maintenance and upgrading operations and coordinated response by the end users. During the reporting period, efforts were allocated in implementing the first parts of the communication, networking, and middleware system layers.
Communication and dissemination activities focused on the implementation of various types of actions aimed to create and enhance visibility of the project, including clustering activities with related EU projects. A comprehensive study was conducted to define a common strategy on how to exploit the project results and to ensure that the results are taken up by relevant stakeholders during and after the project. The study first dealt with the definition of the Individual Exploitation Plans of the consortium partners, and to the exploitable foreground produced.
The use cases and the requirement of the end-users have been defined. Their needs and “usual” way of managing and repairing the road infrastructure has been documented. Moreover, existing tools and methodologies, currently available, have been reviewed and considered for integration to the HERON robotic platform.
Multiple AI-based algorithms and computer vision tools have been investigated, for recognition, classification, and localization of points of interest (PoIs). The work is related to the transformation of 2D optical inputs, to high-level feature maps, through deep learning algorithms that can recognize, classify, and precisely localize the defects of the RI. PoIs include cracks, potholes, faded road markings, and traffic cones.
One of the research fields involves the development of the robotic control and planning approaches, for aiding with RI intervention tasks. Conventional robotic manipulation has limited applicability, when involving fluid-like materials (sand, bitumen), dynamic environments with obstacles and imperfect sensory information. The work to this day was about the design of a low-level control pipeline, including different sensing devices and strategies for end-effectors and tool handling with the arm, motor drivers and low-level actuators to perform the planned motion.
HERON utilizes a heterogeneous fleet of aerial and ground vehicles that aim at making a reconstruction of the environment and assess the area by multi-sensor data analysis. Thus, the hardware/software of the vehicle should follow a modular design. The main outcomes, on this field, are the selection of SLAM algorithms to obtain the 3D reconstruction of the area while the ground vehicle is navigating. For the UAVs, we initiated planning on the flight paths, capturing methods and drone means for deployment and data exchange.
The establishment of a secure data communication framework, for the HERON ecosystem is important. An incident management system is being developed to support maintenance and upgrading operations and coordinated response by the end users. During the reporting period, efforts were allocated in implementing the first parts of the communication, networking, and middleware system layers.
Communication and dissemination activities focused on the implementation of various types of actions aimed to create and enhance visibility of the project, including clustering activities with related EU projects. A comprehensive study was conducted to define a common strategy on how to exploit the project results and to ensure that the results are taken up by relevant stakeholders during and after the project. The study first dealt with the definition of the Individual Exploitation Plans of the consortium partners, and to the exploitable foreground produced.
HERON brings the first real multi-purpose maintenance robotic system within the market. This robotic platform allows RI interventiosns, increasing operations speed and efficiency, and reducing maintenance cost, workers exposure and traffic disruptions.
The first step involves detection of defects, by extending the DL algorithms via an approach more targeted to RI scenarios. The processing frameworks will use 3D information retrieval techniques. Detections provide location information. Next, the correct positioning and navigation of the UGV will be achieved by combining the SLAM-based with the GNSS-based approaches, providing the necessary robustness in featureless and infrastructure-less open space environments.
HERON develops a robotic manipulation and interaction pipeline for complex maintenance and intervention tasks. Learning the low-level system capabilities and the associated pre- and postconditions of successful execution, will allow the robot to automatically synthesize plans that are robust to the noise of real-world deployment.
The project impacts on three areas:
Technical, by showcasing a fully functional and efficient semi-automated system composed by commercial machines working in a integrated manner, using SOTA localization, navigation, AI/ML, comprehensive planning, decision-making, and automated manipulator techniques to carry out complex maintenance tasks.
Economic by reducing personnel needed to perform on-site roadworks. These people could be assigned to perform other tasks in the same company, which would reduce overall execution times or carry out a greater number of activities.
Occupational Safety and Health (OSH) administration by allowing fewer people to work on a road while there is still traffic on the rest of the lanes, reducing the chances of accidents.
The first step involves detection of defects, by extending the DL algorithms via an approach more targeted to RI scenarios. The processing frameworks will use 3D information retrieval techniques. Detections provide location information. Next, the correct positioning and navigation of the UGV will be achieved by combining the SLAM-based with the GNSS-based approaches, providing the necessary robustness in featureless and infrastructure-less open space environments.
HERON develops a robotic manipulation and interaction pipeline for complex maintenance and intervention tasks. Learning the low-level system capabilities and the associated pre- and postconditions of successful execution, will allow the robot to automatically synthesize plans that are robust to the noise of real-world deployment.
The project impacts on three areas:
Technical, by showcasing a fully functional and efficient semi-automated system composed by commercial machines working in a integrated manner, using SOTA localization, navigation, AI/ML, comprehensive planning, decision-making, and automated manipulator techniques to carry out complex maintenance tasks.
Economic by reducing personnel needed to perform on-site roadworks. These people could be assigned to perform other tasks in the same company, which would reduce overall execution times or carry out a greater number of activities.
Occupational Safety and Health (OSH) administration by allowing fewer people to work on a road while there is still traffic on the rest of the lanes, reducing the chances of accidents.