Risk-aware Automated Port Inspection Drone(s)

Project Information

RAPID

Grant agreement ID: 861211

DOI

10.3030/861211

Project closed

EC signature date 19 March 2020

Start date 1 June 2020

End date 30 November 2023

Funded under

• SOCIETAL CHALLENGES - Smart, Green And Integrated Transport

Total cost

€ 4 997 133,75

EU contribution

€ 4 997 133,75

Coordinated by

UNIVERSITY OF THE WEST OF SCOTLAND
United Kingdom

Periodic Reporting for period 2 - RAPID (Risk-aware Automated Port Inspection Drone(s))

Reporting period: 2021-12-01 to 2023-11-30

Summary of the context and overall objectives of the project

Public safety is jeopardized by the deteriorating state of Civil Engineering Infrastructure, especially transport system infrastructure such as bridges. Globally, 10% of bridges are deemed high-risk for collapse. The Risk Aware Port Inspection Drones (RAPID) consortium was established to develop and demonstrate intelligent technology solutions to improve the efficiency and productivity of maintenance inspection of large infrastructure and safeguard the public from the consequences of unexpected structural failure.

Prototyping innovative Artificial Intelligence on swarms of autonomous aerial and maritime drones leading to a 20-fold increase in productivity of structural condition monitoring and fault detection. The results were demonstrated and validated in the Port City of Hamburg which is home to Europe’s third largest container port and is the hub of an arterial transport system that connects large-scale maritime and urbanised multimodal transport networks.

RAPID achieved the following objectives :
1. Improve the operational safety of using the UAS and contributing to the risk management when planning their operations.
2. Minimise the risk of collision between the UAS assets and contributing to the robustness of operating within a swarm.
3. Extend the reach and operational duty cycle of the UAS. RAPID will develop smart energy management increasing the reliability of the power systems on the UAS.
4. Improve the efficiency of monitoring the structural condition by introducing smart automation into the insight of the key infrastructure in our test environment.
5. Raising public awareness of the laws and standards for UAS/USV operations. Helping the public to be aware of the technologies and the contribution they can make within civil society.
6. Fast track outcomes of the RAPID project and the technologies developed within RAPID allowing the business model to be scaled.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

RAPID developed and equipped intelligent drones with extended autonomy incorporating hazard perception, autonomous navigation, swarm coordination, and smart energy management such that they can be deployed unsupervised and beyond visual line of sight in complex and dynamic environments. Combining digital twin technologies with embedded artificial intelligence, the drones are capable of detecting sub-mm faults in large scale concrete infrastructure, while also performing tasks to achieve UN Sustainable Development Goals such as air quality monitoring.

UWS LiDAR simulator can synthesise real-world representative UAS perception-sensor data. It successfully virtualises the cost and risk of testing safety-critical flight software (for sense and detect hazard perception) and enables automated pre-flight risk assessment.
UWS developed a UAS perception digital twin, and worked on digitalising the EASA Specific operations Risk Assessment (SORA) and mitigating the risk of air-air swarm collision.

UL led the UAS Flight control work package, developing embedded detect and avoid systems, and within the RAPID use cases achieved a 3D perception of the environment by fusing data from multiple sensors. In a GPS-denied scenario this is addressed through a Simultaneous Localisation and Mapping (SLAM) system.
The Extended Energy Autonomy work package focused on reducing the energy loss of having a landing platform, for autonomous take-off and landing, as well as a Battery Hot swap (BHS) system on the USV developed by Fraunhofer. UL’s real-time automated take-off and landing approaches are developed on a drone, demonstrating cooperative UAV-USV solutions with extended autonomy. Port of Hamburg trials, showcase robustness and adaptability marking a significant advancement in maritime operations.

THL led the Maintenance-Inspection workflow and they were able to demonstrate YOLO algorithms can be used for crack detection. UL developed a photogrammetry workflow to undertake close quarter inspection of the cracks identified by the crack detector. UWS developed an Augmented reality display for 5D models accessible via the web with visualisation through wearable virtual reality headsets.

Results are on the social media platforms and website with videos showcasing the system tests. Conferences attended - TRA Lisbon 2022, Oceans Conference 2023, EASN 2023. The RAPID project has produced 20 scientific publications that will be available through the RAPID website and partner repositories. RAPID achieved all the objectives and hit every KPI within the project plan.

The impact of the project focuses on saving lives through an early warning system, and to ensure the outcomes were achieved the RAPID consortium actively contributed to advancing the current drone and inspection safety systems, engaging public stakeholder and informing the international standards and legal frameworks that will allow the work of RAPID to become commonplace.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

First time airborne sense and detect was demonstrated on a drone with LiDAR and RADAR.
Successfully adapted and extended point-based object detection neural networks from ground-ground to air-air contexts.
Validated a synthetic data augmentation scheme for neural network training that outperformed the state-of-the-art in terms of accuracy (10-15%).
Demonstrated a novel method of performing neural network convolution calculations with data reuse that achieved real-time operating system performance targets.

New approach to GPS-denied localisation within a close-quarter inspection use case through the use of multi-sensor fusion algorithms.
Successfully adapted off-the-shelf UAS for automation real-time perception utilising embedded sensors and compute.
Novel method of performing optimized multi-sensor fusion approach to localisation, enabling GPS-denied localisation capability to operate effectively within a 2-meter proximity to the target.
Embedded Detect and Avoid systems deployed in real-time on the drone, demonstrating safety platforms for small UAS. Incursion detection from uncooperative airborne asset out to 2.9 KM and close-quarter hazard detection to 50m.
Demonstrated a service based, platform agonistic, real-time unmanned traffic management system (UTM, or U-Space) with live situational awareness, embedded robotic-as-a-service request layer, live-streaming functionality, and SORA-compliant risk management features.

State-of-the-art algorithms have been developed and demonstrated (simulation) to provide the best possible solution to the specific constraints of RAPID's use cases.
Demonstrated that YOLO algorithms are effective for crack detection, and after optimization, above the state-of-the-art.
Demonstration that crack detection can be carried out in real time using an embedded architecture on a UAV.

Enhancing the current workflow when combining UAV and USV allows allow a far larger number of inspections to be carried out on infrastructure throughout Europe, providing the early insight necessary to inform and prioritise pre-emptive maintenance and repair of critical infrastructure.

Embedded Detect and Avoid systems for close-quarter hazards detection up to 50m and incursion detect

Harburg Lock Site - RAPID Final demo

Dissemination and knowledge exchange with the coordinator James Riordan at Amsterdam Drone Week.

LiDAR of Harburg Lock - software that can synthesise real-world representative UAS perception-sensor

Battery Hot swap after autonomous landing on the USV for data offload.

Visualisation of a simulated mission in the service layer

Three stage of crack detection and Photogrammetry workflow.

Final Demo for RAPID team in Hamburg