Periodic Reporting for period 1 - CARMA (Collaborative Autonomous Robots for eMergency Assistance)
Reporting period: 2024-09-01 to 2025-08-31
Context
First responders (FRs) operate in chaotic, dangerous, and highly dynamic environments where reliable information is scarce and uncertainty is high. These inherent risks can lead to tragic fatalities for both victims and rescue teams. While robotics are used for support, the critical need is for a next-generation system that acts as a trustworthy, collaborative partner to human FRs.
CARMA addresses is the need for socially acceptable, autonomous, and semi-autonomous robotic systems that can work in symbiosis with human FRs across hazardous conditions, including those with very low visibility.
Objectives
CARMA’s core objective is to co-create a ground-breaking, secure, modular, and intuitive platform through a user-centred iterative methodology. This platform will offer complementary semi-autonomous and autonomous robotic systems capable of working in symbiosis with humans to support and supplement FRs.
The project achieves this through the following key objectives:
• Co-design a legally, ethically & operationally acceptable and modular platform improving the efficiency and safety of first responders as well as the assistance of affected people for various use cases
• Develop semi and fully autonomous UGVs to increase FRs’ efficiency & safety
• Enable intuitive and socially acceptable human-robot collaboration for emergency operations
• Train End-Users and Affected People and Evaluate CARMA results during realistic pilots
• Maximise Impact, Facilitate Take-up, and Prepare Market Entry
First responders (FRs) operate in chaotic, dangerous, and highly dynamic environments where reliable information is scarce and uncertainty is high. These inherent risks can lead to tragic fatalities for both victims and rescue teams. While robotics are used for support, the critical need is for a next-generation system that acts as a trustworthy, collaborative partner to human FRs.
CARMA addresses is the need for socially acceptable, autonomous, and semi-autonomous robotic systems that can work in symbiosis with human FRs across hazardous conditions, including those with very low visibility.
Objectives
CARMA’s core objective is to co-create a ground-breaking, secure, modular, and intuitive platform through a user-centred iterative methodology. This platform will offer complementary semi-autonomous and autonomous robotic systems capable of working in symbiosis with humans to support and supplement FRs.
The project achieves this through the following key objectives:
• Co-design a legally, ethically & operationally acceptable and modular platform improving the efficiency and safety of first responders as well as the assistance of affected people for various use cases
• Develop semi and fully autonomous UGVs to increase FRs’ efficiency & safety
• Enable intuitive and socially acceptable human-robot collaboration for emergency operations
• Train End-Users and Affected People and Evaluate CARMA results during realistic pilots
• Maximise Impact, Facilitate Take-up, and Prepare Market Entry
Activities Performed
1. Foundational Framework and Architecture
Co-Design and Requirements: Finalized user-centred requirements and four core use-case scenarios through a continuous co-design process.
System Architecture: Translated technical needs into the final Functional Specifications and the essential CARMA System Architecture blueprint.
Ethical Blueprint: Developed the Societal, Ethical, Legal, and Privacy (SELP) framework to ensure the platform is trustworthy by design.
Standardization: Established a Technology and Standardization Framework, continuously monitoring and aggregating partner reporting to ensure project alignment with emerging regulations.
2. Robotic Platform Development
All-Environment Navigation: Delivered the first modular "Perception and Navigation Hardware Module" and developed a ROS-MQTT bridge to unify communication from all UGVs to the Control Center.
Network Infrastructure: Successfully set up and tested the Symbiotic Operations Network Infrastructure in real-world scenarios, demonstrating consistent, high-speed connectivity.
Detection: Development of a deep learning-based victim detection algorithm.
3. Symbiotic Human-Robot Platform
Symbiotic Mission Orchestration: Scientific investigation confirmed that Reinforcement Learning (RL) models provide superior robustness for path planning and scan area optimization in disaster environments compared to conventional methods.
Continuous Integration (CI): Established Agile sprints and deployed a secure integration platform continuous technical validation.
Main Scientific & Technical Achievements
Zero-Visibility Navigation: Final design and initial assembly of the Perception and Navigation Hardware Module, validating the core technical approach for UGV operation in zero-visibility conditions via the fusion of 3D radar and other complementary sensors.
Advanced Human-Robot Interaction: Developed core victim assistance modules, including a) A functional LLM dialog management system that determines a victim's status and automatically adjusts its conversation strategy. b) A validated remote Heart Rate estimation algorithm using visual cues for non-contact victim health assessment.
Human and Object Detection: Developed a lightweight, on-board machine learning-based object detection method that detects humans.
Functional Symbiotic Architecture: Completion of the System Architecture and the initial functional Mission Controller prototype, providing the blueprint for synergistic human-robot task allocation.
Integrated Command and Control Platform (C2): Deployment of the functional CARMA C2 front/back-end services, enabling remote UGV management and, real-time situational awareness.
AI Orchestration Insight: Scientific validation that Reinforcement Learning is the most robust approach for optimal path planning in complex disaster environments.
1. Foundational Framework and Architecture
Co-Design and Requirements: Finalized user-centred requirements and four core use-case scenarios through a continuous co-design process.
System Architecture: Translated technical needs into the final Functional Specifications and the essential CARMA System Architecture blueprint.
Ethical Blueprint: Developed the Societal, Ethical, Legal, and Privacy (SELP) framework to ensure the platform is trustworthy by design.
Standardization: Established a Technology and Standardization Framework, continuously monitoring and aggregating partner reporting to ensure project alignment with emerging regulations.
2. Robotic Platform Development
All-Environment Navigation: Delivered the first modular "Perception and Navigation Hardware Module" and developed a ROS-MQTT bridge to unify communication from all UGVs to the Control Center.
Network Infrastructure: Successfully set up and tested the Symbiotic Operations Network Infrastructure in real-world scenarios, demonstrating consistent, high-speed connectivity.
Detection: Development of a deep learning-based victim detection algorithm.
3. Symbiotic Human-Robot Platform
Symbiotic Mission Orchestration: Scientific investigation confirmed that Reinforcement Learning (RL) models provide superior robustness for path planning and scan area optimization in disaster environments compared to conventional methods.
Continuous Integration (CI): Established Agile sprints and deployed a secure integration platform continuous technical validation.
Main Scientific & Technical Achievements
Zero-Visibility Navigation: Final design and initial assembly of the Perception and Navigation Hardware Module, validating the core technical approach for UGV operation in zero-visibility conditions via the fusion of 3D radar and other complementary sensors.
Advanced Human-Robot Interaction: Developed core victim assistance modules, including a) A functional LLM dialog management system that determines a victim's status and automatically adjusts its conversation strategy. b) A validated remote Heart Rate estimation algorithm using visual cues for non-contact victim health assessment.
Human and Object Detection: Developed a lightweight, on-board machine learning-based object detection method that detects humans.
Functional Symbiotic Architecture: Completion of the System Architecture and the initial functional Mission Controller prototype, providing the blueprint for synergistic human-robot task allocation.
Integrated Command and Control Platform (C2): Deployment of the functional CARMA C2 front/back-end services, enabling remote UGV management and, real-time situational awareness.
AI Orchestration Insight: Scientific validation that Reinforcement Learning is the most robust approach for optimal path planning in complex disaster environments.
1. Zero-Visibility Resilient Perception The current generation of robots fails in smoke or dust. CARMA overcomes this by fusing data from 3D radar with thermal and depth sensors.
2. Advanced Human-Robot Interaction (HRI) CARMA moves beyond basic command-response to enable truly symbiotic interaction and critical health assessment using a) an Adaptive LLM Dialogue System that collects victim information and automatically adjusts its conversation strategy based on the victim's status b) a Remote Biometric Assessment algorithm that estimates the victim's Heart Rate using only video for non-contact, instant health triage.
3. Adaptive Mission Intelligence Scientific investigation confirmed that Reinforcement Learning (RL) models provide superior robustness and optimality for path planning and scan area optimization in disaster environments compared to conventional methods.
4. Integrated Command and Control A functional CARMA C2 Platform has been deployed enabling remote UGV management and maintaining real-time situational awareness through integrated video and telemetry feeds. It incorporates a lightweight, on-board object detection method optimized for visual victim detection and 3D position estimation.
Key Needs for Further Uptake and Success
- Standardization Framework: Actively contribute innovative protocols (HRI, data formats) to relevant European and international Standardization Bodies to ensure interoperability and supportive regulatory frameworks.
- Rigorous Demonstration and Validation: Conduct extensive field validation across all planned pilot scenarios to build the trust and confidence of end-users (First Responders) under operational stress.
- Commercialization and IPR Protection: Establish a clear commercialization strategy, secure IPR for core innovative components, and establish licensing pathways to accelerate technology transfer and market access for technical partners.
2. Advanced Human-Robot Interaction (HRI) CARMA moves beyond basic command-response to enable truly symbiotic interaction and critical health assessment using a) an Adaptive LLM Dialogue System that collects victim information and automatically adjusts its conversation strategy based on the victim's status b) a Remote Biometric Assessment algorithm that estimates the victim's Heart Rate using only video for non-contact, instant health triage.
3. Adaptive Mission Intelligence Scientific investigation confirmed that Reinforcement Learning (RL) models provide superior robustness and optimality for path planning and scan area optimization in disaster environments compared to conventional methods.
4. Integrated Command and Control A functional CARMA C2 Platform has been deployed enabling remote UGV management and maintaining real-time situational awareness through integrated video and telemetry feeds. It incorporates a lightweight, on-board object detection method optimized for visual victim detection and 3D position estimation.
Key Needs for Further Uptake and Success
- Standardization Framework: Actively contribute innovative protocols (HRI, data formats) to relevant European and international Standardization Bodies to ensure interoperability and supportive regulatory frameworks.
- Rigorous Demonstration and Validation: Conduct extensive field validation across all planned pilot scenarios to build the trust and confidence of end-users (First Responders) under operational stress.
- Commercialization and IPR Protection: Establish a clear commercialization strategy, secure IPR for core innovative components, and establish licensing pathways to accelerate technology transfer and market access for technical partners.