Robust Mobility and Dexterous Manipulation in Disaster Response by Fullbody Telepresence in a Centaur-like Robot

Project Information

CENTAURO

Grant agreement ID: 644839

DOI

10.3030/644839

Project closed

EC signature date 8 December 2014

Start date 1 April 2015

End date 30 September 2018

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 4 124 915,00

EU contribution

€ 4 124 915,00

Coordinated by

RHEINISCHE FRIEDRICH-WILHELMS-UNIVERSITAT BONN
Germany

Periodic Reporting for period 2 - CENTAURO (Robust Mobility and Dexterous Manipulation in Disaster Response by Fullbody Telepresence in a Centaur-like Robot)

Reporting period: 2016-10-01 to 2018-09-30

Summary of the context and overall objectives of the project

Disaster scenarios, like the Fukushima nuclear accident, clearly showed that the capabilities of disaster-response robots were not sufficient for providing the support to rescue workers needed for resolving the situation. The CENTAURO project aimed at the development of a human-robot symbiotic system where a human operator is whole-body telepresent in a Centaur-like robot, which is capable of robust locomotion and dexterous manipulation in the rough terrain and austere conditions characteristic of disasters.
To this end, a versatile robot consisting of a four-legged basis and an anthropomorphic upper body, driven by lightweight, compliant actuators, was to be developed. Navigation in affected man-made environments cluttered with debris and partially collapsed, was to be realized. Manipulation skills should include using unmodified human tools and complex bimanual tasks, such as connecting a hose. For the main operator, a telepresence suit that provides visual, auditory, and upper-body haptic feedback was to be developed. The robot was to be equipped with a rich suit of multimodal environment perception sensors in order to provide the necessary situation awareness. Robot percepts and suggested actions were to be displayed to the operators with augmented reality techniques. For routine manipulation and navigation tasks, autonomous robot skills were to be developed. Operation should be further supported by a physics-based simulation of the robot and its environment.
A series of increasingly complex tests with corresponding evaluation criteria was to be devised from end-user requirements to systematically benchmark the capabilities of the developed disaster response system.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Together with end-users, requirements for the CENTAURO disaster-response system have been defined. Evaluation tasks and performance metrics have been specified. From this, specifications for system components have been derived and concepts for robot design, operator interfaces, modelling & simulation, and autonomous navigation & manipulation have been devised.
The Centauro robot has been designed and manufactured. It has a centaur-like body plan with four legs and an anthropomorphic upper body. The robot is driven by newly developed high-performance torque-controlled series-elastic actuators. Its lower body possesses four articulated legs with five joints each, ending in steerable, actively driven wheels to allow for omnidirectional driving as well as stepping locomotion. The two robot arms have seven joints each and its multi- fingered hands have complementary capabilities. The robot is equipped with a multimodal sensor head consisting of a 3D laser-range finder with spherical field-of-view, panoramic color cameras, and a depth camera. Additional sensors for environment perception are placed at the robot wrist and its base. Real-time control software has been developed, which includes whole-body motion generation and dynamic balance. The robot can be powered by a rechargeable battery and a wireless data link enables untethered operation.
A bi-manual exoskeleton has been designed and manufactured, to capture the arm motion of the main operator and to provide force-feedback during teleoperation of the robot. The exoskeleton has seven joints per arm. Finger motion capture and force feedback is provides by a hand exoskeleton. Bilateral teleoperation control over delayed communication has been realized. Furthermore, a head-mounted display providing live immersive visualization of the robot environment has been integrated.
A simulatable model of the robot and its environment has been developed as a virtual test bed. This digital twin has been used to predict the outcome of specific maneuvers before the real robot executed the selected action.
Efficient software for 3D mapping and semantic terrain classification has been developed to assess locomotion costs. This served as basis for planning hybrid driving-stepping locomotion. Perception software for 3D reconstruction of the manipulation work space, object detection, semantic segmentation, and 6D object pose estimation has been developed. Methods to grasping unknown objects by transferring knowledge from other instances of the same object category have been developed. They are integrated with efficient manipulator trajectory optimization.
The developed disaster-response system has been evaluated according to the identified requirements and performance measures. In the final evaluation, the Centauro robot demonstrated a large variety of challenging locomotion and manipulation tasks.
During the project, 93 scientific publications and 97 other dissemination activities have been reported. Potential customers and stakeholders have been analyzed and suitable ways to approach them have been identified. First hardware components have been introduced to the market and multiple software components and data sets have been released open source. The results achieved in CENTAURO form a solid basis for already running and future research projects.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Many of the developed components advance the state-of-the art, e.g. the highly integrated torque-controlled compliant robot actuators, the real-time robot communication system, the dual-arm exoskeleton, the hand exoskeleton, teleoperation under latency, efficient 3D environment mapping, semantic terrain classification, hybrid driving-stepping locomotion planning, object detection, semantic segmentation, object pose estimation, object grasping, and arm trajectory optimization.
The integrated CENTAURO disaster-response system pushes the state of the art in multiple ways. It provides an unprecedented degree of flexibility regarding its capabilities for locomotion and manipulation. Another novelty is immersive teleoperation via a telepresence suit with unmatched complexity to track precise operator motions and provide detailed force-feedback. Advanced support-operator interfaces and autonomous functionalities allow for efficient robot operation. The compliant series-elastic actuators are designed for robust real-world application which includes unintended rough interactions with the environment. The quadruped base constitutes a robust platform whose stability goes far beyond bipedal robots. Finally, the rich environment data delivered by the installed array of multimodal sensors and a successive generation of intuitive visualizations for the operators and representations for autonomous functionalities is a novelty in this degree of situation awareness.
It is to be expected that many of the developed methods and technologies will be applied in real disaster-response systems and related use cases in the near future. This will have a positive effect on the competitiveness of the European robotics industry and on public safety.

CENTAURO Evaluation Camp Team

Centauro robot traversing uneven terrain

Project logo

Centauro robot

Centauro robot performing two-armed tool use

CENTAURO Operator in Telepresence Suit