Advancing the physical intelligence and performance of roBOTs towards human-like bi-manual objects MANipulation

Project Information

MANiBOT

Grant agreement ID: 101120823

DOI

10.3030/101120823

EC signature date 29 June 2023

Start date 1 November 2023

End date 30 April 2027

Funded under

• Digital, Industry and Space

Total cost

€ 6 285 273,75

EU contribution

€ 6 285 273,75

Coordinated by

ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXIS
Greece

Periodic Reporting for period 2 - MANiBOT (Advancing the physical intelligence and performance of roBOTs towards human-like bi-manual objects MANipulation)

Reporting period: 2024-11-01 to 2025-10-31

Summary of the context and overall objectives of the project

Despite the advances in robotics, fast, dexterous and robust objects handling, resembling human’s performance, is limited to industrial applications and takes place in controlled settings with pre-programmed tasks focusing on a-priori known objects. Endowing service robots with advanced physical intelligence that enables them to efficiently interact with their environment and manipulate a wide variety of diverse objects which are not known in advance, in a human-like manner and performance is still an unmet goal. The challenge gets even harder when the objects touch or exceed the robot’s payload capacity as well as when the manipulation takes place in real world, human-populated environments with spatial constraints.
The MANiBOT project aims to research and develop a bimanual mobile robotic platform able to address the aforementioned challenges through advanced perception, control and cognition methods and novel mechatronics. New multimodal environment understanding and object/pose recognition methods are developed based on real-time adaptive fusion of vision, proximity and tactile sensing, enabling fast, safe and efficient manipulation of diverse objects with various sizes, shapes, materials, and rigidity, which are not a-priori known or modelled, in diverse human populated environments. The object handling is performed through the use of a novel suite of manipulation primitives along with bimanual manipulation aiming to achieve performance close to that of humans even under significant spatial constraints. Through non-prehensile primitives (e.g. push, pitch, drag), which utilize supporting surfaces from the environment, the manipulation of heavy or cluttered items is enabled in an energy-efficient way. An advanced multi-level robot cycle framework for cognitive functions orchestrates the above sensing and actuation methods towards efficient and trustworthy behavior that allows learning, composing and swiftly adapting for complex manipulations. The innovative methods of MANiBOT are coupled with novel cognitive mechatronics based on tactile and proximity sensors, integrated within state of the art bimanual mobile manipulation robot, optimized for energy efficiency and increased autonomy, including HRI capabilities for trustworthy and efficient operation.
The above capabilities of the MANiBOT system could have tremendous impact in major sectors of industry and services, from logistics and transport to retail, agri-food and manufacturing, where the use of MANiBOT technologies could give added value and provide a drastic boost in robot utilization in such sectors. MANiBOT’s use cases are indicative of the wide range of areas of potential use of the proposed technologies, since they entail very diverse handling tasks, i.e. shelves’ restocking in supermarkets and baggage handling in airports.
For more information on MANiBOT project, please visit: https://manibot-project.eu/

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Requirements and specifications: The SoA, use cases and technical requirements analysis was performed. The safety and legal aspects of the proposed technologies were analysed.
Robot perception: Developed vision-based methods for semantic segmentation, 6D pose estimation, and detailed subpart detection of objects in cluttered environments, enabling robust object detection despite occlusions and without requiring prior object models. Developed a novel method to generate structural relationships among luggage, incorporating structure-specific data augmentation and multiple fusion techniques. Transfer learning was implemented for accurate tactile inference and a novel method was developed for human detection, 3D pose estimation and gesture recognition using proximity sensors. Developed novel methods for considering explainability metrics in federated learning setups.
Control and navigation: Human-aware navigation methods with high accuracy were developed. Joint reference velocities for prioritized bimanual tasks on a mobile bimanual robot were generated online, with task priorities defined and validated. Learning by Demonstration produced handle-engaging motion primitives integrated with visual feedback. Synchronized reaching, initial contact control, and bimanual manipulation of supported objects were successfully tested across objects with varying physical properties. Object trajectory tracking was achieved using single-point pushing methods, including side and top-down pushing.
Robot cognition and HRI: Developed MANiBOT ontology, semantic task representations and task-graph-based reasoning for mobile bi-manual manipulation. Refined cognitive interfaces between task-level planning and downstream manipulation and platform components ensuring that cognitive abstractions and planning outputs remain consistent with platform capabilities.
Mechatronics: The bimanual mobile robotic platform was designed and a preliminary version was built. Novel tactile and proximity sensors were developed along with a novel proximity robotic skin prototype. The integration of core functionalities of MANiBOT modules has started.
Testing and validation: The testing and demonstration plan was developed while the laboratory environment simulating real-world scenarios was prepared. Iterative component testing has been initiated based on the established plan.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The main innovation results achieved so far are:
•Developed robust vision-based methods for semantic segmentation, 6D pose estimation, affordance prediction, and detailed subpart detection in cluttered environments, under occlusions, without prior object models
•Developed a method to generate structural relationships among piled objects incorporating fusion techniques
•Human detection, 3D object localization and gesture recognition based on proximity sensing
•Development of transfer learning methods for improved tactile inference accuracy
•Novel methods for considering explainability metrics in federated learning setups. Development of a federated model for capacitive sensor data.
•Development of human-aware navigation methods with high accuracy and enhanced resilience in crowded, dynamic environments
•Design of primitive controllers for non-prehensile manipulations for single-item shelf replenishment
•Development of novel methods on bimanual manipulation of sizeable objects and single-arm pushing methods to follow a trajectory
•Developed semantic scene-graph and task-graph representations for structured task reasoning over object relations, affordances, and constraints, supporting adaptive mobile bi-manual manipulation in cluttered environments
•Novel methods for learning task graphs from human demonstrations, improving the transferability and adaptability of robot task execution
•Developed language-driven affordance reasoning to support intuitive task specification and HRI at the cognitive level
•Novel high-resolution tactile sensor using optical fibres under development
•Developed a scalable proximity modular sensing unit prototype

MANIBOT logo