Periodic Reporting for period 3 - MERGING (Manipulation Enhancement through Robotic Guidance and Intelligent Novel Grippers)
Période du rapport: 2022-05-01 au 2023-10-31
The ambition of the MERGING project was to design a turnkey robotic solution and provide manufacturers with an end-to-end solution for the automation of flexible or fragile objects handling tasks. It consists of a gripper equipped with an adaptive electro-adhesive skin. Electro-adhesion increases the attraction forces between the fingers and the object. The skin has also the ability to conform to the objects to handle in order to increase the contact surface. Both allow reducing the gripper's clamping forces and avoid damaging the objects, while increasing overall gripping efficiency. Our solution includes firstly perception and supervision functions to adapt the system's behavior in real time to the execution conditions and to the possibly high variability of the flexible object’s behavior during the task, and secondly control abilities to make safer the human- or multi-robot co-manipulation of the flexible object. It also proposes functions for robot system easy programming, making it accessible to non-specialists, like in SMEs.
Academic partners and technology providers have supported the 3 end-users in order to achieve the guidelines for the technical work packages. Thus, the work carried out has contributed to the definition of the 3 use-cases needs, the evaluation criteria in relation to the use-cases, the hardware and software technical specifications and the hardware and software architectures.
Electro-adhesive (EA) skins have been fabricated. Their adhesion forces on the 3 use-cases materials have been measured. Shielded soft force sensors compatible with the EA skins have been designed to detect slipping at the object-finger contact. Different gripping strategies have been studied theoretically, based on an in depth use-cases analysis. These led to design-guidelines for the new enhanced gripper. Then, two innovative low-cost grippers embedding EA functionality have been designed, optimized and tested, for two of the use-cases. An alternative multi-tool end-effector was designed for the composite use-case, due to physical limitation of EA on the glass-fiber material to manipulate.
A set of tools dedicated to easy programming was designed and developed. Firstly, a programming framework provides skill concept and execution engine, easy teaching by demonstration, and learning strategies for skills parameters automatic optimization. Secondly, a toolbox dedicated to human-robot co-manipulation includes a set of control strategies for large and fragile objects handling, using a mix of hybrid force-position virtual guides and remote control, where the operator can adjust the trajectories and visualize them using augmented reality. A new prototype of sensorized glove has been also designed and built, with its associated hand gesture library for remote robot guiding during co-manipulation.
A person tracking and movement decomposition system has been implemented, covering the safety and the human robot interaction. The obtained system is able to decompose the worker movement in shared space with robots. Regarding the human-robot interaction, we have obtained a robust system for static gestures identification using the tracking system information. The perception for process control focused on highly flexible materials, with a point cloud-based approach using mixed 3D-2D sensing. Textile wrinkle-detection and wrinkle-removal modules were developed to ensure the required manipulation quality. A slip detection strategy using the developed soft sensor was implemented, allowing the system to anticipate a potential slippage during manipulation.
The MERGING Workcell Controller has been integrated with an intuitive user interface and tested, that incorporates: a) high-level orchestration of manipulation processes, b) response to errors, and c) creation, editing, storage, execution and monitoring of orders. In parallel, the Simulator for Ply Manipulation (including automatic calibration for improved reliability) has been developed, enabling high frequency fabric reconstruction and simulation for various scenarios. Practices for seamless inter-resource and inter-module communication have been applied, empowering the twinning of the digital scene and the different hierarchical levels supervision.
By considering hardware and software building blocks previously developed, the 3 use-cases robotic cell demonstrators have been integrated, starting at the RTOs' premises. The communication needs and the ROS interfaces of the software modules have been implemented. Generic capabilities of dual-arm manipulators were enhanced with software and hardware modules for easy programming, handling, cognition, human-robot and robot-robot co-manipulation, interaction, supervision and safety. Investigation on standards has been performed, in order to identify existing regulations or standardization gaps in the project’s main technological areas.
Then, a series of physical and hybrid workshops with the project partners ensured the successful implementation of the 3 final robotic cell demonstrators at the industrial end-users’ premises, involving the latest developments from each technological work package, novel grippers included. These demonstrators have been evaluated through validation campaigns, involving tests with the operators, and proved that MERGING solutions can address main challenges of flexible material robotic manipulation in different industries, by up-scaling ergonomics and/or efficiency.
The dissemination, communication and exploitation plans were completed. MERGING participated in European events (e.g. ERF, IndTech) and it led workshops organization clustering projects related to flexible material handling. The results have been published in journals, and presented at scientific conferences. The exploitable results have been analyzed in terms of cost and turnover. Many patents have been registered. Finally, the project outcomes were presented during the final MERGING public event, as well as training material, also available through the project website.
Electro-adhesive (EA) skins have been fabricated. Their adhesion forces on the 3 use-cases materials have been measured. Shielded soft force sensors compatible with the EA skins have been designed to detect slipping at the object-finger contact. Different gripping strategies have been studied theoretically, based on an in depth use-cases analysis. These led to design-guidelines for the new enhanced gripper. Then, two innovative low-cost grippers embedding EA functionality have been designed, optimized and tested, for two of the use-cases. An alternative multi-tool end-effector was designed for the composite use-case, due to physical limitation of EA on the glass-fiber material to manipulate.
A set of tools dedicated to easy programming was designed and developed. Firstly, a programming framework provides skill concept and execution engine, easy teaching by demonstration, and learning strategies for skills parameters automatic optimization. Secondly, a toolbox dedicated to human-robot co-manipulation includes a set of control strategies for large and fragile objects handling, using a mix of hybrid force-position virtual guides and remote control, where the operator can adjust the trajectories and visualize them using augmented reality. A new prototype of sensorized glove has been also designed and built, with its associated hand gesture library for remote robot guiding during co-manipulation.
A person tracking and movement decomposition system has been implemented, covering the safety and the human robot interaction. The obtained system is able to decompose the worker movement in shared space with robots. Regarding the human-robot interaction, we have obtained a robust system for static gestures identification using the tracking system information. The perception for process control focused on highly flexible materials, with a point cloud-based approach using mixed 3D-2D sensing. Textile wrinkle-detection and wrinkle-removal modules were developed to ensure the required manipulation quality. A slip detection strategy using the developed soft sensor was implemented, allowing the system to anticipate a potential slippage during manipulation.
The MERGING Workcell Controller has been integrated with an intuitive user interface and tested, that incorporates: a) high-level orchestration of manipulation processes, b) response to errors, and c) creation, editing, storage, execution and monitoring of orders. In parallel, the Simulator for Ply Manipulation (including automatic calibration for improved reliability) has been developed, enabling high frequency fabric reconstruction and simulation for various scenarios. Practices for seamless inter-resource and inter-module communication have been applied, empowering the twinning of the digital scene and the different hierarchical levels supervision.
By considering hardware and software building blocks previously developed, the 3 use-cases robotic cell demonstrators have been integrated, starting at the RTOs' premises. The communication needs and the ROS interfaces of the software modules have been implemented. Generic capabilities of dual-arm manipulators were enhanced with software and hardware modules for easy programming, handling, cognition, human-robot and robot-robot co-manipulation, interaction, supervision and safety. Investigation on standards has been performed, in order to identify existing regulations or standardization gaps in the project’s main technological areas.
Then, a series of physical and hybrid workshops with the project partners ensured the successful implementation of the 3 final robotic cell demonstrators at the industrial end-users’ premises, involving the latest developments from each technological work package, novel grippers included. These demonstrators have been evaluated through validation campaigns, involving tests with the operators, and proved that MERGING solutions can address main challenges of flexible material robotic manipulation in different industries, by up-scaling ergonomics and/or efficiency.
The dissemination, communication and exploitation plans were completed. MERGING participated in European events (e.g. ERF, IndTech) and it led workshops organization clustering projects related to flexible material handling. The results have been published in journals, and presented at scientific conferences. The exploitable results have been analyzed in terms of cost and turnover. Many patents have been registered. Finally, the project outcomes were presented during the final MERGING public event, as well as training material, also available through the project website.
Through new EA-enhanced grippers, ensuring gentle manipulation of flexible or fragile objects, MERGING contributed to the rise of a new low-cost robotic technology, versatile and easily adaptable to many industrial sectors.
To facilitate process adaptation and robotics ease-of-use, MERGING provided a comprehensive set of hardware and software tools, including partly AI-based functionalities: intuitive robot steering and teaching, real-time flexible materials and digital scene representation, perception system able to track workers in shared space and understand the highly flexible materials behavior. The deployed work cell controller, which orchestrates all software and hardware tools developed, involves high-level control of manufacturing process and autonomous decision making and planning.
It becomes now possible to create relevant robotic systems for the semi- or full-automation of processes involving flexible objects, but also for the agile robotisation of areas where conventional robotics is not economically viable (such as small series) and/or of processes with expert gestures or unpredictable context…
Therefore, MERGING is expected to have a noticeable impact on the industry (productivity, efficiency) and on the society (job creation and quality/ergonomics improvement). Low-cost and versatility of MERGING solutions should allow bring back to Europe production relying today on low-wage labor.
To facilitate process adaptation and robotics ease-of-use, MERGING provided a comprehensive set of hardware and software tools, including partly AI-based functionalities: intuitive robot steering and teaching, real-time flexible materials and digital scene representation, perception system able to track workers in shared space and understand the highly flexible materials behavior. The deployed work cell controller, which orchestrates all software and hardware tools developed, involves high-level control of manufacturing process and autonomous decision making and planning.
It becomes now possible to create relevant robotic systems for the semi- or full-automation of processes involving flexible objects, but also for the agile robotisation of areas where conventional robotics is not economically viable (such as small series) and/or of processes with expert gestures or unpredictable context…
Therefore, MERGING is expected to have a noticeable impact on the industry (productivity, efficiency) and on the society (job creation and quality/ergonomics improvement). Low-cost and versatility of MERGING solutions should allow bring back to Europe production relying today on low-wage labor.