Periodic Reporting for period 2 - MERGING (Manipulation Enhancement through Robotic Guidance and Intelligent Novel Grippers)
Período documentado: 2021-05-01 hasta 2022-04-30
The ambition of the MERGING project is to design a turnkey robotic solution and provide manufacturers with an end-to-end solution for the automation of tasks involving the handling of flexible or fragile objects. It will consist of a gripper equipped with an adaptive electro-adhesive skin. Electro-adhesion will increase the attraction forces between the fingers and the object. The skin will also have ability to conform to the objects to handle in order to increase the contact surface. Both will allow reducing the gripper's clamping forces and avoid damaging the objects, while increasing the overall gripping efficiency of the so enhanced gripper. Our solution will include firstly perception and supervision functions to adapt the system's behavior in real time to the execution conditions and to the possibly high variability of the flexible object’s behavior during the task, and secondly control abilities to make safer the human or multi-robot co-manipulation of the flexible object. It will also propose functions for robot system easy programming to make it accessible to non-specialists, like in SME’s.
Academic partners and technology providers have supported the 3 end-users in order to achieve the guidelines for the technical work packages. Thus, the work carried out has contributed to the definition of the 3 use-case needs, the evaluation criteria in relation to the use-cases, the hardware and software technical specifications and the hardware and software architectures.
Electro-adhesive (EA) skins have been fabricated. Their adhesion forces on the 3 use-cases materials have been measured. Shielded soft force sensors compatible with the EA skins have been designed to detect slipping at the object-finger contact. A low-cost product development roadmap has been built for Shadow gripper. Different gripping strategies have been studied theoretically, based on an in depth use-cases analysis. These lead to design-guidelines for the new enhanced gripper.
A set of tools dedicated to easy programming was designed and developed. Firstly, a programming framework provides skill concept and execution engine, easy teaching by demonstration, and learning strategies for skills parameters automatic optimization. It has been validated on Thimonnier use-case, using its GUI. Secondly, a toolbox dedicated to human-robot comanipulation (HRC) includes a set of control strategies for large fragile objects handling, using a mix of hybrid force-position virtual guides and remote-control. The operator can adjust the trajectories and visualize them using augmented reality. For flexible fabrics comanipulation, a new prototype of sensorized glove has been designed and built, with its associated hand gesture library. The HRC toolbox is currently in evaluation on VDL use-case inspired set-ups.
A person tracking and movement decomposition system has been implemented, covering the safety and the human robot interaction. The obtained system is able to decompose the worker movement in shared space with robots. Regarding the human-robot interaction, we have obtained a robust system for the identification of static gestures using the tracking system information. The perception for process control focused on flexible and high deformable materials, with a point cloud-based approach using mixed 3D and 2D sensing. For the textile use-case, wrinkle detection and removal modules were developed to ensure the required manipulation quality. A slip detection strategy using the soft sensor from WP3 was implemented, allowing the system to anticipate a potential slippage during manipulation.
The MERGING’s Workcell Controller has been developed and tested through a first prototype release that incorporates the mechanisms for: a) high-level orchestration of manipulation processes, b) response to errors, and c) creation, edit, storing, execution and monitoring of orders. An intuitive user interface makes these functionalities available to operators with novice programming skills. The Simulator for Ply Manipulation can provide ply modelling at high refresh rates allowing functionalities, such as model-based robot control for diverse comanipulation scenarios. A framework for the automated calibration of the fabric has also been investigated aiming even more reliable fabric reconstruction.
Data collection, required to start integration, has been done. A backup gripper has been validated to manipulate VDL fiberglass fabric. Selmark use-case integration has started. SW integration for the VDL use-case has started, where the Workcell controller is interfaced with the Simulator for ply manipulation. Furthermore, the communication needs and the first version of the ROS interfaces of the software modules have been documented. Investigation started on standards, in order to identify existing regulations or standardization gaps in the project’s main technological areas.
Discussion has started on the project final demonstration details and architecture for the 3 use-cases with all partners. A detailed and updated planning is being built, following the progress of the latest developments with adaptations due to the evolution of the hardware solution (gripper). Pre-trials specifically linked to the use-cases have started with the available building blocks.
The main communication products were developed and updated. A joint workshop at the ERF2021 was organized. 6 research papers have been published in journals or conference proceedings. There were 4 presentations at scientific conferences. One patent request was filed. Up to date Dissemination and Communication Plan, and Exploitation plan, were prepared. Key exploitable results were analyzed for the cost and turnover structure. A content structure and roadmap for the Training Plan was prepared.
Electro-adhesive (EA) skins have been fabricated. Their adhesion forces on the 3 use-cases materials have been measured. Shielded soft force sensors compatible with the EA skins have been designed to detect slipping at the object-finger contact. A low-cost product development roadmap has been built for Shadow gripper. Different gripping strategies have been studied theoretically, based on an in depth use-cases analysis. These lead to design-guidelines for the new enhanced gripper.
A set of tools dedicated to easy programming was designed and developed. Firstly, a programming framework provides skill concept and execution engine, easy teaching by demonstration, and learning strategies for skills parameters automatic optimization. It has been validated on Thimonnier use-case, using its GUI. Secondly, a toolbox dedicated to human-robot comanipulation (HRC) includes a set of control strategies for large fragile objects handling, using a mix of hybrid force-position virtual guides and remote-control. The operator can adjust the trajectories and visualize them using augmented reality. For flexible fabrics comanipulation, a new prototype of sensorized glove has been designed and built, with its associated hand gesture library. The HRC toolbox is currently in evaluation on VDL use-case inspired set-ups.
A person tracking and movement decomposition system has been implemented, covering the safety and the human robot interaction. The obtained system is able to decompose the worker movement in shared space with robots. Regarding the human-robot interaction, we have obtained a robust system for the identification of static gestures using the tracking system information. The perception for process control focused on flexible and high deformable materials, with a point cloud-based approach using mixed 3D and 2D sensing. For the textile use-case, wrinkle detection and removal modules were developed to ensure the required manipulation quality. A slip detection strategy using the soft sensor from WP3 was implemented, allowing the system to anticipate a potential slippage during manipulation.
The MERGING’s Workcell Controller has been developed and tested through a first prototype release that incorporates the mechanisms for: a) high-level orchestration of manipulation processes, b) response to errors, and c) creation, edit, storing, execution and monitoring of orders. An intuitive user interface makes these functionalities available to operators with novice programming skills. The Simulator for Ply Manipulation can provide ply modelling at high refresh rates allowing functionalities, such as model-based robot control for diverse comanipulation scenarios. A framework for the automated calibration of the fabric has also been investigated aiming even more reliable fabric reconstruction.
Data collection, required to start integration, has been done. A backup gripper has been validated to manipulate VDL fiberglass fabric. Selmark use-case integration has started. SW integration for the VDL use-case has started, where the Workcell controller is interfaced with the Simulator for ply manipulation. Furthermore, the communication needs and the first version of the ROS interfaces of the software modules have been documented. Investigation started on standards, in order to identify existing regulations or standardization gaps in the project’s main technological areas.
Discussion has started on the project final demonstration details and architecture for the 3 use-cases with all partners. A detailed and updated planning is being built, following the progress of the latest developments with adaptations due to the evolution of the hardware solution (gripper). Pre-trials specifically linked to the use-cases have started with the available building blocks.
The main communication products were developed and updated. A joint workshop at the ERF2021 was organized. 6 research papers have been published in journals or conference proceedings. There were 4 presentations at scientific conferences. One patent request was filed. Up to date Dissemination and Communication Plan, and Exploitation plan, were prepared. Key exploitable results were analyzed for the cost and turnover structure. A content structure and roadmap for the Training Plan was prepared.
Through the integration of EA-skins to a new mechanical gripper, the project will contribute to a versatile and low cost technology adaptable to many industrial sectors. This result will ensure non-damaging manipulation and adaptability, especially for fragile objects for which no robotic gripping solution currently exists on the market.
To facilitate process adaptation and robotics ease of use, a toolbox will provide a comprehensive set of software and hardware tools, dedicated to intuitive robot steering and teaching, including AI-based functionalities. These tools will be supported by a perception system able to track workers in shared space and understand the behavior of the manipulated deformable materials. Finally, the project activities will lead to the deployment of a controller that incorporates a) high-level control of manufacturing process, b) tools for autonomous decision making and planning based on real-time digital scene representation, c) tools for flexible materials modelling and simulation. The results will contribute to the automation or semi-automation of manufacturing processes that prior-project state-of-the-art could not address. The endgame is to enable agile robotization for many areas of manufacturing where conventional robotics are not economically viable, such as small series, processes with expert gestures or unpredictable context…
Therefore, Merging project should have a noticeable impact on productivity, efficiency and job quality improvement, and help to re-localize productions that rely today on low-wage labor: low-cost and versatility of all the solutions should allow bring production back to Europe.
To facilitate process adaptation and robotics ease of use, a toolbox will provide a comprehensive set of software and hardware tools, dedicated to intuitive robot steering and teaching, including AI-based functionalities. These tools will be supported by a perception system able to track workers in shared space and understand the behavior of the manipulated deformable materials. Finally, the project activities will lead to the deployment of a controller that incorporates a) high-level control of manufacturing process, b) tools for autonomous decision making and planning based on real-time digital scene representation, c) tools for flexible materials modelling and simulation. The results will contribute to the automation or semi-automation of manufacturing processes that prior-project state-of-the-art could not address. The endgame is to enable agile robotization for many areas of manufacturing where conventional robotics are not economically viable, such as small series, processes with expert gestures or unpredictable context…
Therefore, Merging project should have a noticeable impact on productivity, efficiency and job quality improvement, and help to re-localize productions that rely today on low-wage labor: low-cost and versatility of all the solutions should allow bring production back to Europe.