The technical work carried out has been geared towards achieving the project's technical and industrial objectives:
TO1: Self-supervised grasp planning policies identification and control.
HARTU has developed the Local Grasp Planner, which allows to automatically identify the best grasping point for an object given its CAD and the CAD and characteristics of the Gripper. The planner is ready and has been partially validated using the parts defined in the use cases. The next step is to use it intensively in the demonstrators and to perform the final validation.
HARTU is developing the Global Grasp planner, which will select the best candidate among multiple possible candidates in a scene where there are several instances of the same part. The approach consists of creating a neural network trained using Deep Reinforcement Learning (DRL). First, the neural network is initially trained offline using previously collected experience from a real bin-picking setup that uses a manually tuned heuristic policy. Once a base behaviour is learned, the policy is refined online both in the simulated environment and in the real setup, using a typical reinforcement learning setup.
Both planners have been implemented as ROS2 nodes and the corresponding Behaviour Tree Node for the Global Planner are ready for use, although the final version of the Grasp Planner is expected by M24.
TO2: To learn and control contact-rich assembly skills from human demonstrations
HARTU is developing the Learning by demonstration framework in T4.2. The objective is to reduce the need for explicit programming, generalize assembly skills and handle contact-based tasks. As part of the activity, assembly datasets are being recorded and the initial set are available.
In this task we are addressing the open question of how to represent the robot´s skills. The proposed approach is to represent the robot´s movements as Dynamic Movement Primitives (DMP). HARTU is developing the ability to learn DMPs in Cartesian space (CDMP).
TO3: To develop an AI-based multi-modal perception for visual-servoing and continuous monitoring in handling operations, supported by virtual and continuous learning
HARTU is developing a set of tools to obtain a good perception system to identify the geometry and pose of objects that the robot has to manipulate, and to monitor some features of the environment. In this period the focus has been on facilitating object segmentation on scenes and the first version of the pose estimation .
This activity uses deep learning to create an object detection model that is then used as input by a generic image segmentation model (FastSAM) to improve segmentation results. The training of the model is done automatically by creating a dataset of virtual images of the object in the simulation tool created in Unity.
HARTU is also developing high sensibility and low latency acquisition rate sensors using embedded Fibre Brag Gratings (FBGs) to enable continuous monitoring of grasp quality to anticipate the robot behaviour in order to dynamically prevent part slippage.
TO4: To develop versatile and dexterous soft grippers with electro-active fingertips
In WP5, HARTU is researching in the use of electro-active techniques to create new sensors to monitor the contact between grippers and objects and above all, to create a new generation of very versatile soft grippers using electro-adhesion principles. The focus is on the application of this kind of grippers for delicate products.
In this period, a first version of the gripper has been developed, demonstrating not only the grasping capability but some other advantages of these type of gripper: silent, low power consumption (the one implemented can be powered with commercial batteries).
IO1: To increase the flexibility and efficiency of manufacturing lines through easy to integrate and configure, safe and reliable handling systems
This objective is being addressed mainly through the development of the Application Manager Tool, which facilitates the creation of a robotic application through a visual tool, avoiding the need to code the application. The tool uses behaviour trees to represent and control the logic of an application.
The initial version of the tool provides the key nodes to create basic handling and assembly applications and is open to integrate new nodes to respond to specific needs of an application.
