Periodic Reporting for period 1 - HARTU (Handling with AI-enhanced Robotic Technologies for flexible manufactUring)
Reporting period: 2023-01-01 to 2024-06-30
The handling of parts in manufacturing environments is a concatenation of some of these basic actions: (1) part grasping; (2) part manipulation, it includes several types of actions such as assembly or execution of a process either in contact with the environment or not (welding, painting, machining, etc.); (3)transportation of the object, carrying it to a destination where it is (4) released or placed in a specific position (machine tending, sorting and packaging).
HARTU will provide components for automatic planning and control of grasping, release and contact assembly tasks, proposing innovative gripping concepts based on electroadhesion for the handling of many different products. These components are integrated through a reference architecture and supported by perception capabilities and application development support tools, with the overall goal of making manufacturing lines more efficient, flexible and reconfigurable.
Adopting a human-centric approach for the design of tools and interfaces developed in the project allows for a comprehensive inclusion of users, making it possible to design technologies that can be better accepted and trusted by users and can improve the user experience at the same time. The interdisciplinary approach shifts the view from technology driven solutions, to user driven ones, by integrating practices from human-factors, usability. In particular:
1. The definition of a smooth collaboration in the human-machine teams
2. Legal and liability aspects: HARTU is studying responsibility and answerability, and thus accountability as the acknowledgement and reporting of any potential negative implications of AI system adoption, especially in domains where system reliability is crucial like in critical industrial application.
3. Stakeholders’ awareness levels and their perception with respect to the introduction of new innovative technologies will be studied to early identify potentialities. HARTU will measure the evolution of the perception to assess the impact in terms of likeability (affective/experience evaluation), and costs (both the financial costs and the social and organizational consequences of adopting the solutions) of the developed solutions.
4. Human Capital and workforce. Three main challenges are being explored in HARTU when finding skilled employees and training the existing workforce: (i) up-skilling the workforce; (ii) the impact of re-skilling of the labour force in case of job displacement or new ones to be created; (iii) mindset change.
HARTU will provide components for automatic planning and control of grasping, release and contact assembly tasks, proposing innovative gripping concepts based on electroadhesion for the handling of many different products. These components are integrated through a reference architecture and supported by perception capabilities and application development support tools, with the overall goal of making manufacturing lines more efficient, flexible and reconfigurable.
Adopting a human-centric approach for the design of tools and interfaces developed in the project allows for a comprehensive inclusion of users, making it possible to design technologies that can be better accepted and trusted by users and can improve the user experience at the same time. The interdisciplinary approach shifts the view from technology driven solutions, to user driven ones, by integrating practices from human-factors, usability. In particular:
1. The definition of a smooth collaboration in the human-machine teams
2. Legal and liability aspects: HARTU is studying responsibility and answerability, and thus accountability as the acknowledgement and reporting of any potential negative implications of AI system adoption, especially in domains where system reliability is crucial like in critical industrial application.
3. Stakeholders’ awareness levels and their perception with respect to the introduction of new innovative technologies will be studied to early identify potentialities. HARTU will measure the evolution of the perception to assess the impact in terms of likeability (affective/experience evaluation), and costs (both the financial costs and the social and organizational consequences of adopting the solutions) of the developed solutions.
4. Human Capital and workforce. Three main challenges are being explored in HARTU when finding skilled employees and training the existing workforce: (i) up-skilling the workforce; (ii) the impact of re-skilling of the labour force in case of job displacement or new ones to be created; (iii) mindset change.
The technical work carried out has been geared towards achieving the project's technical and industrial objectives:
TO1: Self-supervised grasp planning policies identification and control.
HARTU has developed the Local Grasp Planner, which allows to automatically identify the best grasping point for an object given its CAD and the CAD and characteristics of the Gripper. The planner is ready and has been partially validated using the parts defined in the use cases. The next step is to use it intensively in the demonstrators and to perform the final validation.
HARTU is developing the Global Grasp planner, which will select the best candidate among multiple possible candidates in a scene where there are several instances of the same part. The approach consists of creating a neural network trained using Deep Reinforcement Learning (DRL). First, the neural network is initially trained offline using previously collected experience from a real bin-picking setup that uses a manually tuned heuristic policy. Once a base behaviour is learned, the policy is refined online both in the simulated environment and in the real setup, using a typical reinforcement learning setup.
Both planners have been implemented as ROS2 nodes and the corresponding Behaviour Tree Node for the Global Planner are ready for use, although the final version of the Grasp Planner is expected by M24.
TO2: To learn and control contact-rich assembly skills from human demonstrations
HARTU is developing the Learning by demonstration framework in T4.2. The objective is to reduce the need for explicit programming, generalize assembly skills and handle contact-based tasks. As part of the activity, assembly datasets are being recorded and the initial set are available.
In this task we are addressing the open question of how to represent the robot´s skills. The proposed approach is to represent the robot´s movements as Dynamic Movement Primitives (DMP). HARTU is developing the ability to learn DMPs in Cartesian space (CDMP).
TO3: To develop an AI-based multi-modal perception for visual-servoing and continuous monitoring in handling operations, supported by virtual and continuous learning
HARTU is developing a set of tools to obtain a good perception system to identify the geometry and pose of objects that the robot has to manipulate, and to monitor some features of the environment. In this period the focus has been on facilitating object segmentation on scenes and the first version of the pose estimation .
This activity uses deep learning to create an object detection model that is then used as input by a generic image segmentation model (FastSAM) to improve segmentation results. The training of the model is done automatically by creating a dataset of virtual images of the object in the simulation tool created in Unity.
HARTU is also developing high sensibility and low latency acquisition rate sensors using embedded Fibre Brag Gratings (FBGs) to enable continuous monitoring of grasp quality to anticipate the robot behaviour in order to dynamically prevent part slippage.
TO4: To develop versatile and dexterous soft grippers with electro-active fingertips
In WP5, HARTU is researching in the use of electro-active techniques to create new sensors to monitor the contact between grippers and objects and above all, to create a new generation of very versatile soft grippers using electro-adhesion principles. The focus is on the application of this kind of grippers for delicate products.
In this period, a first version of the gripper has been developed, demonstrating not only the grasping capability but some other advantages of these type of gripper: silent, low power consumption (the one implemented can be powered with commercial batteries).
IO1: To increase the flexibility and efficiency of manufacturing lines through easy to integrate and configure, safe and reliable handling systems
This objective is being addressed mainly through the development of the Application Manager Tool, which facilitates the creation of a robotic application through a visual tool, avoiding the need to code the application. The tool uses behaviour trees to represent and control the logic of an application.
The initial version of the tool provides the key nodes to create basic handling and assembly applications and is open to integrate new nodes to respond to specific needs of an application.
TO1: Self-supervised grasp planning policies identification and control.
HARTU has developed the Local Grasp Planner, which allows to automatically identify the best grasping point for an object given its CAD and the CAD and characteristics of the Gripper. The planner is ready and has been partially validated using the parts defined in the use cases. The next step is to use it intensively in the demonstrators and to perform the final validation.
HARTU is developing the Global Grasp planner, which will select the best candidate among multiple possible candidates in a scene where there are several instances of the same part. The approach consists of creating a neural network trained using Deep Reinforcement Learning (DRL). First, the neural network is initially trained offline using previously collected experience from a real bin-picking setup that uses a manually tuned heuristic policy. Once a base behaviour is learned, the policy is refined online both in the simulated environment and in the real setup, using a typical reinforcement learning setup.
Both planners have been implemented as ROS2 nodes and the corresponding Behaviour Tree Node for the Global Planner are ready for use, although the final version of the Grasp Planner is expected by M24.
TO2: To learn and control contact-rich assembly skills from human demonstrations
HARTU is developing the Learning by demonstration framework in T4.2. The objective is to reduce the need for explicit programming, generalize assembly skills and handle contact-based tasks. As part of the activity, assembly datasets are being recorded and the initial set are available.
In this task we are addressing the open question of how to represent the robot´s skills. The proposed approach is to represent the robot´s movements as Dynamic Movement Primitives (DMP). HARTU is developing the ability to learn DMPs in Cartesian space (CDMP).
TO3: To develop an AI-based multi-modal perception for visual-servoing and continuous monitoring in handling operations, supported by virtual and continuous learning
HARTU is developing a set of tools to obtain a good perception system to identify the geometry and pose of objects that the robot has to manipulate, and to monitor some features of the environment. In this period the focus has been on facilitating object segmentation on scenes and the first version of the pose estimation .
This activity uses deep learning to create an object detection model that is then used as input by a generic image segmentation model (FastSAM) to improve segmentation results. The training of the model is done automatically by creating a dataset of virtual images of the object in the simulation tool created in Unity.
HARTU is also developing high sensibility and low latency acquisition rate sensors using embedded Fibre Brag Gratings (FBGs) to enable continuous monitoring of grasp quality to anticipate the robot behaviour in order to dynamically prevent part slippage.
TO4: To develop versatile and dexterous soft grippers with electro-active fingertips
In WP5, HARTU is researching in the use of electro-active techniques to create new sensors to monitor the contact between grippers and objects and above all, to create a new generation of very versatile soft grippers using electro-adhesion principles. The focus is on the application of this kind of grippers for delicate products.
In this period, a first version of the gripper has been developed, demonstrating not only the grasping capability but some other advantages of these type of gripper: silent, low power consumption (the one implemented can be powered with commercial batteries).
IO1: To increase the flexibility and efficiency of manufacturing lines through easy to integrate and configure, safe and reliable handling systems
This objective is being addressed mainly through the development of the Application Manager Tool, which facilitates the creation of a robotic application through a visual tool, avoiding the need to code the application. The tool uses behaviour trees to represent and control the logic of an application.
The initial version of the tool provides the key nodes to create basic handling and assembly applications and is open to integrate new nodes to respond to specific needs of an application.
The preliminar list of results beyond the state of the art (to be completed and validated in the second period) are:
• Grasp planners based on AI and simulation
• Learning from demonstration of assembly tasks
• New contact monitoring sensors
• New soft gripper concept based on electro-adhesion techniques
• Grasp planners based on AI and simulation
• Learning from demonstration of assembly tasks
• New contact monitoring sensors
• New soft gripper concept based on electro-adhesion techniques