Periodic Reporting for period 2 - PICKPLACE (Flexible, safe and dependable robotic part handling in industrial environments)
Reporting period: 2019-07-01 to 2021-04-30
Pick and place are basic operations in most robotic applications, whether in industrial setups (e.g. machine tending,assembling or bin picking) or in a service robotic domain (e.g. agriculture or at home). In some structured scenarios and
with certain types of parts, picking and placing is a mature process. However, that is not the case when it comes to manipulating parts with high variability or in less structured environments.
Handling systems are present in any logistics system to interface between the storage and the transportation systems. For non-structured scenarios, picking, packing and unpacking systems do exist at laboratory level. However, they have not reached the market yet due to factors like the lack of efficiency, robustness and flexibility of currently available manipulation and perception technologies.
The market demands systems that allow for a reduction of costs in the supply chain, increasing the competitiveness for manufacturers and bringing a cost reduction for consumers. Handling systems represent the highest impact in the shorttomidterm in warehouse-based systems (mainly at order picking and distribution centres) and in intra-logistics operations in factories and retail.
The technology gap is the lack of flexible solutions that can handle objects of variable size, shape and weight as well as different surface properties and stiffness.
PICKPLACE proposes combining human and robot capabilities to achieve a safe, flexible, dependable and efficient hybrid pick-and-package (PAK) solution. It includes dynamic package configuration planning, flexible grasping strategies using an innovative multifunctional gripper, robust environment perception and mechanisms and strategies for safe human-robot collaboration.
The Technological and Industrial objectives (TO and IO) identified are:
TO 1: To develop a new generation of multifunctional grippers to handle products of different morphology, weight and rigidity and that are able to reach difficult to access target positions
SO 1: To develop reactive grasp-planning algorithm based on cognitive capabilities and allows the robot to effectively grasp different objects
SO 2: Robust and efficient bin-picking solution based on object pose identification and fast and safe robot path planning
SO 3: Human and robot affordance aware dynamic package planning for mono and multireference configurations.
SO 4: Dynamic robot planning based on cognitive capabilities exploiting perceived monitoring and human activity.
SO 5: Reliable environment perception system and strategies for safe collaborative scenarios based on Speed and Separation Monitoring combined with Power and Force Limiting.
IO 1: To increase the pick-and-package global performance in terms of flexibility, dependability and error reduction.
IO 2: Improvement of the working conditions of operators by a proper layout design and task allocation between worker and robot.
with certain types of parts, picking and placing is a mature process. However, that is not the case when it comes to manipulating parts with high variability or in less structured environments.
Handling systems are present in any logistics system to interface between the storage and the transportation systems. For non-structured scenarios, picking, packing and unpacking systems do exist at laboratory level. However, they have not reached the market yet due to factors like the lack of efficiency, robustness and flexibility of currently available manipulation and perception technologies.
The market demands systems that allow for a reduction of costs in the supply chain, increasing the competitiveness for manufacturers and bringing a cost reduction for consumers. Handling systems represent the highest impact in the shorttomidterm in warehouse-based systems (mainly at order picking and distribution centres) and in intra-logistics operations in factories and retail.
The technology gap is the lack of flexible solutions that can handle objects of variable size, shape and weight as well as different surface properties and stiffness.
PICKPLACE proposes combining human and robot capabilities to achieve a safe, flexible, dependable and efficient hybrid pick-and-package (PAK) solution. It includes dynamic package configuration planning, flexible grasping strategies using an innovative multifunctional gripper, robust environment perception and mechanisms and strategies for safe human-robot collaboration.
The Technological and Industrial objectives (TO and IO) identified are:
TO 1: To develop a new generation of multifunctional grippers to handle products of different morphology, weight and rigidity and that are able to reach difficult to access target positions
SO 1: To develop reactive grasp-planning algorithm based on cognitive capabilities and allows the robot to effectively grasp different objects
SO 2: Robust and efficient bin-picking solution based on object pose identification and fast and safe robot path planning
SO 3: Human and robot affordance aware dynamic package planning for mono and multireference configurations.
SO 4: Dynamic robot planning based on cognitive capabilities exploiting perceived monitoring and human activity.
SO 5: Reliable environment perception system and strategies for safe collaborative scenarios based on Speed and Separation Monitoring combined with Power and Force Limiting.
IO 1: To increase the pick-and-package global performance in terms of flexibility, dependability and error reduction.
IO 2: Improvement of the working conditions of operators by a proper layout design and task allocation between worker and robot.
TO 1:
A multifunctional gripper, composed by 3 different end effectors (suction, magnetic and grasping) to handle the variety of products included in both end-users’ scenarios, with some restrictions. It is in line with other grippers that are being developed by other research groups.
SO 1:
Reactive grasp-planning is understood as the adaptation of an initial grasp of an object in response to sensory feedback (in our case haptic feedback) to achieve a more stable and reliable grasping. The algorithms developed allow adapting the robot motion planning in accordance with the pressure feedback from tactile sensors embedded in the grippers. It has been validated for different sensor configurations in suction cups and finger grippers’.
SO 2:
A software component that uses 3D images and deep learning models has been developed to estimate the best grasping points for objects, without the need for ad-hoc configuration.
SO 3:
A mosaic creation algorithm has been designed and implemented to handle the problem of stackable and non-stackable parts packaging configuration. The software takes as input the characteristics of the part, the actual way it has been grasped and the current status of the destination box where it has to be placed.
SO 4:
An algorithm to optimize a cost function that takes into account the human 3D occupancy map to off-line plan paths that reduce the probably of interference with the operator has been developed.
SO5:
It has been developed a system that combines machine vision and deep learning to monitor the shared human – robot working environment. It has been included the detection of up to 18 different keypoints, and the relative distance of them with respect to the robot.
A multifunctional gripper, composed by 3 different end effectors (suction, magnetic and grasping) to handle the variety of products included in both end-users’ scenarios, with some restrictions. It is in line with other grippers that are being developed by other research groups.
SO 1:
Reactive grasp-planning is understood as the adaptation of an initial grasp of an object in response to sensory feedback (in our case haptic feedback) to achieve a more stable and reliable grasping. The algorithms developed allow adapting the robot motion planning in accordance with the pressure feedback from tactile sensors embedded in the grippers. It has been validated for different sensor configurations in suction cups and finger grippers’.
SO 2:
A software component that uses 3D images and deep learning models has been developed to estimate the best grasping points for objects, without the need for ad-hoc configuration.
SO 3:
A mosaic creation algorithm has been designed and implemented to handle the problem of stackable and non-stackable parts packaging configuration. The software takes as input the characteristics of the part, the actual way it has been grasped and the current status of the destination box where it has to be placed.
SO 4:
An algorithm to optimize a cost function that takes into account the human 3D occupancy map to off-line plan paths that reduce the probably of interference with the operator has been developed.
SO5:
It has been developed a system that combines machine vision and deep learning to monitor the shared human – robot working environment. It has been included the detection of up to 18 different keypoints, and the relative distance of them with respect to the robot.
Two prototypes have been developed and validated at TRL6 in certain conditions. They respond to the two types of logistic scenarios proposed by ULMA (ASR based order preparation and order return) and TOFAS (order preparation in industrial setting without ASR).
The key challenge is the manipulation of multiple parts without ad-hoc configuration and using ‘universal’ grasping tools that allow tackling new parts and unexpected situations (such as dropped parts).
In terms of performance, we have to conclude that, currently, PICKPLACE (nor any other system in the market) cannot compete with human performance in picking and placing parts in highly variable logistic scenarios such as those depicted in PICKPLACE. In fact, humans combine our perception and cognition abilities with our hands’ dexterity:
• To take multiple parts at the same time
• To adapt to almost any geometry
• To adjust the force exerted to the object
• To access areas that the robot cannot reach
• To create mosaics in a very efficient way (e.g. pushing parts to remove any free space).
• Moreover, the placement of an object is not limited by the way it has been picked (as it is the case in PICKPLACE) because, thanks to our two hands we can handover the part and take it in the more appropriate way for the place operation. In this way it is possible to take advantage of any free space.
The results achieved in PICKPLACE cannot be compared, either, with industrial bin-picking systems that allow an ad-hoc configuration of each part reference and use specific grippers fore each part geometry.
There are some relevant benefits:
• Elimination of errors. With a proper setup, it is possible to eliminate most common errors (wrong product or wrong number of items)
• Improvement of working conditions, such as repetitive movements, ‘high’ payload parts manipulation or non-ergonomic postures are avoided (reduced).
A combination of human pickers and robots could be a good option for most applications: robots handle those automatable products and humans take care of the rest. For that, it would be necessary to develop new warehouse management systems that take into consideration the capacities of both actors (human and robots), with a smart division of tasks, and a redesigned workflow and layout, can lead to an industrially feasible solution.
The key challenge is the manipulation of multiple parts without ad-hoc configuration and using ‘universal’ grasping tools that allow tackling new parts and unexpected situations (such as dropped parts).
In terms of performance, we have to conclude that, currently, PICKPLACE (nor any other system in the market) cannot compete with human performance in picking and placing parts in highly variable logistic scenarios such as those depicted in PICKPLACE. In fact, humans combine our perception and cognition abilities with our hands’ dexterity:
• To take multiple parts at the same time
• To adapt to almost any geometry
• To adjust the force exerted to the object
• To access areas that the robot cannot reach
• To create mosaics in a very efficient way (e.g. pushing parts to remove any free space).
• Moreover, the placement of an object is not limited by the way it has been picked (as it is the case in PICKPLACE) because, thanks to our two hands we can handover the part and take it in the more appropriate way for the place operation. In this way it is possible to take advantage of any free space.
The results achieved in PICKPLACE cannot be compared, either, with industrial bin-picking systems that allow an ad-hoc configuration of each part reference and use specific grippers fore each part geometry.
There are some relevant benefits:
• Elimination of errors. With a proper setup, it is possible to eliminate most common errors (wrong product or wrong number of items)
• Improvement of working conditions, such as repetitive movements, ‘high’ payload parts manipulation or non-ergonomic postures are avoided (reduced).
A combination of human pickers and robots could be a good option for most applications: robots handle those automatable products and humans take care of the rest. For that, it would be necessary to develop new warehouse management systems that take into consideration the capacities of both actors (human and robots), with a smart division of tasks, and a redesigned workflow and layout, can lead to an industrially feasible solution.