Final Report Summary - PRACE (The Productive Robot Apprentice)
Executive Summary:
1 Publishable summary
Driven by the trend to a more and more customer specific production the boundary conditions for assembly automation have changed significantly. As the systems available on the market cannot cover this extreme flexibility towards weekly changing applications a new robot system concept was developed within PRACE. An important requirement is the ability to train the robot system with worker skill fast and intuitive.
The PRACE concept basically relies on robot learning by demonstration. We compare the robot learning to a master-apprentice-relation-ship. There, a master teaches an apprentice by instructing certain skills by demonstration. The apprentice watches the actions and effects to categorize this newly gathered knowledge into his knowledge base. Then, while applying this new skill, the master corrects the execution by refining the experience. This loop is iterated until the master is satisfied with the result.
To compensate the offsets between the underlying models and tolerances in pallets and parts a tactile lightweight gripper was developed. The gripper is able to compensate offsets in the range of a few millimeters by a passive compliance, so that fast cycle times and robust assembly processes could be achieved.
Another important aspect of the PRACE robot system is the operation without safeguards to reach the target of fast setup times. Operation without safeguards however limits the maximum robot velocity. To remain competitive with the human worker a dual-armed robot approach is followed to reach a similar working output as the human worker by modest robot velocities.
With the combination of dual-armed manipulation and a mobile platform to provide local mobility within the working place, basically new application tasks may be now automated economically by this new system approach. Using a modular approach the PRACE system can even be recombined to use only parts of the robot system for dedicated applications, i.e. using only a single arm or using the system without mobility.
Different assembly use cases were defined as test environment of the PRACE concept. At end of the project a successful evaluation of the dual arm PRACE demonstrator in a BOSCH diesel plant finalized the project.
Project Context and Objectives:
1.1 Summary description of work and results
1.1.1 WP1 – Workflow Refinement
To interact with a Robot Apprentice a key element is the development of an abstraction level that is both intuitive understandable by the operator and offers on the other hand the complexity and granularity to describe all relevant parameters which are necessary from the system point of view to derive executable control programs.
It could be proved that all use cases can be described by recombination of 10 basic classes. These so called AUAs (assembly unit actions) describe hardware independent basic segments of assembly tasks, like “pick part” or “transport part”. On that level of abstraction it seems reasonable that a non-expert can intuitively understand assembly tasks.
However it is obvious, that beyond that simple description language more layers are necessary to fill the gap between the task description language (AUA) and the executable code which runs on the control system. Therefore conceptual work has been finished to obtain an overall data structure to describe the AUAs in all levels. On machine level so called primitives describe abilities of the system, which are hardware specific. The primitives are integrated in the high level AUAs which represent to the user the machine independent skill level of the PRACE abilities.
Each AUA contains a set of parameters which define the behavior of the AUA under execution mode. Therefore each AUA has to be filled in the learning cycle by either parameter the user can key in directly or by demonstration of example movements.
In automatic mode a task planner is responsible to schedule the single AUAs in the right sequence. Central mechanism is a state machine to process pre- and post conditions and to guarantee the processing of the primitives according to the skill description of the operator.
1.1.2 WP2 -Tracking and Recording Movements and Actions in Manual Assembly Processes
To perform a PRACE learning cycle the operator has to describe the desired task by combining the AUAs representing the capabilities of the system. He can use either existing skills that are modified or start from the scratch. In any case each AUA has to be adapted to the special boundary conditions of the current application. Focused on the underlying data structures the learning cycle has the target to determine the application specific parameters of each single AUA.
To do that parameterization two steps have been suggested. In the first step a human builds an AUA tree by combining primitives and existing AUA´s on a graphical user interface. In the second step the robot system is utilized in combination with tracking devices to parameterize the remaining parameters of the task. Further, parameters can be learned from kinesthetic teaching approaches by physical manipulate the robot manipulator. Regarding the required sensor systems the hardware setup has been specified.
1. A robot manipulator: This manipulator can be either a one or two armed manipulator.
If the robots have to do assembly some kind of force/torque sensor is necessary (either integrated into the manipulator or as an external sensor)
2. A tool camera: The PRACE system makes use of a tool mounted camera to recognize work pieces and calculate where to grasp the work piece.
3. A scene camera: The PRACE system utilizing one or possibly several 3D scene cameras to monitoring the human operator and the objects in the scene under the teaching phase, make rough estimates of work piece locations etc.
4. A gripper: The PRACE System utilizes a compliant gripper to compensate work piece tolerances and sensor inaccuracies.
5. A lightweight human machine interface: In order to be able to program and run the robot programs the PRACE system consists of a tablet or another lightweight HMI interface.
6. A tracking camera system: A handle with mounted gripper is tracked and thus used to record approximate 6D positions in the scene.
7. For precise requirements a teach-in-handle has been developed, which consists of a robot gripper, a handle for manual guidance by the operator and location elements to track the position of the handle in the learning process.
1.1.3 WP3 - Skill-Supporting End-Effectors
For the gripping process a lightweight gripper (250g) was developed. Its weight is under the limited payload range of the ABB dual arm robot (500g per arm) and integrates all the functionalities for tactile assembly processes. The benefit of tactile assembly is a significant improvement of stability and cycle time compared to state of the art solutions basing on vision technologies with the known limitations of poor detection results for metal parts or obscuring of camera by the gripped object.
Using rapid prototyping technology some initial concept studies have been performed and tested in lab. The actual state contains a pneumatic driven gripper actor. The actor may be mounted to a lightweight Z- or X-Y-compliance. As sensors small hall sensors are integrated to measure the actual values of the deflection of the compliance module. It could be proved that offsets can be compensated within 1s in the range of 4 mm in X-Y-direction and 5 mm in Z- direction.
The use cases have been examined to get a classification of the tactile assembly strategies. Basing on that work the structure of the tactile AUAs has been worked out and the set of parameters has been defined that are contained in the AUA.
1.1.4 WP4 – Robot Capabilities
To assess the safety aspects of the PRACE system a risk analysis of all subsystems has been carried out and consequences on the system concept were derived. The SW architecture is the central element of the PRACE system. It has to control both the perception and learning cycle of the system as well as the robot executing system regarding safety aspects, as PRACE is operated without safeguards. The result of the learning process can be visualized to the operator by the real robot system as well as by simulation. All learned knowledge is stored in a knowledge data base so that a transfer to other robot systems is possible. Innovative features of the SW architecture are:
• High level of robot cognition
The robot has a high level of cognitive capabilities that are implemented on many levels of the perception pipeline (from low level sensor fusion to high level information fusion) and provides it with the ability to act intelligently. The cognitive capabilities enhance the robot’s ability to understand both the tasks the robot has to execute as well as the environment it executes within.
• Mixed initiative
The robot will implement the concept of mixed initiative. In mixed initiative communication between the robot and the user is bidirectional meaning the user can interact with the robot as well as the robot can ask the user for clarification or to provide information. This is especially relevant in relation to error handling.
• Innovative robot skills
One of the main benefits gained by using robot skills as defined in PRACE is the possibility to define and execute system independent AUAs, which allows for skill-reuse across different hardware platforms.
• Intuitive teaching
The robot will have a user interface from where the user can instruct the robot to perform tasks. The user interface will be designed with special focus on providing an easy to use interface that does not require substantial training (other than an introductionary course).
• Real time motion planner
The robot will be working in a dynamic environment and therefore the robot needs to adapt to changes in real time. The real time motion planner will constantly get information about the environment and stop the robot or, if possible, move around an obstacle to avoid collision.
Two demonstrators at IPA and LUND have been built up. They consist of the mobile iCOB-Platform and the ABB dual arm robot YuMi.
1.1.5 WP5 – Integration into application
To collect the requirements for the PRACE system an initial investigation of thirteen manual working tasks in assembly have been examined. Typical operations are palletizing, testing, packaging, assembly, machine loading and part feeding. Typical workpiece size is in the range of smaller than 100...200 mm and a weight of less than 100g. As development guideline the system specification has been elaborated.
Furthermore the outcomes of WP1-WP4 have been integrated into two demonstration platforms. The first demonstrator focuses on haptic interfaces for dual arm robots. The second demonstrator, a full dual arm mobile manipulation system, was used for the Automatica Fair demonstration and the field tests. It incorporated the PRACE application development methodologies, the developed motion capturing system, compliant grasping system and a safe mobile manipulation controller.
1.1.6 WP6 – Test and demonstration
Basing on the use cases the plant testing scenario has been determined. By mapping the occurrence of the innovative PRACE features on the use cases a prioritization had been done. The preferred use case was the so called “needle scenario” where Diesel parts have to be transferred from a transport pallet into a washing tray and vice versa.
Towards end of project a two weeks field test was executed in a Diesel plant from BOSCH. The needle picking scenario was implemented on site with the different teach-in and description methodologies that have been developed within the PRACE project. The performance, feasibility and development were evaluated under real life factory conditions. The plant testing could be successfully finalized. The plant manager was convinced by the PRACE concept approach. As decision two prototype cells with the ABB YuMi robot will be adapted on a use case quite similar to the needle use case. Start of production in the plant Feuerbach is targeted for mid 2015.
1.1.7 WP7 – Exploitation and Dissemination
The dissemination activities have been continued with various scientific publications and workshop contributions. As the installation of the PRACE system was completed on 5/2014, the AUTOMATICA 2014 fair in Munich had been chosen for the first public demonstration of the PRACE prototype system. The AUTOMATICA as a lead fair for robotics supported an optimal dissemination due to the large auditorium consisting of decision staff members from the target branches.
To start the exploitation activities all partners contributed the expected exploitable results of PRACE. For the project actually seven exploitable results have been identified, ranging from system components to methods and algorithms. Two more results are currently under negotiation between LUND and ABB. The final PUD is documenting the current status concerning the market boundary conditions and the FGIP aspects.
A website has been established, describing the vision of PRACE, the workpackages and the consortium partners. It also contains the non-confidential deliverables, video clips and disseminations.
Project Results:
1 Work package results and achievements during the project
1.1 WP1 – Workflow Refinement
1.1.1 Task 1.1 Specification of Assembly Unit Actions
To identify representative AUAs the Bosch use cases were investigated. For that reason concrete but hardware independent AUAs were defined and interconnected to perform the wanted tasks. Each AUA represents an intuitive understandable process. The resulting AUA-Chart (in state chart or tabular form) therefore represents typical use-cases. As a result of this investigation ten classes of AUAs (or highlevel AUAs) could be distinct and scored by the number of their appearance. The most prominent AUAs are “Pick part”, “Transport part” and “Place part”. More AUAs are needed to fulfill the requirements of all tasks but as a conclusion the level of abstraction is reasonable to allow non-expert user to intuitively understand the task process.
1.1.2 Task 1.2 Analysis and design of the Assembly Unit Actions
The concept of AUAs from task 1.1 was further concretized. To widen the reusability of AUAs more levels of abstraction were taken into account. However the levels are conceptual equivalent. The visibility by any user may differ by the specific level of expertise (User, Operator, Expert).
E.g. “Pick part”-AUA as seen by a user may consist of one “Move Robot”-AUA followed by a “Close Gripper”-AUA. These two more specific AUAs (called primitives) represent a level that an operator can use to create more specific AUA sequences e.g. if more complicated trajectories of the robot arm movement is necessary. However for an expert each of the previously described primitives may consist of at least one hardware related implementation to obtain the goals represented by the primitives.
Primitives basically consist of two components: On the AUA structure side an AUA interface which makes it possible to use primitives as low-level AUAs, on the hardware side the hardware specific code to execute the desired action. The communication between these components is also located inside the primitive.
Regardless of the level of the AUA certain information must be included into any AUA representation.
To allow runtime checking whether any defined task can be performed preconditions in terms of a necessary environmental state or availability of needed devices and a representation of postconditions (environmental changes due to the execution of this AUA) must be available. Furthermore every AUA will in general need some parameters for concrete execution.
1.1.3 Task 1.3: Conceptualisation of Assembly Unit Actions
For implementation and realization purposes the concepts worked out in task 1.1 and task 1.2 must be conceptualized.
The overall sequence of AUAs representing one specific job of the robotic system can be represented as concurrent state-charts (e.g. SFC). The different abstraction levels can be taken into account as hierarchical concurrent state-charts. Therefore every AUA can be represented as one state-chart with at least one state. A specific robotic job results in a hierarchical concatenation of several AUAs.
To allow automatic semantic checking, as well as logical checking if an execution is possible during runtime each AUA (independent of its level of abstraction) must provide preconditions, which define the necessary state of the environment before the specific AUA can be executed. E.g. before placing a workpiece, some workpieces must be grasped first.
To check certain sequences of AUAs and ensure that due to the order all necessary preconditions can be met, it is also essential to describe postconditions. Postconditions may be distinguished to two kinds, namely effects and outputs. Outputs in this terminology are results by one AUA that might be used as a parameter enabling one following AUA to execute (E.g. localization of a workpiece by a vision system and forwarding the coordinates to a position correction of the robot arm).
Effects are environmental changes that result from the manipulation caused by the execution of an AUA. After a faultless execution the effect normally reflects the reason why an AUA should be executed.
By including all pre- and postconditions two major benefits can be stated:
First, AUAs are interchangeable if a) their effects are identical (their effects are entailing
preconditions of the next AUA) and b) their preconditions are entailed by effects of their predecessors and the constraints of the world (e.g. availability of resources, like sensors), and secondly by setting up an ontology to represent all relationships, an automatic planning algorithm may find a certain sequence to achieve predefined goals (wanted effects) by concatenating several AUAs.
1.1.4 Task 1.4: Synthesis of executable code
In order to connect the abstract and platform independent AUA’s and the platform dependent execution primitives of the robot, interfaces have been specified. These interfaces encapsulate the platform dependent configuration and behavior. After the specification of the AUA sequence by the end user the AUA’s are connected to trigger the primitives by these interfaces. After this process an actual execution of the AUA sequence is possible with the robot.
Following the concepts developed in task 1.3 four distinct steps will be performed to sequentially concretize generic AUAs, to applicable Primitives and finally to executable programs represented in state charts of primitives with conditioned transitions.
In a first step a generic AUA is interconnected to a component description of the device the AUA shall be perform by. Thereby a primitive of AUA description is generated. A next step needs the user to choose of define a task description of the sequence or concatenation of generic AUAs. It should be emphasized that both, the first and the second step do not need any preceding computational action or information gathering. Therefore these steps do not necessarily need to be performed after each other, but in an arbitrary order or in parallel. However after the above steps were performed the third step uses both the AUA description or primitive (representing a generic AUA concretized with one specific device) and the task description provided by the user input, to generate an executable state chart. The generated state chart is finally handed to a dedicated executor, performing and controlling the state transitions and action execution. Depending on the type of used device, its possibilities for communication and needed performance level and the AUA itself, the concrete execution of different primitives can be coordinated triggers using ROS services or messages, execution of native machine code on dedicated controllers (e.g. the robot controller) or direct (inline) execution of specific commands during the state evaluation.
By pure realization of the above approach all possible degree of freedom is granted to the user. However convenient for advanced skilled operators and developers, it was found unsuitable for every day factory realization. For this reason an additional approach was integrated in the overall synthesis. Using the tool BRIDE (more on BRIDE in description of workpackages 5 and 6), a developer is able to create and encapsulate higher level primitives by concatenation and sequencing of lower level primitives. The created primitive is fully encapsulated and integrated into the overall synthesis scheme between step one and step three. For example a developer can predefine e.g. a pickup primitive. The pickup primitive is in fact a simple sequence of lower level primitives like move and close gripper. From a naïve point of view such a sequence can be easily create by the user himself, however it showed very intuitive to provide the end-user with this functionality. Thus, the user can choose this functionality and handle it as it would be only one single AUA. By closer observation the integration of predefined higher level primitive can a multi-system realization of hierarchical state charts.
1.1.5 Task 1.5: Specific dynamic simulation
A simulation environment based on the RobWork off-line robot simulation software package have been developed and integrated in the PRACE framework. The simulation environment has formed the basis for the motion planning results described under Task 1.6 and in Deliverable D1.2.
1.1.6 Task 1.6: Offline/online motion planning
The integration of Motion Planning in the PRACE project is studied and reported in Deliverable D1.2. As hardware platform for the PRACE project, the two armed robot FRIDA from ABB has been chosen. The two arms will not be used for co-operation but for speeding up the process by doing tasks in parallel. So for the Bosch needle case the task will be to move all needles from one tray to another using both arms to move needles independently.
The use of RGB-D sensors (such as Kinect© or Asus Xtion©) for motion planning is also studied in deliverable D1.2. Such sensors have several limitations. They have a too low accuracy (worse than 5mm at 1m) for fine obstacles detection, and as shown in Figure 17, they badly reconstruct some of the PRACE workbench object (transparent plastic, metal grid). The use of a single sensor (i.e. a single viewpoint on the scene) also produces occlusions.
However, such sensor offer a real-time update of the scene (colorized 3D cloud at 30Hz), they are cheap and easy to use in a workbench (Asus Xtion© sensor only requires an USB plug). The final outcome of the experiments with these sensors is that they can be used for rough estimation of position and obstacle detection, but not for precise motion planning for manipulation task close to obstacles.
1.1.7 Task 1.7: Visualisation
Motion planning and simulation results are necessary to visualize in order to evaluate performance and develop well performing algorithms. To support this development, visualization of the FRIDA robot, PRACE work pieces and tools (grippers, compliance unit, etc.) have been implemented into the ROS visualization tool RViz. The visualization can display both real robots or simulated variants without any modifications and additional input sources such as 3D models, live camera and point-cloud information can be imbedded directly.
3D and augmented reality visualization tests were performed, to evaluate the possible ways to visualize additional information on a given 3D scene. Those visualizations principle could be useful for:
- The motion planner / collision checker
- The learning machine demonstration of the learnt information
Next steps in T1.7 were to investigate which visualization strategies should be applied to present motion planning, instruction and system status results to the operator. Which visualization modalities should be applied, what information should be available and which technology should be applied will be important questions in this research.
Visualization of the robot movement has been initially integrated in the end-user interface on the DTI testbed. Next steps are to indentify which elements could be added to the visualization to support the end-user in following and being familiar with the flow of the robot system.
For simple tasks or tasks that be realized by pure sequencing of available AUAs or higher level primitives (as discussed in T1.4) an html-based front-end was created for the facility staff support. This lightweight gui reflects only a small variety of the available possibilities of the task specification, but still was capable of allowing all use-cases to be created. Due to its html implementation it furthermore allows to enter the task creation gui using any browser, even to manipulate a task in parallel from more than one device.
1.2 WP2 -Tracking and Recording Movements and Actions in Manual Assembly Processes
Work package 2 has a core focus on instruction of the robot system by input from the so-called Master. Using a user interface and a number of sensor modalities, the instructions from the Master must be transformed into a sequence of AUAs, and the parameters of each AUA must be tuned and adapted to fit the individual requirements of the process. The parameter adaptions can be capturing forces applied during assembly e.g. when performing a snap-fit or counting the number of turns when screwing.
While fully acknowledging that the Master is the expert on the individual processes which must be automated, the focus of WP2 is not on fully mimicking the actions of the master instructor, but has chosen a more object centric approach of task instruction. Sensor modalities are preferably used to track the objects for assembly and record individual parameters such as forces, with the purpose of extracting information on how the individual objects are assembled. Detailed information and interaction with the robotic system will be performed using touch-based tablets as input modalities.
Human centric sensing, such as motion capturing, naturally gathers information on how the humans performs parts of the process and will be utilized where the modality fits into the instruction work-flow and where adequate accuracy can be obtained.
At the current state in WP2, a number of individual results have been obtained. Using sensor gloves and fingertip force sensors to capture input from the Master instructor, the measurement of forces in grips and hand positions are in its final states. A task of combining these measurements using sensor fusion are just starting to achieve results useful for robot instruction. Excellent work is also ongoing on extraction of human motion using the Primesense based depth cameras, but efforts in converting these measurements into robot instruction are yet to be started.
The results on this topic are for most parts described in deliverable D2.1 which describes the obtainable data using the investigated sensor modalities and conceptual work on the PRACE task-instruction methodology.
In the project period M12 till M18, focus in WP2 has been towards getting a first functional implementation of the instruction concept ready for evaluation of the concept and the various modalities. The results are documented in D2.2.
1.2.1 Task 2.1 Evaluation of appropriate sensor devices
Task 2.1 focuses on investigation into a number of sensor modalities useful for task instruction in the PRACE system.
The sensors under current investigation are:
• Flexible force-sensors for measurement of forces applied at fingertips.
• Accelerometer-based sensor gloves for tracking of finger positions.
• Touch-based tablet computer for instructions.
• Singular and stereo cameras for object pose estimation.
Evaluation of the sensor modalities are progressing as planned. Results obtained using the force-sensors and sensor gloves are documented in deliverable D2.1.
The short conclusion of these investigations are, that data capturing gloves can only be used to capture simple information e.g. like single forces or orientations. But the actual identification of finger positions are not possible to an excend where the results are useful for robot instruction. As a result of this, the use of human fingers and tracking gloves have been discarded in favor for using a tracking handle who mimics the robot tool and allows a much better geometric tracking.
1.2.2 Task 2.2 Data capturing and tracking mechanisms
Task 2.2 focuses on the data acquisition, initially dedicated to human body tracking. As the task instruction concept of PRACE have re-focused to be object centric, the last part of this task reflects a shift of approach towards object-oriented technologies, including teach-in handle tracking and robot-tool camera technologies.
The deliverable D2.1 contains some results of this task in respect to human centric capturing and tracking. A state of the art was first conducted to validate the choice of the Kinect as a human body sensor compatible with PRACE project. Software was also implemented to access the Kinect data (RGB, depth, users’ segmentation and skeleton), and display it in a 3D viewer. At last, performances of the skeleton tracker were tested on simple criteria (temporal, spatial, validity, accuracy, scene artifacts, etc...).
The multi-cameras (and/or multi-kinects) calibration software has been developed and performs intrinsic and extrinsic calibration. The design aim of the calibration software is to be generic for different setups of vision sensors. It shall for instance be able to calibrate two cameras, three Kinect, a mix of these setup, etc. In practice, we have experimented that this generic feature comes at the cost of usability, and an application specific to a given setup is preferred by users that need to often calibrate the same system. Therefore, the software has been used by Magellium to calibrate the setup of the Teach-In Handle and it has been adapted to calibrate the validation setup of human body tracking composed of a Motion Capture system localized with respect to the Kinect system.
A ground-truth comparison mean was developed, in collaboration with the LAAS laboratory, at Toulouse. It is based on a reference system, a Motion Capture system composed of 10 IR Sensors and passive markers, giving the human tracking reference (the ground-truth). An acquisition campaign dedicated to absolute accuracy evaluation was performed and processed.
This acquisition gives a useful base for human body tracking algorithms validation. 3 Kinects© were co-localized, and simultaneously used to track a moving human. As shown in Figure 24, those 3 skeleton tracking results were compared with the ground truth skeleton.
The real-time synchronization and the data handling to be performed in this task have been postponed to task 2.4 as the sensors to connect and the data to handle depend on each learning (or execution) method.
As the human body tracking was excluded from PRACE learning principle (as a WP2 risk reduction decision), has been involved in the Teach-In Handle tracking. As shown in Figure 25, the teach-in handle consists in the robot gripper, mounted on a specific handle. This teach-in handle is manipulated by the human, to show specific actions to the learning machine, with a robot representative tool. The goal of this task is to be able to locate the teach-in handle with respect to the workbench.
The study results are given in deliverable D2.2. They consist in a state of the art concerning the existing off-the-shelf sensors and software, and in the development of a first teach-in-handle tracking demonstrator. The first demonstrator which is described in the Deliverable D2.2 was tracked with a 3D-marker composed of four colored balls. Based on the results of testing and evaluating the accuracy and stability of marker tracking, the number of colored balls increased to five during the project. Additional to this optimization process the teach-in-handle gets illuminated colored balls with infrared LEDs in each ball, an electrical gripper and a wireless connection with a Sony Move controller to communicate the status of the gripper with the master controller. The teach-in handle based learning method, dedicated to specific actions or area teaching is handled in task 2.3.
When executing tasks roughly instructed by humans and where work pieces have been placed by human operators, it will be impossible to execute purely off-line programs. So solve this issue, sensors must be used at execution to compensate for uncertainty and process variation.
To refine positions of objects at both instruction and execution, a tool-mounted vision system has been developed.
Using the Halcon vision library, 2D and 3D vision algorithms have been applied to efficiently detect objects to grasp and the pose of the destination objects. Using a 2D matching algorithm, objects can reliably be detected by their 2D shape as in Figure 27, with the only constraint that the height between robot and object should be known.
When a number of unique features on the object can be identified and their physical dimensions can be measured, a full 6D pose of objects can be estimated as in where the Danfoss heat sink is detected using four screw holes, which allows detection of all six degrees of freedom with a high accuracy.
The needles are also fairly straight forward to detect using 2D matching. The 2D matching approach also makes it possible to detect multiple instances of any given object and return the best match. This allows the robot to keep picking needles in a sequence until no further objects can be detected.
As the FRIDA robot platform only have a payload of 500 g total, a significantly lighter tool-camera platform is needed. Furthermore, FRIDA only have the possibility of running 4 signal conductors through the arms, which are needed for tactile sensing in the gripper. To accommodate for this challenge a lightweight wireless camera solution is developed, which suits the FRIDA Platform. As this is primarily an integration task, it is further elaborated under T4.4
1.2.3 Task 2.3 Motion and action extraction from the raw data
The activities of this task are focused towards extracting action information useful for parameterization of AUAs from sensor sources who stream sequences of data. Furthermore, this task involves the development of methodologies to control and represent the process flow of tasks that contain information of this type.
A kinesthetic learning system has been implemented using the Universal Robots UR5 robot platform. The joint-force measurements and internal control system of the UR5 robot support manual guidance of the robot to learn positions and trajectories. This learning methodology has been implemented to parameterize AUAs with end-positions and trajectories for motion commands.
In order to execute most robot tasks, process control more advanced than sequential actions are needed. Most tasks can be represented by having conditional sections (If / if else) and conditional loops (e.g. loop until no more needles are detected). Although these concepts might be easy to understand for anyone with a programming background, they are complicated to visualize for operators without programming experience.
Robot tasks can also be built using several higher level skills, which internally are based on one or more AUAs.
Furthermore the teach-in handle based learning method will be implemented in this task. This learning method will consist in identifying information from the human action, if possible using the scene camera (or a scene Kinect©). The main reasonable data to identify are:
- The workbench main plane,
- The pick pallet area,
- The place pallet area.
The use of the teach-in handle for precise position and orientation teaching is still to be studied. Indeed, this may require perceiving the teach-in handle from a close camera, as the gripper one.
1.2.4 Task 2.4 Data merging algorithms
The purpose of Task 2.4 is to merge the data contribution from the different instruction modalities developed by the different partners of PRACE.
A data fusion algorithm was implemented for the human body tracking. Indeed, the multi skeleton tracker developed for PRACE uses several Kinects sensors to refine the position of a unique resulting skeleton. Delayed logic is thus involved to find the more reliable skeleton between raw skeleton of each sensor, and a Kalman filter predicted one. Our approach combines Viterbi and Kalman, and runs at 20Hz (in a mono-thread CPU application on an Intel Core i7 2760Q – 2.4Ghz). This algorithm was described in (Masse, Lerasle, Devy, Monin, Lefebvre, & Mas, 2013). The proportion of acquisitions whose joints were closer than a distance from ground truth (50, 100, 150 and 200 mm, those are cumulative).
The accuracy was estimated for 5 given skeleton joints (head, left and right hands, left and right foots), and for several skeleton estimators:
- the 3 raw Kinect acquisitions (sensor 1, sensor 2 and sensor 3),
- a virtual sensor which would return the best nodes of the 3 raw sensors (sensor best5),
- a Kalman result (0-order, no speed in state),
- a Viterbi result.
The data fusion of our approach increases the reliability of the system, as fewer frames are outside the outstanding distance of 200mm from ground truth, and it also increases its accuracy, as the amount of positions lower than 50mm mark increases. The analysis of the results of this task has oriented future works towards the fusion of data coming from human tracking and object tracking.
1.2.5 Task 2.5 Set of design rules for further Assembly Unit Actions
According to the definitions in D1.1 a skill in the PRACE system is a capability which is system independent and therefore can be executed on two different systems. Each skill can be realized by system primitives and/or be a composition of other skills. Assembly Unit Actions (AUAs) are skills at a user-defined, assembly-relevant level.
When a robot in PRACE needs new capabilities it is required that the needed primitives is implemented at the specific robot system. This implementation has to be conducted by an AUA design expert which is able to program new primitives from scratch.
To create new primitives which can be executed at the PRACE system the overall primitive has to obey the inputs/output table.
The designer needs to list the required parameter needed to be able to execute the primitive. The data type of each parameter can be all defined types in the system like int, strings or specified data types like 6D pose or a trajectory. Furthermore, the pre-/postcondition has to be constructed as logical conditions. Conditions can vary from simple boolean conditions which are evaluated as true or false or more complex evaluation like robot states or other more complex data structures.
An example on a precondition for a sensor primitive for locating objects with a tool mounted camera, in a palette could be a condition that evaluates whether the robot is in the correct state above the pallet or not before executing the primitive. Simpler preconditions could be that the gripper has to be open before closing it with a closeGripper primitive.
In Deliverable 2.3 guidelines for constructing new primitives are presented. The guidelines are meant as a listing of required knowledge that an AUA design expert needs before starting programming the primitive. The guidelines try to make the developer thinking the right thoughts to determine the correct, pre-/post conditions, parameters, outputs and effects. Following these guidelines will help the developer identify missing knowledge about the functionality of the primitive. If the developer cannot answer the questions or list data types etc. the designer need knowledge regarding the required functionality of the primitive
1.2.6 Task 2.6 Learning routines
Learning tasks directly from the Master instructor are a very efficient method of instructing robotics tasks. To capture the full workflow of a pick-and-place task, a teach-in handle based learning system were developed. Using a combination of the tracking handle and the instruction user interface, the system gave the possibility of building entire processes by using the handle for providing positions and the handle buttons to mark waypoints, as well as grasp/release points.
A full video of the demonstration can be found here:
1.3 WP3 - Skill-Supporting End-Effectors
A main target of PRACE is the fast and intuitive learning of industrial assembly tasks. A new application should run in production mode within a single day. Besides the flexibility and usability of the learning strategy another very important boundary condition is the stability and reliability of the generated movements out of the learning cycle. To guarantee this stability the PRACE system needs the ability:
a. to compensate offsets between learning cycle and execution cycle, due to inaccuracies within the sensor data generated in the learning cycle or due to general modelling inaccuracies.
b. to react on changements in the environment that typically take place in industrial environment, like deformation of pallets or tolerances/offsets in part providing.
The duty of WP3 lies in the development of a gripper which is able to detect and compensate such distortions within the assembly process. State of the art for many years has been the integration of force sensors into the gripper flange to detect such offsets in a tactile way. The problem of that approach however is the stiffness of the system. In case of contact between robot and workpiece a force is generated a necessary input for force controlled algorithms to compensate the detected offset by path modification of the robot. This approach is not suitable for industrial use, as due to the stiff construction the contact force ramps up very fast, which is crucial due to the potential damage of automotive parts with very high surface quality requirements. Therefore an approach is pursued with compliances gripping elements in PRACE.
On the other hand a compensation of the differences between learning and execution mode has to be performed. Additionally a gripper guided camera detected such offsets which are bigger than the tactile compensation abilities of the tactile gripper.
1.3.1 Task 3.1: Design principles of lightweight devices for precise assembly
In Task 3.1 the design principles of a lightweight gripper for precise assembly is demanded.
From an industrial point of view a major requirement of the gripper is the reliable operation under production environment in three shift mode. Additional to this requirement the gripper system weight with the handling part must be under 500 g which arise from the payload of one arm of the robot.
The first step was to use market available components to determine which functionalities may be integrated under the boundary condition of maximum 500 g payload of the robot. These market available components are too heavy whereon Rapid-Prototyping lightweight components were developed and built. A contrast to conventional springs is that a CT-joint on tension or pressure can be claimed.
The X-Y-compliance module is a two stage module and provides deflection in X and Y direction, one stage for each direction. This unit consists of two equal one-directional compliance units. By connecting them with a rotation offset of 90° a X-Y-compliance is realized. A linear plain bearing is mounted in the center of Z-Compliance to reduce the tilt under non-uniform load.
Due to the lightweight requirements for the gripper, the module is manufactured by rapid prototyping. The technical drawings can be found in deliverable D3.1.
In each compliance element a hall sensor is included in the design, to measure the actual deflection of the module. The sensors are easily configurable and can be adapted very well to the needed maximal deflection.
On the compliance module a market available gripper of the type MPG-plus 25 from Schunk is mounted. This gripper is characterized by its less weight and the compact form. The fingers which are attached to the gripper are also constructed to be lightweight and easily adaptable at the same time. Therefore, for each use case, different fingers built by rapid prototyping are provided.
1.3.2 Task 3.2 Development of pick and place strategies for precise assembly
The goal in this task is the development of pick and place strategies for precise assembly. The challenging part of the pick and place process is in general the place process. Due to that the development of place strategies is handled with prioritization. The robot is developed to manage work tasks generally handled by humans. Thus the strategies are derived from human behavior.
Therefore the use cases have been analyzed and place strategies have been derived. These strategies require several kinds of compliance. For each use case and strategy the desired compliance is noted. Depending on the strategy, each dimension of compliance has to be actively controlled (deflection control of the compliance module) or it can just deliver some passive compliance without any control.
Out of these use case specific strategies three general strategies are developed. As an example the strategy “cylindric peg cylindric hole” is derived from use case 1.
While running a specific strategy the robot will follow a basic path. Additionally to this path tactile events can occur. ,
As the insertion strategy is dependent on the application the three general strategies are listed with their requirements in Table 1.
Insertion Strategy Catalogue
Direct insertion Tilted insertion Spiral insertion
- Short cycle time (< 1 s)
- For own learning of positions in execution mode
- For insert position with a great tolerance range (> 1 mm) - For insert position with a small tolerance range
(> 0,1mm )
- middle cycle time - For accurate insert position with a tolerance range
(< 0,1mm )
- bad cycle time
Description of process parameters
Every place strategy requires a set of parameters. For the „cylindric peg - cylindric hole“ strategy these parameters are:
- Peg length (to gain the TCP at the peg tip)
- Peg diameter
- Hole diameter (to find the center point of the hole)
- Approach Tilt Angle
- Insertion depth
- Maximal contact force
Out of the peg length, peg diameter and the hole diameter the approach offset is derived. The insertion depth will define the end position. When this position is reached without contact forces the place task is successfully finished.
1.3.3 Task 3.3: Implementation of pick and place strategies for precise assembly
The main focus in this task was to implement the pick and place strategies which are developed in task 3.2. These strategies were implemented without recording the integrated compliance sensors. The robot follows a fixed path and once an inaccuracy in position occurs, the gripper is able to correct this inaccuracy by its compliant behavior. Thereby it is possible to implement the maximum robot speed by the insertion movement. A disadvantage is that the robot moves into safety stop if the robot clashes with the pallet because of the too large position error. All implemented strategies were tested in the lab. A useful strategy for most insertion applications at Bosch is the tilted insertion strategy which was tested during the field test at Bosch. This strategy reached a very stabile process with sufficient compensation of offset errors up to 4 mm (dependent on the application).
1.3.4 Task 3.4: Implementation of gripping devices
The gripping system was developed so that it can be easily attached to the existing tool-change system of the robot FRIDA. The robot provides on each arm one pneumatic connection and a six wire cable. The cable is connected directly with the controller of the robot and can be used for an Ethernet connection (10 Mbit). It is necessary for the PRACE-System to unplug the cable from the robot controller and connect it to the master controller of the PRACE-system. The wirers are used to transfer the signals from the analog sensor in the compliance elements. The gripper camera must be connected over WiFi, due to the few numbers of cables which goes through the robot arm. For operation of the gripper the pneumatic connection is used. The pneumatic valve is installed at the bottom of the robot. Only one pneumatic tube is necessary because the gripper is spring resettable.
In the gripper a circuit board is integrates with a connector to plug in it to the Robot connector. Additionally a DC-DC Converter is integrated which converts the normal 24 V from the robot to 5 V for the sensors and camera system.
The gripper compliances are made with the rapid prototyping process technology Fused Deposition Modeling (FDM) with the material acrylonitrile-butadiene-styren (ABS). The result of the lifetime cycle was that the X-Y-compliance breaks by a maximum deflection after about 3600 cycle and the Z-compliance by about 24000. Therefore the rapid prototyping process technology was changed to Selective Laser Sintering (SLS) with the material polyamide (PA). At over 300 000 cycles the attempt was stopped with the result that no changes can be measured of the spring characteristics of the compliance elements.
1.3.5 Task 3.5: Implementation of tactile pick and place strategies
The goal in task 3.5 is the implementation of tactile pick and place strategies. The difference to task 3.3 is that additionally the compliance sensors are used. The use of the compliance sensors provides the possibility to touch the position and orientation of the pallete to calculate the insertion position. In addition the robot can feel collisions and too large inaccuracies of the insertion position. Thereby the robot can adjust the position so automatically that there is no force influence on the work piece during the process
1.3.6 Task 3.6: Integration of bin-picking with grippers
Task 3.6 has been cancelled as a Steering Committee decision on M12 plenary meeting due to missing expertise in bin picking algorithms of the partners. In the running amendment those resources are shifted to support the setup of a second PRACE demonstrator at LUND. This activity has not been planned in the initial DoW, but it has pushed PRACE strongly forward concerning the integration and testing stage.
1.4 WP4 – Robot Capabilities
1.4.1 Task 4.1: Holistic assessment of safety aspects
The holistic assessment of safety aspects was carried out by Fraunhofer and Bosch There are separate forms created from the assessment, and the deliverable D4.1 "Analysis of safety concept" was submitted right on time. Refer to that report from further details.
1.4.2 Task 4.2: Architecture
A major effort in WP4 during the first period has been the specification of the architecture, and represented by Task 4.2 and the deliverable D4.2 both named Architecture. A major effort has been put into representing the architectural design, without getting stuck in details that the further research will clarify. For instance, both UML and SysML design tools directly lead to specification of system details while the overall design (that should enable the PRACE results) gets lost. Led by DTI, and developed through a long series of meetings and emails, less formal figures and models were developed and the deliverable reached a complete status.
One source of confusion in this task, as in this work package and the overall project, has been the relations to the results of the Rosetta project. The planned usage of the FRIDA robot was intended to go together with some of the architectural solutions from Rosetta, but due to the unclear situation with both FRIDA and ABB, that interplay has been lagging until recently.
1.4.3 Task 4.3: Mobile manipulation
During period 1 the efforts in this task were focused on the deliverable D4.3 "Hardware components", driven by BOSCH. That report outlines the usage of the FRIDA robot, mounted on a mobile platform, and how various peripherals will be added to meet the PRACE application demands. During period 2 the mobile platform and the FRIDA robot has been delivered and made available for demonstrations in Stuttgart (Bosch and IPA). Beyond the deliverable there are still issues to be worked out for the integration of mobility and manipulation. While that is mainly a topic for Tasks 4.4 and 4.5 the availability of the hardware platform (as of this task) is not quite complete, and therefore a completion of 98% is reported as the status. Part of this picture is the exit of Axelius, the entrance of ABB, and the tested setup at ULUND (including ABB sensor interfaces). The final 2% of the work was done in the second period of the project. Mainly the consolidation of the configuration of the hardware components, including final work on the modularity of the system was done. The outcome of this task is the integration of the mobile manipulation system in Task 4.4 and WP5.
1.4.4 Task 4.4: Integration of mobility and manipulation
The work on integrating mobility and manipulation started as planned late during the first period. The main efforts, by ULUND, have been on establishing the mobile platform (according to the Fraunhofer design, including the ROS-based software) and the tailoring of the FRIDA control system (based on Rosetta results for PRACE applications). In line with Task 4.3 as a result of period 2 efforts, there are demonstrator-like setups in Stuttgart (mobile platform at IPA still with UR5 arm, FRIDA robot delivered to Bosch for integration with assistance from ABB), Lund (mobile platform as in Stuttgart, preliminary mounting of FRIDA on it, but charging and networking not done) and Odense (mobile platform with UR5 arms, sufficient for testing key demonstrator scenarios).
For technical/control integration of mobility and manipulation, the sensor and motions control interfaces to the manipulation part (ABB controller) play a central role. Manipulator motions are accurately actuated with rather high bandwidth, compared to the mobility which has a bigger moving mass and a suspension system that makes the motions compliant and not fully controlled (vertically, in addition to the risk for horizontal slip motions). Hence, the mobile motions need to be fast and accurately measured, and the actual location of the robot base then needs to be feed forwarded to the robot motion control. For reasons of performance that control is after the trajectory generation of the robot and the interfaces on a very low level. The progress during period 2 has been along these lines of development, also building on the developments of the just finished Rosetta project.
In the work of WP2, tool-camera based vision systems have been investigated. The instruction and functionality have been evaluated using traditional GigE cameras on the DTI Universal Robot platform. To suit the requirements of the ABB FRIDA platform, a lightweight camera solution is integrated.
The prototype are integrated using the low-cose Raspberry Pi as platform, as illustrated in Figure 40.
The integration of mobility and manipulation in the final phase of the project mainly focused on the PRACE project demonstrator for the lab tests, the automatic fair the final factory tests. The tight integration of the rob@work 3 platform, the Frida robot and the grippers and the motion tracking system was done in multiple iteration. In the beginning of the final year the platform as shown in Figure 41 was created, incorporating all required components. Based on the experiences in the factory tests the prototype version of this integrated platform was extended with a lifting unit, an extended sensor head and a larger pneumatic tank to be ready for the factory tests.
1.4.5 Task 4.5: Control system
The final Task on Control system, with an emphasis on the safety aspects, is related to all the other tasks of this work package. While formally started, most upcoming results are not yet possible to distinguish from other results. The safety of the mobile platform (based on dual laser scanners) and the safety of the dual-arm manipulator (due to the inherently safe mechatronic design) forms a good basis for this task, but safety is not compositional (two safe subsystems does not automatically comprise a safe composed system, as Task 4.1 reports). Further analysis of this situation has been made, and no major obstacles have been found, but this task is still in its early stages.
During the development of the PRACE system and during the Lab tests multiple ways of utilizing the control system of the PRACE mobile manipulator were required. Therefore, three control concepts were implemented during the project. On the robot system, we have a controller running within the ROS middleware on a dedicated PC. This robot system integrates high level planning capabilities for collision free dual arm motion as well as the high level state machines and the interfaces to the engineering system. To execute the motion the ABB controller that is inside of the FRIDA prototype was interfaced. Figure 43shows this interfaces on the ABB side (Rapid program) and the ROS side (trajectory downloader and joint_state component). During the project, multiple improvement on the performance have been made. The interface allows additionally to the control capabilities of ROS the direct utilization of the control capabilities of the ABB controller itself as a second control mode.
Additionally an ExtCtrl control PC was integrated on the platform allowing advanced real time control applications to be performed. The ExtCtrl PC is directly connected to the FRIDA controller using the LabCom protocol. Additionally an interface between the ROS controller and the ExtCtrl PC was implemented using the RAPID connection to FRIDA. This makes it possible to utilize the real time control capabilities as a third control mode.
The implementation described allows seamless switching between the different control modalities depending on the application demands.
1.5 WP5 – Integration into application
1.5.1 Task 5.1: Use cases and requirements elicitation
In task 5.1 possible use cases were identified for the PRACE-system and were described more in detail. These are typical industrial applications which are operated today manually and could be automated with the PRACE-System.
The operations may be categorized into six operation basic classes:
- Palletizing of parts
- Testing operations (functional testing, optical inspection, type verification)
- Packaging
- Assembly operations
- Machine loading/unloading
- Part feeding
The use cases have been chosen to cover every basic operation class at least once.
Based on thirteen selected use cases emerged following results:
1.5.1.1 Workpiece properties
Most workpieces are not larger than 100...150 mm, although the maximum possible size of dedicated use cases is significant higher. Those use cases are most interesting for the two handed manipulation as a single handed manipulation with the reduced payload of FRIDA would be difficult.
1.5.1.2 Workpiece weight
Typical workpiece weight is less than 100g. That would be the basic requirement for the gripper construction, although the aspect should be regarded to handle higher weights with reduced velocity, i.e. handling of empty pallets which have a weight of 500g up to 2,5kg. It has also to be examined under which conditions the maximum payload of the FRIDA robot of 500g per hand may be exceeded, i.e. to perform seldom required operations like handling of pallets.
1.5.1.3 Workpiece material
Obvious there is no preference towards a special kind of material. However all workpieces have a stable structure, so that no change in shape has to be expected when handling the parts.
1.5.1.4 Tolerances between workpiece and pallet
Typical offset between workpiece and tray lies in the range of 1 +/- 0,5 mm.
1.5.1.5 Additional manipulation related information
Required DOF for object recognition Typical 3 DOF (x,y,rot z)
Box/pallet size 400mm x 300mm / 600mm x 400mm
Weight of box/pallet 500g – 2000g
1.5.1.6 Interfacing
Mainly binary 24V input/output required for - handshake PRACE – equipment to start local process- testing result ok/not ok
MMI user information/input is necessary to coordinate supporting user actions (changing of pallets, providing of parts, packaging of parts).
1.5.1.7 Use case properties
For developing the structure of the AUAs in WP1 a representative use case has to be defined. The use case must provide a broad and typical scenario to prove the adaptability of the AUA structure within its definition stage. Therefore the AUA relevant use case should cover as many as possible of the innovative PRACE features. To determine the AUA relevant use case the innovative PRACE features have been mapped to each use case.
1.5.2 Task 5.2: Multi-Agent based control system
Task 5.2 is entitled “Multi-Agent based control system” and focuses on the development of a Multi Agent System to be applied with the PRACE System.
The work in the task so far has been:
• Development of a suitable MAS concept that can be applied in the PRACE context that both defines the software and hardware implementation of the MAS concept
• Documentation of the developed MAS concept and preparation for discussions with partners in T5.2
In the following the partners of T5.2 will discuss the overall MAS concept as proposed by DTI. The concept has been discussed at several workshops with ULUND and the proposed MAS system concept is currently in prototype development.
After the prototype development, the main required subsystems of the MAS structure have been defined. In the integration phase of the project towards the factory tests these main subsystems have been implemented based on the ROS middleware.
1.5.3 Task 5.3: Software integration for workcell control
The implemented primitives all have defined interfaces that can be utilized using the ROS actionlib mechanisms. These mechanisms are prepared to be used by PLC’s and SCADA systems using e.g. the OPC-UA protocol stack. An additional way of integration can be done over the knowledge database server directly. New AUA sequences can be created and triggered by database access and standard web protocol interfaces that are available in any PLC, SCADA or industrial PC system.
Since a direct integration in a factory-wide work cell control system is not required for the factory tests, the integration will be left as prove of concept state. During the factory tests, the worker will be directly interacting with the robot and therefore the project focus will be on this interaction.
1.5.4 Task 5.4: Integration of simulation environment
ABB FRIDA, the mobile base, the grippers and the tracking system have been completely integrated in a working simulated scenario. Gazebo, the simulation that is part of the ROS system, has been used for this purpose. The physics simulation can simulate sensor signals from the cameras, 3D cameras and laser scanners and also simulated an environment model. For testing the scenarios of the Automatica fair and the factory test the environment was adapted to satisfy the needle scenario that is introduced in deliverable 5.2.
A simulated handle is used to also test the interaction mechanisms between the handle, the knowledge database and the AUA programming system.
As the simulation is available for all partners of the consortium tight interaction on the development during the project is possible.
1.5.5 Task 5.5 Integration of motion capture equipment
Integration of tracking equipment has been started at several of the testbeds. The BOSCH Teach-In-Handle applied to the instruction workflow teaching pallet locations on the workbench, a set of desired gripper positions and respective intermediate move actions.
For tracking the teach-in-handle, different tracking systems were evaluated:
- The teach-in handle tracking software “TIHTracker” developed by Magellium as part of WP2 using a mono camera setup for detecting the colored balls of the BOSCH handle and computing their positions in the image frame, and thus the handle’s 3D position and orientation itself.
- The new open-source Track-It-Yourself (TIY) 3D Marker Tracking Software (see also https://code.google.com/p/tiy/) using a stereo camera setup and triangulation for computing the positions of infrared light reflecting little balls in the scene (see Figure 52).
- The commercial tracking system OPTITRACK, a 6DOF optical tracking device with multiple cameras and capture software also computing positions of infrared light reflecting balls (see also http://www.naturalpoint.com/optitrack/products/v120-trio/). OPTITRACK was used as a ground truth system throughout the evaluation of the Magellium THITracker and the TIY tracking.
In order to track colored spheres using the Magellium TIHTracker, corresponding camera parameters (e.g. gain and exposure) need to be configured strongly dependent on current lighting conditions. Hence, those parameters need adjustments every time lighting conditions are changing. Furthermore, the detection rate of the system dropped significantly in the rather bright environments of industrial facilities. In order to cope with bright environments, actively illuminated colored spheres (see Figure 51) were used in connection with turned down camera exposures to have the illuminated balls to be the only color information in the image frame. In the course of extensive testing, the Magellium TIHTracker software proofs to deliver a more robust tracking performance than the TIY software. Particularly, the TIHTracker software shows less sensitivity to light perturbation (e.g. severe reflections from the surfaces of the palettes due to external IR illumination) and offers a larger workspace for the handle to be tracked in. Hence, with regard to the demonstration at AUTOMATICA 2014 fair and the field tests, the Magellium tracking software is integrated for presenting usage and functionalities of the teach-in handle.
Even though state of integration and performance of the motion capture equipment are satisfying and ready for presentation, there are still a few minor software improvements left to be implemented
in order to increase robustness and reliability of the tracker.
1.5.6 Task 5.6: Lab test
For the lab tests a dedicated room with tables similar to the Bosch factory tests has been build up at Fraunhofer IPA. The development of the robot system has been done in this environment continuously from January to August 2014. For the defined use cases multiple iterations have been done mixing development and testing.
Particularly, testing and evaluation of the motion capture equipment from Section 1.3.5.5 as well as integration of the results from WP 1 and 2 have been done in the course of the lab tests. Palletizing scenarios with different types of needles and different types of traces (see Figure 54) were implemented and tested considering control modalities from Section 1.3.4.5.
With these continuous lab tests large improvements have been possible towards the AUTOMATICA 2014 demo and the field tests.
1.6 WP6 – Test and demonstration
1.6.1 Task 6.1: Specification of scenarios
To determine the field testing scenario the use cases have been ranked according to the following requirements:
- (a) Mapping of all important innovative capabilities of the PRACE system
- (b) Boundary conditions
- (c) System properties fit to PRACE system design
- (d) Agreement of partner from production site who to contribute and support the testing stage
In D5.1 a prioritization of the thirteen presented use cases has been done already to determine which use cases require most of the new PRACE features. The target is to identify those use cases that require a broad set of the AUAs (Assembly Unit Actions). Therefore an evaluation had been done by comparing the amount of the new PRACE features that are mapped in each application.
After taking all those aspects into account the top ranked use case had been the fast palletizing use case that is located in a big Bosch plant near Stuttgart (D). In the initial state of the project a meeting had been arranged at that production site, so all partners are familiar with the use case and the boundary conditions of that application. All those partners who are building up demonstrators of subsystems have been provided with sample parts and pallets for early testing.
As the chosen scenario requires several parallel running manual working places in three shift mode there is a high interest from the production site towards a more economic solution at a high cost location like Germany. Presently there are investigations running to automate this application with a single arm robot system. However this approach is critical due to the high manual output as the worker can perform the double amount of handling operations compared a single arm robot (assuming the same velocity).
Another critical aspect of a single arm solution is the missing ability of handling the emptied/filled pallet. In that case the worker has to periodically change pallets to keep the system operative. Especially when trying to bridge the worker breaks this is a negative aspectsaspect towards high productivity of the single arm robotic solution.
In project year two the integration and testing of the PRACE demonstrator takes places. In that period the application planning for the field testing period will have to be done more in detail and matched with the boundary conditions given by the production site.
1.6.2 Task 6.2: Specification of field test and benchmarks
The aim of the Task 6.2 is to define the metrics and criteria for the field tests that will be executed at a BOSCH factory at the end of the PRACE project. The robot system is developed for a number of scenarios as described in deliverable D5.1 and a selection of these will be executed during the field tests (see task D6.1). Therefore, a number of factors have been specified in cooperation with BOSCH to make field test measurable and comparable to the state of the art in production.
Following list displays all benchmark parameters which will be surveyed later in the field tests.
1.6.3 Task 6.3: Field test
The field tests that have been specified in the DOW were extended during the end of the project. Additionally to the foreseen field tests a demonstrator has been built up to show the project results on the Automatica industrial fair in Munich. During 3. - 6. June 2014 PRACE had a space on the Fraunhofer booth at Automatica (see Figure 56). The dual arm mobile manipulation capabilities were shown as well as the teach-in functionalities. As this demonstration was scheduled 3 months before the actual field tests much improvement from all PRACE WP’s could already be integrated and helpful experiences have been made in order to prepare the actual field test successfully..
In August 2014 two weeks of factory tests have been made as defined in the DOW (see detailed schedule in Figure 57: Time plan of field tests at Bosch plant Feuerbach). During that time, the PRACE demonstrator was used in production at the Bosch plant in Feuerbach, Stuttgart. A needle picking was implemented with the different process methodologies that have been developed within the PRACE project on site. The performance, feasibility and development was evaluated under real life factory conditions.
1.6.4 Task 6.4: Evaluation of field test
Besides the detailed evaluation of the benchmarks defined in T6.2 the experiences during the field tests can be summarized for further improvements of the PRACE system after the project. This experiences are grouped into two groups and are further detailed in Deliverable 6.4.
For the manipulation capabilities, the following experiences have been made:
• With the multimodal control strategies developed in WP4 a stable switching between the control modes could be established. This was a major benefit for the flexible and rapid development of the field test applications. The determinism of the used path planners has to be improved. The results of the path planner sometimes were odd and not optimal.
• Although effort has been focused on improving the performance between the different controllers, there is room for improvement regarding the communication between path planners and the ABB controllers.
The dynamics of platform, e.g. the suspension, influence the precision of manipulation especially during fast movements. This can be bad when doing precise manipulation. An adjustable suspension could be a solution for this problem or an active compensation based on acceleration sensors can be done.
• When using the different tracking systems during the field tests the following experiences have been made:The covered workspace of the tracking systems are too small. Additional measures have been taken to increase the tracking workspace, such as a pan-tilt unit on top of the robot. These measures are expensive and introduce additional challenges for the precision of the overall system.
• The tracking system is very sensitive to lightening conditions. Multiple calibration runs were necessary during the preparations of the field tests.
• The different implementations have abroad scale of accuracy. An analysis of the required precision of the application has a high impact on the effort and costs involved with the tracking system.
For overall performance please refer to D6.3.
During the field test we had 10 parts which were used by each pass through to inspect the surface damages. The results were that no surface damages were visible and the tactile assembly strategy is compatible to the manual process.
Potential Impact:
1. Expected Impact
PRACE aims to develop a new robot learning and execution concept which is able to be trained very fast and effective and which can deal with changing boundary conditions in execution mode. These are key technologies towards an application flexible robot generation which can be used for frequent changing applications with minimal engineering effort. Today’s robot systems are limited on the classical high production volumes with a certain number of different types. The main obstacle for these automation systems towards a broader deployment lies within the high engineering effort to transfer a robot system from one application to another.
The approach in PRACE is not to develop a robot system from the scratch but to use existing, innovative market available components or prototypes and to combine them to a system with complete new abilities. Hardware components are only developed in the case that market available products are not able to meet the requirements. Therefore it is not astonishing that the expected exploitation results from the partners are focusing much on concepts, methods and algorithms. The approach of PRACE is therefore more or less hardware independent and may be adapted to different platforms.
That approach of PRACE will be a big advantage thinking in terms of the future benefit of the results that are expected from the project. As most of the robot suppliers are confronted with a growing saturation of the market for classical robot applications, especially in automotive production, new application fields are targeted, but no breakthrough could be noticed until now. The basic technologies being developed in PRACE might be an important contribution to boost the development of next generation robots with much more flexibility towards the further automation of manual working places. The basic technologies are gathered in the following tables of this deliverable.
It can be expected, that aside the mentioned saturated classical robotic domains a complete new market is already under development. The classical robotic domains cover high automation with short cycle times at high cost locations. Driver is cost reduction. New, upcoming market can be found in two areas:
SME companies that have demand for automation solution, but are not familiar with robotic technology and therefore are still avoiding an investment. However, new suppliers as for example “Universal Robots” are showing that currently about 1000 robots per year are already sold into that segment. The key technology are robots without safeguards, user friendly HMI and a flexible hardware to avoid changeover processes when changing products have to be manufactured.
The other upcoming domain are robots in low cost locations, which are still have strong capabilities with manual assembly. However, due to the rising labor cost the trend already started to start with automation. Famous example is FoxCon, where initially a million robots were targeted. Although that target has been reduced, still a significant number of new robots at high cost locations can be expected. Due to the very flexible manual lines those types of robots will have to meet quite the same requirements as for high cost SME companies.
As PRACE addresses a lot of those upcoming requirements a significant impact on the further development of robot technology towards those new application classes can be stated. In following as a summary the exploitable results are listed. They cover both methods and components that are possible key technologies for the upcoming robot classes.
Due to the still increasing automation trend worldwide it is expected that in terms of societal implications the requirements of education will continuously change. The demand for low skilled workers will be decreasing, while on the other hand more and more qualified workers will be necessary to design, maintain and operate the upcoming automation solutions. The parallel ongoing demographic development will increase the demand for high skilled workers.
2. Exploitable Results
* Task instruction system
USP: Fast and intuitive instruction of new tasks to a robot system
Market: Estimated 500.000 SMEs in EU. Robots like the Universal Robot have introduced simple instruction systems, which have enabled first-time end-users to program the robot for simple tasks within minutes.Currently sales ranges within 1000 robots a year.
* Task execution system
USP: Fast and intuitive execution of new tasks in a robot system without manual coding
Market: please refer to “task instruction system”
* Tactile Gripper System for Automatic Assembly
USP: Compensation of tolerances in automatic handling (pallet deformations, fixture tolerances) by simple, robust system (instead of sensitive camera guided systems) to increase OEE
Market: EU- market for assembly and handling robots: 217.000 robots/a. Market potential for flexible gripper estimated on 10-20%.
Double market potential is estimated in non-robotic handling sector (VDMA: turnover 2011 in D: industrial robot 2,8 Mrd €, handling sector: 5,9 Mrd €)
* Flexible production assistant based on rob@work 3 product (mobile platform)
USP: Full integrated industrial mobile manipulation system, human-robot collaboration
Market covered: Mobile robots, flexible production
* Engineering system for industrial service robots
USP: Multivendor and multi-application engineering system with direct link to the end user
Market potential within European manufacturing industry
* Concept for assembly task definition for non-robot experts
USP: Robot programming done on the shopfloor without programming experts
Expected market:end users of robot systems at manufacturing industry with medium lot sizes
* Calibration method and framework for multi-sensor system
USP: Ease of use by user not expert in vision to establish a vision guiding system for robots
Expected market: about 1000 applications/year.
* Mobile platform with detachable manipulation system
Please refer to comments of “Dual-arm haptics for definition of manipulation tasks”
* Dual-arm haptics for definition of manipulation tasks
This exploitable result has not been described in the initial DoW, but has been developed from ULUND as an additional activity. Therefore no detailed information sheet are provided in this report. Currently negotiations are running between LUND and ABB to clarify the integration of the result into ABB product portfolio.
This is a summary of the final PUD which will is documented in D7.3 (and an updated version is included in the final report).
3. Dissemination activities
In total 10 scientific publications (IEEE, IFAC,..) with the focus on intuitive, skill based robot programming methods has been worked out. This is in accordance with a main focus of PRACE to boost robotics by the development of much more intuitive robot learning approaches.
Another 16 dissemination activities, mainly workshop contributions, had been conducted. The workshops had been selected to address a broad auditorium by the size of the event (ICRA, European Robotic Forum, VDI,...). The event with the probably broadest dissemination impact had been the participation on the AUTOMATICA fair 2014, where the PRACE demonstrator was displayed to a broad publicity, consisting with a high rate of robotic end users.
For more detailed information please refer to the dissemination list included in ECAS – “Final report”.
List of Websites:
www.prace-fp7.eu
1 Publishable summary
Driven by the trend to a more and more customer specific production the boundary conditions for assembly automation have changed significantly. As the systems available on the market cannot cover this extreme flexibility towards weekly changing applications a new robot system concept was developed within PRACE. An important requirement is the ability to train the robot system with worker skill fast and intuitive.
The PRACE concept basically relies on robot learning by demonstration. We compare the robot learning to a master-apprentice-relation-ship. There, a master teaches an apprentice by instructing certain skills by demonstration. The apprentice watches the actions and effects to categorize this newly gathered knowledge into his knowledge base. Then, while applying this new skill, the master corrects the execution by refining the experience. This loop is iterated until the master is satisfied with the result.
To compensate the offsets between the underlying models and tolerances in pallets and parts a tactile lightweight gripper was developed. The gripper is able to compensate offsets in the range of a few millimeters by a passive compliance, so that fast cycle times and robust assembly processes could be achieved.
Another important aspect of the PRACE robot system is the operation without safeguards to reach the target of fast setup times. Operation without safeguards however limits the maximum robot velocity. To remain competitive with the human worker a dual-armed robot approach is followed to reach a similar working output as the human worker by modest robot velocities.
With the combination of dual-armed manipulation and a mobile platform to provide local mobility within the working place, basically new application tasks may be now automated economically by this new system approach. Using a modular approach the PRACE system can even be recombined to use only parts of the robot system for dedicated applications, i.e. using only a single arm or using the system without mobility.
Different assembly use cases were defined as test environment of the PRACE concept. At end of the project a successful evaluation of the dual arm PRACE demonstrator in a BOSCH diesel plant finalized the project.
Project Context and Objectives:
1.1 Summary description of work and results
1.1.1 WP1 – Workflow Refinement
To interact with a Robot Apprentice a key element is the development of an abstraction level that is both intuitive understandable by the operator and offers on the other hand the complexity and granularity to describe all relevant parameters which are necessary from the system point of view to derive executable control programs.
It could be proved that all use cases can be described by recombination of 10 basic classes. These so called AUAs (assembly unit actions) describe hardware independent basic segments of assembly tasks, like “pick part” or “transport part”. On that level of abstraction it seems reasonable that a non-expert can intuitively understand assembly tasks.
However it is obvious, that beyond that simple description language more layers are necessary to fill the gap between the task description language (AUA) and the executable code which runs on the control system. Therefore conceptual work has been finished to obtain an overall data structure to describe the AUAs in all levels. On machine level so called primitives describe abilities of the system, which are hardware specific. The primitives are integrated in the high level AUAs which represent to the user the machine independent skill level of the PRACE abilities.
Each AUA contains a set of parameters which define the behavior of the AUA under execution mode. Therefore each AUA has to be filled in the learning cycle by either parameter the user can key in directly or by demonstration of example movements.
In automatic mode a task planner is responsible to schedule the single AUAs in the right sequence. Central mechanism is a state machine to process pre- and post conditions and to guarantee the processing of the primitives according to the skill description of the operator.
1.1.2 WP2 -Tracking and Recording Movements and Actions in Manual Assembly Processes
To perform a PRACE learning cycle the operator has to describe the desired task by combining the AUAs representing the capabilities of the system. He can use either existing skills that are modified or start from the scratch. In any case each AUA has to be adapted to the special boundary conditions of the current application. Focused on the underlying data structures the learning cycle has the target to determine the application specific parameters of each single AUA.
To do that parameterization two steps have been suggested. In the first step a human builds an AUA tree by combining primitives and existing AUA´s on a graphical user interface. In the second step the robot system is utilized in combination with tracking devices to parameterize the remaining parameters of the task. Further, parameters can be learned from kinesthetic teaching approaches by physical manipulate the robot manipulator. Regarding the required sensor systems the hardware setup has been specified.
1. A robot manipulator: This manipulator can be either a one or two armed manipulator.
If the robots have to do assembly some kind of force/torque sensor is necessary (either integrated into the manipulator or as an external sensor)
2. A tool camera: The PRACE system makes use of a tool mounted camera to recognize work pieces and calculate where to grasp the work piece.
3. A scene camera: The PRACE system utilizing one or possibly several 3D scene cameras to monitoring the human operator and the objects in the scene under the teaching phase, make rough estimates of work piece locations etc.
4. A gripper: The PRACE System utilizes a compliant gripper to compensate work piece tolerances and sensor inaccuracies.
5. A lightweight human machine interface: In order to be able to program and run the robot programs the PRACE system consists of a tablet or another lightweight HMI interface.
6. A tracking camera system: A handle with mounted gripper is tracked and thus used to record approximate 6D positions in the scene.
7. For precise requirements a teach-in-handle has been developed, which consists of a robot gripper, a handle for manual guidance by the operator and location elements to track the position of the handle in the learning process.
1.1.3 WP3 - Skill-Supporting End-Effectors
For the gripping process a lightweight gripper (250g) was developed. Its weight is under the limited payload range of the ABB dual arm robot (500g per arm) and integrates all the functionalities for tactile assembly processes. The benefit of tactile assembly is a significant improvement of stability and cycle time compared to state of the art solutions basing on vision technologies with the known limitations of poor detection results for metal parts or obscuring of camera by the gripped object.
Using rapid prototyping technology some initial concept studies have been performed and tested in lab. The actual state contains a pneumatic driven gripper actor. The actor may be mounted to a lightweight Z- or X-Y-compliance. As sensors small hall sensors are integrated to measure the actual values of the deflection of the compliance module. It could be proved that offsets can be compensated within 1s in the range of 4 mm in X-Y-direction and 5 mm in Z- direction.
The use cases have been examined to get a classification of the tactile assembly strategies. Basing on that work the structure of the tactile AUAs has been worked out and the set of parameters has been defined that are contained in the AUA.
1.1.4 WP4 – Robot Capabilities
To assess the safety aspects of the PRACE system a risk analysis of all subsystems has been carried out and consequences on the system concept were derived. The SW architecture is the central element of the PRACE system. It has to control both the perception and learning cycle of the system as well as the robot executing system regarding safety aspects, as PRACE is operated without safeguards. The result of the learning process can be visualized to the operator by the real robot system as well as by simulation. All learned knowledge is stored in a knowledge data base so that a transfer to other robot systems is possible. Innovative features of the SW architecture are:
• High level of robot cognition
The robot has a high level of cognitive capabilities that are implemented on many levels of the perception pipeline (from low level sensor fusion to high level information fusion) and provides it with the ability to act intelligently. The cognitive capabilities enhance the robot’s ability to understand both the tasks the robot has to execute as well as the environment it executes within.
• Mixed initiative
The robot will implement the concept of mixed initiative. In mixed initiative communication between the robot and the user is bidirectional meaning the user can interact with the robot as well as the robot can ask the user for clarification or to provide information. This is especially relevant in relation to error handling.
• Innovative robot skills
One of the main benefits gained by using robot skills as defined in PRACE is the possibility to define and execute system independent AUAs, which allows for skill-reuse across different hardware platforms.
• Intuitive teaching
The robot will have a user interface from where the user can instruct the robot to perform tasks. The user interface will be designed with special focus on providing an easy to use interface that does not require substantial training (other than an introductionary course).
• Real time motion planner
The robot will be working in a dynamic environment and therefore the robot needs to adapt to changes in real time. The real time motion planner will constantly get information about the environment and stop the robot or, if possible, move around an obstacle to avoid collision.
Two demonstrators at IPA and LUND have been built up. They consist of the mobile iCOB-Platform and the ABB dual arm robot YuMi.
1.1.5 WP5 – Integration into application
To collect the requirements for the PRACE system an initial investigation of thirteen manual working tasks in assembly have been examined. Typical operations are palletizing, testing, packaging, assembly, machine loading and part feeding. Typical workpiece size is in the range of smaller than 100...200 mm and a weight of less than 100g. As development guideline the system specification has been elaborated.
Furthermore the outcomes of WP1-WP4 have been integrated into two demonstration platforms. The first demonstrator focuses on haptic interfaces for dual arm robots. The second demonstrator, a full dual arm mobile manipulation system, was used for the Automatica Fair demonstration and the field tests. It incorporated the PRACE application development methodologies, the developed motion capturing system, compliant grasping system and a safe mobile manipulation controller.
1.1.6 WP6 – Test and demonstration
Basing on the use cases the plant testing scenario has been determined. By mapping the occurrence of the innovative PRACE features on the use cases a prioritization had been done. The preferred use case was the so called “needle scenario” where Diesel parts have to be transferred from a transport pallet into a washing tray and vice versa.
Towards end of project a two weeks field test was executed in a Diesel plant from BOSCH. The needle picking scenario was implemented on site with the different teach-in and description methodologies that have been developed within the PRACE project. The performance, feasibility and development were evaluated under real life factory conditions. The plant testing could be successfully finalized. The plant manager was convinced by the PRACE concept approach. As decision two prototype cells with the ABB YuMi robot will be adapted on a use case quite similar to the needle use case. Start of production in the plant Feuerbach is targeted for mid 2015.
1.1.7 WP7 – Exploitation and Dissemination
The dissemination activities have been continued with various scientific publications and workshop contributions. As the installation of the PRACE system was completed on 5/2014, the AUTOMATICA 2014 fair in Munich had been chosen for the first public demonstration of the PRACE prototype system. The AUTOMATICA as a lead fair for robotics supported an optimal dissemination due to the large auditorium consisting of decision staff members from the target branches.
To start the exploitation activities all partners contributed the expected exploitable results of PRACE. For the project actually seven exploitable results have been identified, ranging from system components to methods and algorithms. Two more results are currently under negotiation between LUND and ABB. The final PUD is documenting the current status concerning the market boundary conditions and the FGIP aspects.
A website has been established, describing the vision of PRACE, the workpackages and the consortium partners. It also contains the non-confidential deliverables, video clips and disseminations.
Project Results:
1 Work package results and achievements during the project
1.1 WP1 – Workflow Refinement
1.1.1 Task 1.1 Specification of Assembly Unit Actions
To identify representative AUAs the Bosch use cases were investigated. For that reason concrete but hardware independent AUAs were defined and interconnected to perform the wanted tasks. Each AUA represents an intuitive understandable process. The resulting AUA-Chart (in state chart or tabular form) therefore represents typical use-cases. As a result of this investigation ten classes of AUAs (or highlevel AUAs) could be distinct and scored by the number of their appearance. The most prominent AUAs are “Pick part”, “Transport part” and “Place part”. More AUAs are needed to fulfill the requirements of all tasks but as a conclusion the level of abstraction is reasonable to allow non-expert user to intuitively understand the task process.
1.1.2 Task 1.2 Analysis and design of the Assembly Unit Actions
The concept of AUAs from task 1.1 was further concretized. To widen the reusability of AUAs more levels of abstraction were taken into account. However the levels are conceptual equivalent. The visibility by any user may differ by the specific level of expertise (User, Operator, Expert).
E.g. “Pick part”-AUA as seen by a user may consist of one “Move Robot”-AUA followed by a “Close Gripper”-AUA. These two more specific AUAs (called primitives) represent a level that an operator can use to create more specific AUA sequences e.g. if more complicated trajectories of the robot arm movement is necessary. However for an expert each of the previously described primitives may consist of at least one hardware related implementation to obtain the goals represented by the primitives.
Primitives basically consist of two components: On the AUA structure side an AUA interface which makes it possible to use primitives as low-level AUAs, on the hardware side the hardware specific code to execute the desired action. The communication between these components is also located inside the primitive.
Regardless of the level of the AUA certain information must be included into any AUA representation.
To allow runtime checking whether any defined task can be performed preconditions in terms of a necessary environmental state or availability of needed devices and a representation of postconditions (environmental changes due to the execution of this AUA) must be available. Furthermore every AUA will in general need some parameters for concrete execution.
1.1.3 Task 1.3: Conceptualisation of Assembly Unit Actions
For implementation and realization purposes the concepts worked out in task 1.1 and task 1.2 must be conceptualized.
The overall sequence of AUAs representing one specific job of the robotic system can be represented as concurrent state-charts (e.g. SFC). The different abstraction levels can be taken into account as hierarchical concurrent state-charts. Therefore every AUA can be represented as one state-chart with at least one state. A specific robotic job results in a hierarchical concatenation of several AUAs.
To allow automatic semantic checking, as well as logical checking if an execution is possible during runtime each AUA (independent of its level of abstraction) must provide preconditions, which define the necessary state of the environment before the specific AUA can be executed. E.g. before placing a workpiece, some workpieces must be grasped first.
To check certain sequences of AUAs and ensure that due to the order all necessary preconditions can be met, it is also essential to describe postconditions. Postconditions may be distinguished to two kinds, namely effects and outputs. Outputs in this terminology are results by one AUA that might be used as a parameter enabling one following AUA to execute (E.g. localization of a workpiece by a vision system and forwarding the coordinates to a position correction of the robot arm).
Effects are environmental changes that result from the manipulation caused by the execution of an AUA. After a faultless execution the effect normally reflects the reason why an AUA should be executed.
By including all pre- and postconditions two major benefits can be stated:
First, AUAs are interchangeable if a) their effects are identical (their effects are entailing
preconditions of the next AUA) and b) their preconditions are entailed by effects of their predecessors and the constraints of the world (e.g. availability of resources, like sensors), and secondly by setting up an ontology to represent all relationships, an automatic planning algorithm may find a certain sequence to achieve predefined goals (wanted effects) by concatenating several AUAs.
1.1.4 Task 1.4: Synthesis of executable code
In order to connect the abstract and platform independent AUA’s and the platform dependent execution primitives of the robot, interfaces have been specified. These interfaces encapsulate the platform dependent configuration and behavior. After the specification of the AUA sequence by the end user the AUA’s are connected to trigger the primitives by these interfaces. After this process an actual execution of the AUA sequence is possible with the robot.
Following the concepts developed in task 1.3 four distinct steps will be performed to sequentially concretize generic AUAs, to applicable Primitives and finally to executable programs represented in state charts of primitives with conditioned transitions.
In a first step a generic AUA is interconnected to a component description of the device the AUA shall be perform by. Thereby a primitive of AUA description is generated. A next step needs the user to choose of define a task description of the sequence or concatenation of generic AUAs. It should be emphasized that both, the first and the second step do not need any preceding computational action or information gathering. Therefore these steps do not necessarily need to be performed after each other, but in an arbitrary order or in parallel. However after the above steps were performed the third step uses both the AUA description or primitive (representing a generic AUA concretized with one specific device) and the task description provided by the user input, to generate an executable state chart. The generated state chart is finally handed to a dedicated executor, performing and controlling the state transitions and action execution. Depending on the type of used device, its possibilities for communication and needed performance level and the AUA itself, the concrete execution of different primitives can be coordinated triggers using ROS services or messages, execution of native machine code on dedicated controllers (e.g. the robot controller) or direct (inline) execution of specific commands during the state evaluation.
By pure realization of the above approach all possible degree of freedom is granted to the user. However convenient for advanced skilled operators and developers, it was found unsuitable for every day factory realization. For this reason an additional approach was integrated in the overall synthesis. Using the tool BRIDE (more on BRIDE in description of workpackages 5 and 6), a developer is able to create and encapsulate higher level primitives by concatenation and sequencing of lower level primitives. The created primitive is fully encapsulated and integrated into the overall synthesis scheme between step one and step three. For example a developer can predefine e.g. a pickup primitive. The pickup primitive is in fact a simple sequence of lower level primitives like move and close gripper. From a naïve point of view such a sequence can be easily create by the user himself, however it showed very intuitive to provide the end-user with this functionality. Thus, the user can choose this functionality and handle it as it would be only one single AUA. By closer observation the integration of predefined higher level primitive can a multi-system realization of hierarchical state charts.
1.1.5 Task 1.5: Specific dynamic simulation
A simulation environment based on the RobWork off-line robot simulation software package have been developed and integrated in the PRACE framework. The simulation environment has formed the basis for the motion planning results described under Task 1.6 and in Deliverable D1.2.
1.1.6 Task 1.6: Offline/online motion planning
The integration of Motion Planning in the PRACE project is studied and reported in Deliverable D1.2. As hardware platform for the PRACE project, the two armed robot FRIDA from ABB has been chosen. The two arms will not be used for co-operation but for speeding up the process by doing tasks in parallel. So for the Bosch needle case the task will be to move all needles from one tray to another using both arms to move needles independently.
The use of RGB-D sensors (such as Kinect© or Asus Xtion©) for motion planning is also studied in deliverable D1.2. Such sensors have several limitations. They have a too low accuracy (worse than 5mm at 1m) for fine obstacles detection, and as shown in Figure 17, they badly reconstruct some of the PRACE workbench object (transparent plastic, metal grid). The use of a single sensor (i.e. a single viewpoint on the scene) also produces occlusions.
However, such sensor offer a real-time update of the scene (colorized 3D cloud at 30Hz), they are cheap and easy to use in a workbench (Asus Xtion© sensor only requires an USB plug). The final outcome of the experiments with these sensors is that they can be used for rough estimation of position and obstacle detection, but not for precise motion planning for manipulation task close to obstacles.
1.1.7 Task 1.7: Visualisation
Motion planning and simulation results are necessary to visualize in order to evaluate performance and develop well performing algorithms. To support this development, visualization of the FRIDA robot, PRACE work pieces and tools (grippers, compliance unit, etc.) have been implemented into the ROS visualization tool RViz. The visualization can display both real robots or simulated variants without any modifications and additional input sources such as 3D models, live camera and point-cloud information can be imbedded directly.
3D and augmented reality visualization tests were performed, to evaluate the possible ways to visualize additional information on a given 3D scene. Those visualizations principle could be useful for:
- The motion planner / collision checker
- The learning machine demonstration of the learnt information
Next steps in T1.7 were to investigate which visualization strategies should be applied to present motion planning, instruction and system status results to the operator. Which visualization modalities should be applied, what information should be available and which technology should be applied will be important questions in this research.
Visualization of the robot movement has been initially integrated in the end-user interface on the DTI testbed. Next steps are to indentify which elements could be added to the visualization to support the end-user in following and being familiar with the flow of the robot system.
For simple tasks or tasks that be realized by pure sequencing of available AUAs or higher level primitives (as discussed in T1.4) an html-based front-end was created for the facility staff support. This lightweight gui reflects only a small variety of the available possibilities of the task specification, but still was capable of allowing all use-cases to be created. Due to its html implementation it furthermore allows to enter the task creation gui using any browser, even to manipulate a task in parallel from more than one device.
1.2 WP2 -Tracking and Recording Movements and Actions in Manual Assembly Processes
Work package 2 has a core focus on instruction of the robot system by input from the so-called Master. Using a user interface and a number of sensor modalities, the instructions from the Master must be transformed into a sequence of AUAs, and the parameters of each AUA must be tuned and adapted to fit the individual requirements of the process. The parameter adaptions can be capturing forces applied during assembly e.g. when performing a snap-fit or counting the number of turns when screwing.
While fully acknowledging that the Master is the expert on the individual processes which must be automated, the focus of WP2 is not on fully mimicking the actions of the master instructor, but has chosen a more object centric approach of task instruction. Sensor modalities are preferably used to track the objects for assembly and record individual parameters such as forces, with the purpose of extracting information on how the individual objects are assembled. Detailed information and interaction with the robotic system will be performed using touch-based tablets as input modalities.
Human centric sensing, such as motion capturing, naturally gathers information on how the humans performs parts of the process and will be utilized where the modality fits into the instruction work-flow and where adequate accuracy can be obtained.
At the current state in WP2, a number of individual results have been obtained. Using sensor gloves and fingertip force sensors to capture input from the Master instructor, the measurement of forces in grips and hand positions are in its final states. A task of combining these measurements using sensor fusion are just starting to achieve results useful for robot instruction. Excellent work is also ongoing on extraction of human motion using the Primesense based depth cameras, but efforts in converting these measurements into robot instruction are yet to be started.
The results on this topic are for most parts described in deliverable D2.1 which describes the obtainable data using the investigated sensor modalities and conceptual work on the PRACE task-instruction methodology.
In the project period M12 till M18, focus in WP2 has been towards getting a first functional implementation of the instruction concept ready for evaluation of the concept and the various modalities. The results are documented in D2.2.
1.2.1 Task 2.1 Evaluation of appropriate sensor devices
Task 2.1 focuses on investigation into a number of sensor modalities useful for task instruction in the PRACE system.
The sensors under current investigation are:
• Flexible force-sensors for measurement of forces applied at fingertips.
• Accelerometer-based sensor gloves for tracking of finger positions.
• Touch-based tablet computer for instructions.
• Singular and stereo cameras for object pose estimation.
Evaluation of the sensor modalities are progressing as planned. Results obtained using the force-sensors and sensor gloves are documented in deliverable D2.1.
The short conclusion of these investigations are, that data capturing gloves can only be used to capture simple information e.g. like single forces or orientations. But the actual identification of finger positions are not possible to an excend where the results are useful for robot instruction. As a result of this, the use of human fingers and tracking gloves have been discarded in favor for using a tracking handle who mimics the robot tool and allows a much better geometric tracking.
1.2.2 Task 2.2 Data capturing and tracking mechanisms
Task 2.2 focuses on the data acquisition, initially dedicated to human body tracking. As the task instruction concept of PRACE have re-focused to be object centric, the last part of this task reflects a shift of approach towards object-oriented technologies, including teach-in handle tracking and robot-tool camera technologies.
The deliverable D2.1 contains some results of this task in respect to human centric capturing and tracking. A state of the art was first conducted to validate the choice of the Kinect as a human body sensor compatible with PRACE project. Software was also implemented to access the Kinect data (RGB, depth, users’ segmentation and skeleton), and display it in a 3D viewer. At last, performances of the skeleton tracker were tested on simple criteria (temporal, spatial, validity, accuracy, scene artifacts, etc...).
The multi-cameras (and/or multi-kinects) calibration software has been developed and performs intrinsic and extrinsic calibration. The design aim of the calibration software is to be generic for different setups of vision sensors. It shall for instance be able to calibrate two cameras, three Kinect, a mix of these setup, etc. In practice, we have experimented that this generic feature comes at the cost of usability, and an application specific to a given setup is preferred by users that need to often calibrate the same system. Therefore, the software has been used by Magellium to calibrate the setup of the Teach-In Handle and it has been adapted to calibrate the validation setup of human body tracking composed of a Motion Capture system localized with respect to the Kinect system.
A ground-truth comparison mean was developed, in collaboration with the LAAS laboratory, at Toulouse. It is based on a reference system, a Motion Capture system composed of 10 IR Sensors and passive markers, giving the human tracking reference (the ground-truth). An acquisition campaign dedicated to absolute accuracy evaluation was performed and processed.
This acquisition gives a useful base for human body tracking algorithms validation. 3 Kinects© were co-localized, and simultaneously used to track a moving human. As shown in Figure 24, those 3 skeleton tracking results were compared with the ground truth skeleton.
The real-time synchronization and the data handling to be performed in this task have been postponed to task 2.4 as the sensors to connect and the data to handle depend on each learning (or execution) method.
As the human body tracking was excluded from PRACE learning principle (as a WP2 risk reduction decision), has been involved in the Teach-In Handle tracking. As shown in Figure 25, the teach-in handle consists in the robot gripper, mounted on a specific handle. This teach-in handle is manipulated by the human, to show specific actions to the learning machine, with a robot representative tool. The goal of this task is to be able to locate the teach-in handle with respect to the workbench.
The study results are given in deliverable D2.2. They consist in a state of the art concerning the existing off-the-shelf sensors and software, and in the development of a first teach-in-handle tracking demonstrator. The first demonstrator which is described in the Deliverable D2.2 was tracked with a 3D-marker composed of four colored balls. Based on the results of testing and evaluating the accuracy and stability of marker tracking, the number of colored balls increased to five during the project. Additional to this optimization process the teach-in-handle gets illuminated colored balls with infrared LEDs in each ball, an electrical gripper and a wireless connection with a Sony Move controller to communicate the status of the gripper with the master controller. The teach-in handle based learning method, dedicated to specific actions or area teaching is handled in task 2.3.
When executing tasks roughly instructed by humans and where work pieces have been placed by human operators, it will be impossible to execute purely off-line programs. So solve this issue, sensors must be used at execution to compensate for uncertainty and process variation.
To refine positions of objects at both instruction and execution, a tool-mounted vision system has been developed.
Using the Halcon vision library, 2D and 3D vision algorithms have been applied to efficiently detect objects to grasp and the pose of the destination objects. Using a 2D matching algorithm, objects can reliably be detected by their 2D shape as in Figure 27, with the only constraint that the height between robot and object should be known.
When a number of unique features on the object can be identified and their physical dimensions can be measured, a full 6D pose of objects can be estimated as in where the Danfoss heat sink is detected using four screw holes, which allows detection of all six degrees of freedom with a high accuracy.
The needles are also fairly straight forward to detect using 2D matching. The 2D matching approach also makes it possible to detect multiple instances of any given object and return the best match. This allows the robot to keep picking needles in a sequence until no further objects can be detected.
As the FRIDA robot platform only have a payload of 500 g total, a significantly lighter tool-camera platform is needed. Furthermore, FRIDA only have the possibility of running 4 signal conductors through the arms, which are needed for tactile sensing in the gripper. To accommodate for this challenge a lightweight wireless camera solution is developed, which suits the FRIDA Platform. As this is primarily an integration task, it is further elaborated under T4.4
1.2.3 Task 2.3 Motion and action extraction from the raw data
The activities of this task are focused towards extracting action information useful for parameterization of AUAs from sensor sources who stream sequences of data. Furthermore, this task involves the development of methodologies to control and represent the process flow of tasks that contain information of this type.
A kinesthetic learning system has been implemented using the Universal Robots UR5 robot platform. The joint-force measurements and internal control system of the UR5 robot support manual guidance of the robot to learn positions and trajectories. This learning methodology has been implemented to parameterize AUAs with end-positions and trajectories for motion commands.
In order to execute most robot tasks, process control more advanced than sequential actions are needed. Most tasks can be represented by having conditional sections (If / if else) and conditional loops (e.g. loop until no more needles are detected). Although these concepts might be easy to understand for anyone with a programming background, they are complicated to visualize for operators without programming experience.
Robot tasks can also be built using several higher level skills, which internally are based on one or more AUAs.
Furthermore the teach-in handle based learning method will be implemented in this task. This learning method will consist in identifying information from the human action, if possible using the scene camera (or a scene Kinect©). The main reasonable data to identify are:
- The workbench main plane,
- The pick pallet area,
- The place pallet area.
The use of the teach-in handle for precise position and orientation teaching is still to be studied. Indeed, this may require perceiving the teach-in handle from a close camera, as the gripper one.
1.2.4 Task 2.4 Data merging algorithms
The purpose of Task 2.4 is to merge the data contribution from the different instruction modalities developed by the different partners of PRACE.
A data fusion algorithm was implemented for the human body tracking. Indeed, the multi skeleton tracker developed for PRACE uses several Kinects sensors to refine the position of a unique resulting skeleton. Delayed logic is thus involved to find the more reliable skeleton between raw skeleton of each sensor, and a Kalman filter predicted one. Our approach combines Viterbi and Kalman, and runs at 20Hz (in a mono-thread CPU application on an Intel Core i7 2760Q – 2.4Ghz). This algorithm was described in (Masse, Lerasle, Devy, Monin, Lefebvre, & Mas, 2013). The proportion of acquisitions whose joints were closer than a distance from ground truth (50, 100, 150 and 200 mm, those are cumulative).
The accuracy was estimated for 5 given skeleton joints (head, left and right hands, left and right foots), and for several skeleton estimators:
- the 3 raw Kinect acquisitions (sensor 1, sensor 2 and sensor 3),
- a virtual sensor which would return the best nodes of the 3 raw sensors (sensor best5),
- a Kalman result (0-order, no speed in state),
- a Viterbi result.
The data fusion of our approach increases the reliability of the system, as fewer frames are outside the outstanding distance of 200mm from ground truth, and it also increases its accuracy, as the amount of positions lower than 50mm mark increases. The analysis of the results of this task has oriented future works towards the fusion of data coming from human tracking and object tracking.
1.2.5 Task 2.5 Set of design rules for further Assembly Unit Actions
According to the definitions in D1.1 a skill in the PRACE system is a capability which is system independent and therefore can be executed on two different systems. Each skill can be realized by system primitives and/or be a composition of other skills. Assembly Unit Actions (AUAs) are skills at a user-defined, assembly-relevant level.
When a robot in PRACE needs new capabilities it is required that the needed primitives is implemented at the specific robot system. This implementation has to be conducted by an AUA design expert which is able to program new primitives from scratch.
To create new primitives which can be executed at the PRACE system the overall primitive has to obey the inputs/output table.
The designer needs to list the required parameter needed to be able to execute the primitive. The data type of each parameter can be all defined types in the system like int, strings or specified data types like 6D pose or a trajectory. Furthermore, the pre-/postcondition has to be constructed as logical conditions. Conditions can vary from simple boolean conditions which are evaluated as true or false or more complex evaluation like robot states or other more complex data structures.
An example on a precondition for a sensor primitive for locating objects with a tool mounted camera, in a palette could be a condition that evaluates whether the robot is in the correct state above the pallet or not before executing the primitive. Simpler preconditions could be that the gripper has to be open before closing it with a closeGripper primitive.
In Deliverable 2.3 guidelines for constructing new primitives are presented. The guidelines are meant as a listing of required knowledge that an AUA design expert needs before starting programming the primitive. The guidelines try to make the developer thinking the right thoughts to determine the correct, pre-/post conditions, parameters, outputs and effects. Following these guidelines will help the developer identify missing knowledge about the functionality of the primitive. If the developer cannot answer the questions or list data types etc. the designer need knowledge regarding the required functionality of the primitive
1.2.6 Task 2.6 Learning routines
Learning tasks directly from the Master instructor are a very efficient method of instructing robotics tasks. To capture the full workflow of a pick-and-place task, a teach-in handle based learning system were developed. Using a combination of the tracking handle and the instruction user interface, the system gave the possibility of building entire processes by using the handle for providing positions and the handle buttons to mark waypoints, as well as grasp/release points.
A full video of the demonstration can be found here:
1.3 WP3 - Skill-Supporting End-Effectors
A main target of PRACE is the fast and intuitive learning of industrial assembly tasks. A new application should run in production mode within a single day. Besides the flexibility and usability of the learning strategy another very important boundary condition is the stability and reliability of the generated movements out of the learning cycle. To guarantee this stability the PRACE system needs the ability:
a. to compensate offsets between learning cycle and execution cycle, due to inaccuracies within the sensor data generated in the learning cycle or due to general modelling inaccuracies.
b. to react on changements in the environment that typically take place in industrial environment, like deformation of pallets or tolerances/offsets in part providing.
The duty of WP3 lies in the development of a gripper which is able to detect and compensate such distortions within the assembly process. State of the art for many years has been the integration of force sensors into the gripper flange to detect such offsets in a tactile way. The problem of that approach however is the stiffness of the system. In case of contact between robot and workpiece a force is generated a necessary input for force controlled algorithms to compensate the detected offset by path modification of the robot. This approach is not suitable for industrial use, as due to the stiff construction the contact force ramps up very fast, which is crucial due to the potential damage of automotive parts with very high surface quality requirements. Therefore an approach is pursued with compliances gripping elements in PRACE.
On the other hand a compensation of the differences between learning and execution mode has to be performed. Additionally a gripper guided camera detected such offsets which are bigger than the tactile compensation abilities of the tactile gripper.
1.3.1 Task 3.1: Design principles of lightweight devices for precise assembly
In Task 3.1 the design principles of a lightweight gripper for precise assembly is demanded.
From an industrial point of view a major requirement of the gripper is the reliable operation under production environment in three shift mode. Additional to this requirement the gripper system weight with the handling part must be under 500 g which arise from the payload of one arm of the robot.
The first step was to use market available components to determine which functionalities may be integrated under the boundary condition of maximum 500 g payload of the robot. These market available components are too heavy whereon Rapid-Prototyping lightweight components were developed and built. A contrast to conventional springs is that a CT-joint on tension or pressure can be claimed.
The X-Y-compliance module is a two stage module and provides deflection in X and Y direction, one stage for each direction. This unit consists of two equal one-directional compliance units. By connecting them with a rotation offset of 90° a X-Y-compliance is realized. A linear plain bearing is mounted in the center of Z-Compliance to reduce the tilt under non-uniform load.
Due to the lightweight requirements for the gripper, the module is manufactured by rapid prototyping. The technical drawings can be found in deliverable D3.1.
In each compliance element a hall sensor is included in the design, to measure the actual deflection of the module. The sensors are easily configurable and can be adapted very well to the needed maximal deflection.
On the compliance module a market available gripper of the type MPG-plus 25 from Schunk is mounted. This gripper is characterized by its less weight and the compact form. The fingers which are attached to the gripper are also constructed to be lightweight and easily adaptable at the same time. Therefore, for each use case, different fingers built by rapid prototyping are provided.
1.3.2 Task 3.2 Development of pick and place strategies for precise assembly
The goal in this task is the development of pick and place strategies for precise assembly. The challenging part of the pick and place process is in general the place process. Due to that the development of place strategies is handled with prioritization. The robot is developed to manage work tasks generally handled by humans. Thus the strategies are derived from human behavior.
Therefore the use cases have been analyzed and place strategies have been derived. These strategies require several kinds of compliance. For each use case and strategy the desired compliance is noted. Depending on the strategy, each dimension of compliance has to be actively controlled (deflection control of the compliance module) or it can just deliver some passive compliance without any control.
Out of these use case specific strategies three general strategies are developed. As an example the strategy “cylindric peg cylindric hole” is derived from use case 1.
While running a specific strategy the robot will follow a basic path. Additionally to this path tactile events can occur. ,
As the insertion strategy is dependent on the application the three general strategies are listed with their requirements in Table 1.
Insertion Strategy Catalogue
Direct insertion Tilted insertion Spiral insertion
- Short cycle time (< 1 s)
- For own learning of positions in execution mode
- For insert position with a great tolerance range (> 1 mm) - For insert position with a small tolerance range
(> 0,1mm )
- middle cycle time - For accurate insert position with a tolerance range
(< 0,1mm )
- bad cycle time
Description of process parameters
Every place strategy requires a set of parameters. For the „cylindric peg - cylindric hole“ strategy these parameters are:
- Peg length (to gain the TCP at the peg tip)
- Peg diameter
- Hole diameter (to find the center point of the hole)
- Approach Tilt Angle
- Insertion depth
- Maximal contact force
Out of the peg length, peg diameter and the hole diameter the approach offset is derived. The insertion depth will define the end position. When this position is reached without contact forces the place task is successfully finished.
1.3.3 Task 3.3: Implementation of pick and place strategies for precise assembly
The main focus in this task was to implement the pick and place strategies which are developed in task 3.2. These strategies were implemented without recording the integrated compliance sensors. The robot follows a fixed path and once an inaccuracy in position occurs, the gripper is able to correct this inaccuracy by its compliant behavior. Thereby it is possible to implement the maximum robot speed by the insertion movement. A disadvantage is that the robot moves into safety stop if the robot clashes with the pallet because of the too large position error. All implemented strategies were tested in the lab. A useful strategy for most insertion applications at Bosch is the tilted insertion strategy which was tested during the field test at Bosch. This strategy reached a very stabile process with sufficient compensation of offset errors up to 4 mm (dependent on the application).
1.3.4 Task 3.4: Implementation of gripping devices
The gripping system was developed so that it can be easily attached to the existing tool-change system of the robot FRIDA. The robot provides on each arm one pneumatic connection and a six wire cable. The cable is connected directly with the controller of the robot and can be used for an Ethernet connection (10 Mbit). It is necessary for the PRACE-System to unplug the cable from the robot controller and connect it to the master controller of the PRACE-system. The wirers are used to transfer the signals from the analog sensor in the compliance elements. The gripper camera must be connected over WiFi, due to the few numbers of cables which goes through the robot arm. For operation of the gripper the pneumatic connection is used. The pneumatic valve is installed at the bottom of the robot. Only one pneumatic tube is necessary because the gripper is spring resettable.
In the gripper a circuit board is integrates with a connector to plug in it to the Robot connector. Additionally a DC-DC Converter is integrated which converts the normal 24 V from the robot to 5 V for the sensors and camera system.
The gripper compliances are made with the rapid prototyping process technology Fused Deposition Modeling (FDM) with the material acrylonitrile-butadiene-styren (ABS). The result of the lifetime cycle was that the X-Y-compliance breaks by a maximum deflection after about 3600 cycle and the Z-compliance by about 24000. Therefore the rapid prototyping process technology was changed to Selective Laser Sintering (SLS) with the material polyamide (PA). At over 300 000 cycles the attempt was stopped with the result that no changes can be measured of the spring characteristics of the compliance elements.
1.3.5 Task 3.5: Implementation of tactile pick and place strategies
The goal in task 3.5 is the implementation of tactile pick and place strategies. The difference to task 3.3 is that additionally the compliance sensors are used. The use of the compliance sensors provides the possibility to touch the position and orientation of the pallete to calculate the insertion position. In addition the robot can feel collisions and too large inaccuracies of the insertion position. Thereby the robot can adjust the position so automatically that there is no force influence on the work piece during the process
1.3.6 Task 3.6: Integration of bin-picking with grippers
Task 3.6 has been cancelled as a Steering Committee decision on M12 plenary meeting due to missing expertise in bin picking algorithms of the partners. In the running amendment those resources are shifted to support the setup of a second PRACE demonstrator at LUND. This activity has not been planned in the initial DoW, but it has pushed PRACE strongly forward concerning the integration and testing stage.
1.4 WP4 – Robot Capabilities
1.4.1 Task 4.1: Holistic assessment of safety aspects
The holistic assessment of safety aspects was carried out by Fraunhofer and Bosch There are separate forms created from the assessment, and the deliverable D4.1 "Analysis of safety concept" was submitted right on time. Refer to that report from further details.
1.4.2 Task 4.2: Architecture
A major effort in WP4 during the first period has been the specification of the architecture, and represented by Task 4.2 and the deliverable D4.2 both named Architecture. A major effort has been put into representing the architectural design, without getting stuck in details that the further research will clarify. For instance, both UML and SysML design tools directly lead to specification of system details while the overall design (that should enable the PRACE results) gets lost. Led by DTI, and developed through a long series of meetings and emails, less formal figures and models were developed and the deliverable reached a complete status.
One source of confusion in this task, as in this work package and the overall project, has been the relations to the results of the Rosetta project. The planned usage of the FRIDA robot was intended to go together with some of the architectural solutions from Rosetta, but due to the unclear situation with both FRIDA and ABB, that interplay has been lagging until recently.
1.4.3 Task 4.3: Mobile manipulation
During period 1 the efforts in this task were focused on the deliverable D4.3 "Hardware components", driven by BOSCH. That report outlines the usage of the FRIDA robot, mounted on a mobile platform, and how various peripherals will be added to meet the PRACE application demands. During period 2 the mobile platform and the FRIDA robot has been delivered and made available for demonstrations in Stuttgart (Bosch and IPA). Beyond the deliverable there are still issues to be worked out for the integration of mobility and manipulation. While that is mainly a topic for Tasks 4.4 and 4.5 the availability of the hardware platform (as of this task) is not quite complete, and therefore a completion of 98% is reported as the status. Part of this picture is the exit of Axelius, the entrance of ABB, and the tested setup at ULUND (including ABB sensor interfaces). The final 2% of the work was done in the second period of the project. Mainly the consolidation of the configuration of the hardware components, including final work on the modularity of the system was done. The outcome of this task is the integration of the mobile manipulation system in Task 4.4 and WP5.
1.4.4 Task 4.4: Integration of mobility and manipulation
The work on integrating mobility and manipulation started as planned late during the first period. The main efforts, by ULUND, have been on establishing the mobile platform (according to the Fraunhofer design, including the ROS-based software) and the tailoring of the FRIDA control system (based on Rosetta results for PRACE applications). In line with Task 4.3 as a result of period 2 efforts, there are demonstrator-like setups in Stuttgart (mobile platform at IPA still with UR5 arm, FRIDA robot delivered to Bosch for integration with assistance from ABB), Lund (mobile platform as in Stuttgart, preliminary mounting of FRIDA on it, but charging and networking not done) and Odense (mobile platform with UR5 arms, sufficient for testing key demonstrator scenarios).
For technical/control integration of mobility and manipulation, the sensor and motions control interfaces to the manipulation part (ABB controller) play a central role. Manipulator motions are accurately actuated with rather high bandwidth, compared to the mobility which has a bigger moving mass and a suspension system that makes the motions compliant and not fully controlled (vertically, in addition to the risk for horizontal slip motions). Hence, the mobile motions need to be fast and accurately measured, and the actual location of the robot base then needs to be feed forwarded to the robot motion control. For reasons of performance that control is after the trajectory generation of the robot and the interfaces on a very low level. The progress during period 2 has been along these lines of development, also building on the developments of the just finished Rosetta project.
In the work of WP2, tool-camera based vision systems have been investigated. The instruction and functionality have been evaluated using traditional GigE cameras on the DTI Universal Robot platform. To suit the requirements of the ABB FRIDA platform, a lightweight camera solution is integrated.
The prototype are integrated using the low-cose Raspberry Pi as platform, as illustrated in Figure 40.
The integration of mobility and manipulation in the final phase of the project mainly focused on the PRACE project demonstrator for the lab tests, the automatic fair the final factory tests. The tight integration of the rob@work 3 platform, the Frida robot and the grippers and the motion tracking system was done in multiple iteration. In the beginning of the final year the platform as shown in Figure 41 was created, incorporating all required components. Based on the experiences in the factory tests the prototype version of this integrated platform was extended with a lifting unit, an extended sensor head and a larger pneumatic tank to be ready for the factory tests.
1.4.5 Task 4.5: Control system
The final Task on Control system, with an emphasis on the safety aspects, is related to all the other tasks of this work package. While formally started, most upcoming results are not yet possible to distinguish from other results. The safety of the mobile platform (based on dual laser scanners) and the safety of the dual-arm manipulator (due to the inherently safe mechatronic design) forms a good basis for this task, but safety is not compositional (two safe subsystems does not automatically comprise a safe composed system, as Task 4.1 reports). Further analysis of this situation has been made, and no major obstacles have been found, but this task is still in its early stages.
During the development of the PRACE system and during the Lab tests multiple ways of utilizing the control system of the PRACE mobile manipulator were required. Therefore, three control concepts were implemented during the project. On the robot system, we have a controller running within the ROS middleware on a dedicated PC. This robot system integrates high level planning capabilities for collision free dual arm motion as well as the high level state machines and the interfaces to the engineering system. To execute the motion the ABB controller that is inside of the FRIDA prototype was interfaced. Figure 43shows this interfaces on the ABB side (Rapid program) and the ROS side (trajectory downloader and joint_state component). During the project, multiple improvement on the performance have been made. The interface allows additionally to the control capabilities of ROS the direct utilization of the control capabilities of the ABB controller itself as a second control mode.
Additionally an ExtCtrl control PC was integrated on the platform allowing advanced real time control applications to be performed. The ExtCtrl PC is directly connected to the FRIDA controller using the LabCom protocol. Additionally an interface between the ROS controller and the ExtCtrl PC was implemented using the RAPID connection to FRIDA. This makes it possible to utilize the real time control capabilities as a third control mode.
The implementation described allows seamless switching between the different control modalities depending on the application demands.
1.5 WP5 – Integration into application
1.5.1 Task 5.1: Use cases and requirements elicitation
In task 5.1 possible use cases were identified for the PRACE-system and were described more in detail. These are typical industrial applications which are operated today manually and could be automated with the PRACE-System.
The operations may be categorized into six operation basic classes:
- Palletizing of parts
- Testing operations (functional testing, optical inspection, type verification)
- Packaging
- Assembly operations
- Machine loading/unloading
- Part feeding
The use cases have been chosen to cover every basic operation class at least once.
Based on thirteen selected use cases emerged following results:
1.5.1.1 Workpiece properties
Most workpieces are not larger than 100...150 mm, although the maximum possible size of dedicated use cases is significant higher. Those use cases are most interesting for the two handed manipulation as a single handed manipulation with the reduced payload of FRIDA would be difficult.
1.5.1.2 Workpiece weight
Typical workpiece weight is less than 100g. That would be the basic requirement for the gripper construction, although the aspect should be regarded to handle higher weights with reduced velocity, i.e. handling of empty pallets which have a weight of 500g up to 2,5kg. It has also to be examined under which conditions the maximum payload of the FRIDA robot of 500g per hand may be exceeded, i.e. to perform seldom required operations like handling of pallets.
1.5.1.3 Workpiece material
Obvious there is no preference towards a special kind of material. However all workpieces have a stable structure, so that no change in shape has to be expected when handling the parts.
1.5.1.4 Tolerances between workpiece and pallet
Typical offset between workpiece and tray lies in the range of 1 +/- 0,5 mm.
1.5.1.5 Additional manipulation related information
Required DOF for object recognition Typical 3 DOF (x,y,rot z)
Box/pallet size 400mm x 300mm / 600mm x 400mm
Weight of box/pallet 500g – 2000g
1.5.1.6 Interfacing
Mainly binary 24V input/output required for - handshake PRACE – equipment to start local process- testing result ok/not ok
MMI user information/input is necessary to coordinate supporting user actions (changing of pallets, providing of parts, packaging of parts).
1.5.1.7 Use case properties
For developing the structure of the AUAs in WP1 a representative use case has to be defined. The use case must provide a broad and typical scenario to prove the adaptability of the AUA structure within its definition stage. Therefore the AUA relevant use case should cover as many as possible of the innovative PRACE features. To determine the AUA relevant use case the innovative PRACE features have been mapped to each use case.
1.5.2 Task 5.2: Multi-Agent based control system
Task 5.2 is entitled “Multi-Agent based control system” and focuses on the development of a Multi Agent System to be applied with the PRACE System.
The work in the task so far has been:
• Development of a suitable MAS concept that can be applied in the PRACE context that both defines the software and hardware implementation of the MAS concept
• Documentation of the developed MAS concept and preparation for discussions with partners in T5.2
In the following the partners of T5.2 will discuss the overall MAS concept as proposed by DTI. The concept has been discussed at several workshops with ULUND and the proposed MAS system concept is currently in prototype development.
After the prototype development, the main required subsystems of the MAS structure have been defined. In the integration phase of the project towards the factory tests these main subsystems have been implemented based on the ROS middleware.
1.5.3 Task 5.3: Software integration for workcell control
The implemented primitives all have defined interfaces that can be utilized using the ROS actionlib mechanisms. These mechanisms are prepared to be used by PLC’s and SCADA systems using e.g. the OPC-UA protocol stack. An additional way of integration can be done over the knowledge database server directly. New AUA sequences can be created and triggered by database access and standard web protocol interfaces that are available in any PLC, SCADA or industrial PC system.
Since a direct integration in a factory-wide work cell control system is not required for the factory tests, the integration will be left as prove of concept state. During the factory tests, the worker will be directly interacting with the robot and therefore the project focus will be on this interaction.
1.5.4 Task 5.4: Integration of simulation environment
ABB FRIDA, the mobile base, the grippers and the tracking system have been completely integrated in a working simulated scenario. Gazebo, the simulation that is part of the ROS system, has been used for this purpose. The physics simulation can simulate sensor signals from the cameras, 3D cameras and laser scanners and also simulated an environment model. For testing the scenarios of the Automatica fair and the factory test the environment was adapted to satisfy the needle scenario that is introduced in deliverable 5.2.
A simulated handle is used to also test the interaction mechanisms between the handle, the knowledge database and the AUA programming system.
As the simulation is available for all partners of the consortium tight interaction on the development during the project is possible.
1.5.5 Task 5.5 Integration of motion capture equipment
Integration of tracking equipment has been started at several of the testbeds. The BOSCH Teach-In-Handle applied to the instruction workflow teaching pallet locations on the workbench, a set of desired gripper positions and respective intermediate move actions.
For tracking the teach-in-handle, different tracking systems were evaluated:
- The teach-in handle tracking software “TIHTracker” developed by Magellium as part of WP2 using a mono camera setup for detecting the colored balls of the BOSCH handle and computing their positions in the image frame, and thus the handle’s 3D position and orientation itself.
- The new open-source Track-It-Yourself (TIY) 3D Marker Tracking Software (see also https://code.google.com/p/tiy/) using a stereo camera setup and triangulation for computing the positions of infrared light reflecting little balls in the scene (see Figure 52).
- The commercial tracking system OPTITRACK, a 6DOF optical tracking device with multiple cameras and capture software also computing positions of infrared light reflecting balls (see also http://www.naturalpoint.com/optitrack/products/v120-trio/). OPTITRACK was used as a ground truth system throughout the evaluation of the Magellium THITracker and the TIY tracking.
In order to track colored spheres using the Magellium TIHTracker, corresponding camera parameters (e.g. gain and exposure) need to be configured strongly dependent on current lighting conditions. Hence, those parameters need adjustments every time lighting conditions are changing. Furthermore, the detection rate of the system dropped significantly in the rather bright environments of industrial facilities. In order to cope with bright environments, actively illuminated colored spheres (see Figure 51) were used in connection with turned down camera exposures to have the illuminated balls to be the only color information in the image frame. In the course of extensive testing, the Magellium TIHTracker software proofs to deliver a more robust tracking performance than the TIY software. Particularly, the TIHTracker software shows less sensitivity to light perturbation (e.g. severe reflections from the surfaces of the palettes due to external IR illumination) and offers a larger workspace for the handle to be tracked in. Hence, with regard to the demonstration at AUTOMATICA 2014 fair and the field tests, the Magellium tracking software is integrated for presenting usage and functionalities of the teach-in handle.
Even though state of integration and performance of the motion capture equipment are satisfying and ready for presentation, there are still a few minor software improvements left to be implemented
in order to increase robustness and reliability of the tracker.
1.5.6 Task 5.6: Lab test
For the lab tests a dedicated room with tables similar to the Bosch factory tests has been build up at Fraunhofer IPA. The development of the robot system has been done in this environment continuously from January to August 2014. For the defined use cases multiple iterations have been done mixing development and testing.
Particularly, testing and evaluation of the motion capture equipment from Section 1.3.5.5 as well as integration of the results from WP 1 and 2 have been done in the course of the lab tests. Palletizing scenarios with different types of needles and different types of traces (see Figure 54) were implemented and tested considering control modalities from Section 1.3.4.5.
With these continuous lab tests large improvements have been possible towards the AUTOMATICA 2014 demo and the field tests.
1.6 WP6 – Test and demonstration
1.6.1 Task 6.1: Specification of scenarios
To determine the field testing scenario the use cases have been ranked according to the following requirements:
- (a) Mapping of all important innovative capabilities of the PRACE system
- (b) Boundary conditions
- (c) System properties fit to PRACE system design
- (d) Agreement of partner from production site who to contribute and support the testing stage
In D5.1 a prioritization of the thirteen presented use cases has been done already to determine which use cases require most of the new PRACE features. The target is to identify those use cases that require a broad set of the AUAs (Assembly Unit Actions). Therefore an evaluation had been done by comparing the amount of the new PRACE features that are mapped in each application.
After taking all those aspects into account the top ranked use case had been the fast palletizing use case that is located in a big Bosch plant near Stuttgart (D). In the initial state of the project a meeting had been arranged at that production site, so all partners are familiar with the use case and the boundary conditions of that application. All those partners who are building up demonstrators of subsystems have been provided with sample parts and pallets for early testing.
As the chosen scenario requires several parallel running manual working places in three shift mode there is a high interest from the production site towards a more economic solution at a high cost location like Germany. Presently there are investigations running to automate this application with a single arm robot system. However this approach is critical due to the high manual output as the worker can perform the double amount of handling operations compared a single arm robot (assuming the same velocity).
Another critical aspect of a single arm solution is the missing ability of handling the emptied/filled pallet. In that case the worker has to periodically change pallets to keep the system operative. Especially when trying to bridge the worker breaks this is a negative aspectsaspect towards high productivity of the single arm robotic solution.
In project year two the integration and testing of the PRACE demonstrator takes places. In that period the application planning for the field testing period will have to be done more in detail and matched with the boundary conditions given by the production site.
1.6.2 Task 6.2: Specification of field test and benchmarks
The aim of the Task 6.2 is to define the metrics and criteria for the field tests that will be executed at a BOSCH factory at the end of the PRACE project. The robot system is developed for a number of scenarios as described in deliverable D5.1 and a selection of these will be executed during the field tests (see task D6.1). Therefore, a number of factors have been specified in cooperation with BOSCH to make field test measurable and comparable to the state of the art in production.
Following list displays all benchmark parameters which will be surveyed later in the field tests.
1.6.3 Task 6.3: Field test
The field tests that have been specified in the DOW were extended during the end of the project. Additionally to the foreseen field tests a demonstrator has been built up to show the project results on the Automatica industrial fair in Munich. During 3. - 6. June 2014 PRACE had a space on the Fraunhofer booth at Automatica (see Figure 56). The dual arm mobile manipulation capabilities were shown as well as the teach-in functionalities. As this demonstration was scheduled 3 months before the actual field tests much improvement from all PRACE WP’s could already be integrated and helpful experiences have been made in order to prepare the actual field test successfully..
In August 2014 two weeks of factory tests have been made as defined in the DOW (see detailed schedule in Figure 57: Time plan of field tests at Bosch plant Feuerbach). During that time, the PRACE demonstrator was used in production at the Bosch plant in Feuerbach, Stuttgart. A needle picking was implemented with the different process methodologies that have been developed within the PRACE project on site. The performance, feasibility and development was evaluated under real life factory conditions.
1.6.4 Task 6.4: Evaluation of field test
Besides the detailed evaluation of the benchmarks defined in T6.2 the experiences during the field tests can be summarized for further improvements of the PRACE system after the project. This experiences are grouped into two groups and are further detailed in Deliverable 6.4.
For the manipulation capabilities, the following experiences have been made:
• With the multimodal control strategies developed in WP4 a stable switching between the control modes could be established. This was a major benefit for the flexible and rapid development of the field test applications. The determinism of the used path planners has to be improved. The results of the path planner sometimes were odd and not optimal.
• Although effort has been focused on improving the performance between the different controllers, there is room for improvement regarding the communication between path planners and the ABB controllers.
The dynamics of platform, e.g. the suspension, influence the precision of manipulation especially during fast movements. This can be bad when doing precise manipulation. An adjustable suspension could be a solution for this problem or an active compensation based on acceleration sensors can be done.
• When using the different tracking systems during the field tests the following experiences have been made:The covered workspace of the tracking systems are too small. Additional measures have been taken to increase the tracking workspace, such as a pan-tilt unit on top of the robot. These measures are expensive and introduce additional challenges for the precision of the overall system.
• The tracking system is very sensitive to lightening conditions. Multiple calibration runs were necessary during the preparations of the field tests.
• The different implementations have abroad scale of accuracy. An analysis of the required precision of the application has a high impact on the effort and costs involved with the tracking system.
For overall performance please refer to D6.3.
During the field test we had 10 parts which were used by each pass through to inspect the surface damages. The results were that no surface damages were visible and the tactile assembly strategy is compatible to the manual process.
Potential Impact:
1. Expected Impact
PRACE aims to develop a new robot learning and execution concept which is able to be trained very fast and effective and which can deal with changing boundary conditions in execution mode. These are key technologies towards an application flexible robot generation which can be used for frequent changing applications with minimal engineering effort. Today’s robot systems are limited on the classical high production volumes with a certain number of different types. The main obstacle for these automation systems towards a broader deployment lies within the high engineering effort to transfer a robot system from one application to another.
The approach in PRACE is not to develop a robot system from the scratch but to use existing, innovative market available components or prototypes and to combine them to a system with complete new abilities. Hardware components are only developed in the case that market available products are not able to meet the requirements. Therefore it is not astonishing that the expected exploitation results from the partners are focusing much on concepts, methods and algorithms. The approach of PRACE is therefore more or less hardware independent and may be adapted to different platforms.
That approach of PRACE will be a big advantage thinking in terms of the future benefit of the results that are expected from the project. As most of the robot suppliers are confronted with a growing saturation of the market for classical robot applications, especially in automotive production, new application fields are targeted, but no breakthrough could be noticed until now. The basic technologies being developed in PRACE might be an important contribution to boost the development of next generation robots with much more flexibility towards the further automation of manual working places. The basic technologies are gathered in the following tables of this deliverable.
It can be expected, that aside the mentioned saturated classical robotic domains a complete new market is already under development. The classical robotic domains cover high automation with short cycle times at high cost locations. Driver is cost reduction. New, upcoming market can be found in two areas:
SME companies that have demand for automation solution, but are not familiar with robotic technology and therefore are still avoiding an investment. However, new suppliers as for example “Universal Robots” are showing that currently about 1000 robots per year are already sold into that segment. The key technology are robots without safeguards, user friendly HMI and a flexible hardware to avoid changeover processes when changing products have to be manufactured.
The other upcoming domain are robots in low cost locations, which are still have strong capabilities with manual assembly. However, due to the rising labor cost the trend already started to start with automation. Famous example is FoxCon, where initially a million robots were targeted. Although that target has been reduced, still a significant number of new robots at high cost locations can be expected. Due to the very flexible manual lines those types of robots will have to meet quite the same requirements as for high cost SME companies.
As PRACE addresses a lot of those upcoming requirements a significant impact on the further development of robot technology towards those new application classes can be stated. In following as a summary the exploitable results are listed. They cover both methods and components that are possible key technologies for the upcoming robot classes.
Due to the still increasing automation trend worldwide it is expected that in terms of societal implications the requirements of education will continuously change. The demand for low skilled workers will be decreasing, while on the other hand more and more qualified workers will be necessary to design, maintain and operate the upcoming automation solutions. The parallel ongoing demographic development will increase the demand for high skilled workers.
2. Exploitable Results
* Task instruction system
USP: Fast and intuitive instruction of new tasks to a robot system
Market: Estimated 500.000 SMEs in EU. Robots like the Universal Robot have introduced simple instruction systems, which have enabled first-time end-users to program the robot for simple tasks within minutes.Currently sales ranges within 1000 robots a year.
* Task execution system
USP: Fast and intuitive execution of new tasks in a robot system without manual coding
Market: please refer to “task instruction system”
* Tactile Gripper System for Automatic Assembly
USP: Compensation of tolerances in automatic handling (pallet deformations, fixture tolerances) by simple, robust system (instead of sensitive camera guided systems) to increase OEE
Market: EU- market for assembly and handling robots: 217.000 robots/a. Market potential for flexible gripper estimated on 10-20%.
Double market potential is estimated in non-robotic handling sector (VDMA: turnover 2011 in D: industrial robot 2,8 Mrd €, handling sector: 5,9 Mrd €)
* Flexible production assistant based on rob@work 3 product (mobile platform)
USP: Full integrated industrial mobile manipulation system, human-robot collaboration
Market covered: Mobile robots, flexible production
* Engineering system for industrial service robots
USP: Multivendor and multi-application engineering system with direct link to the end user
Market potential within European manufacturing industry
* Concept for assembly task definition for non-robot experts
USP: Robot programming done on the shopfloor without programming experts
Expected market:end users of robot systems at manufacturing industry with medium lot sizes
* Calibration method and framework for multi-sensor system
USP: Ease of use by user not expert in vision to establish a vision guiding system for robots
Expected market: about 1000 applications/year.
* Mobile platform with detachable manipulation system
Please refer to comments of “Dual-arm haptics for definition of manipulation tasks”
* Dual-arm haptics for definition of manipulation tasks
This exploitable result has not been described in the initial DoW, but has been developed from ULUND as an additional activity. Therefore no detailed information sheet are provided in this report. Currently negotiations are running between LUND and ABB to clarify the integration of the result into ABB product portfolio.
This is a summary of the final PUD which will is documented in D7.3 (and an updated version is included in the final report).
3. Dissemination activities
In total 10 scientific publications (IEEE, IFAC,..) with the focus on intuitive, skill based robot programming methods has been worked out. This is in accordance with a main focus of PRACE to boost robotics by the development of much more intuitive robot learning approaches.
Another 16 dissemination activities, mainly workshop contributions, had been conducted. The workshops had been selected to address a broad auditorium by the size of the event (ICRA, European Robotic Forum, VDI,...). The event with the probably broadest dissemination impact had been the participation on the AUTOMATICA fair 2014, where the PRACE demonstrator was displayed to a broad publicity, consisting with a high rate of robotic end users.
For more detailed information please refer to the dissemination list included in ECAS – “Final report”.
List of Websites:
www.prace-fp7.eu
