Periodic Reporting for period 2 - SPIRIT (A software framework for the efficient setup of industrial inspection robots)
Reporting period: 2019-07-01 to 2021-05-31
Inspection robots are more and more often used in industry for quality control of parts with complex shape. The main concept is to move a sensor over the surface of the part so that all relevant areas are inspected. The deployment of such inspection robots proved to be difficult, time-consuming and – consequently – expensive. SPIRIT aims at simplifying the deployment of inspection robots by developing a framework that enables the transition from programming of a robotic inspection task to configuring the task. This requires generic solutions for the following key challenges:
• Motion programs need to be generated automatically to ensure full coverage of the part while avoiding collisions. Manual or semi-automatic motion planning for the robot proved to be excessively difficult and time-consuming.
• Small deviations of the part shape invalidate previously planned inspection paths and motion programs, because full coverage can no longer be ensured. Robots need to adapt to the actual shape of the part.
• To fully assess the quality of the part, back-projection is needed that maps the single measurements seamlessly to the 3D model of the part. Only a seamless mapping allows proper data analysis.
• The performance of inspection robots in terms of cycle time and accuracy is not well known. There are very little operational data available that would allow e.g. a system integrator to assess whether a certain application will meet its target values.
The SPIRIT project will consider the following class of robotic inspection tasks:
• The task requires that the sensor is moving continuously (“scanning”) over the surface.
• The sensor acquires 2D or 3D patches of data (“images”, “point clouds”) from the part.
• The sensor is handled by a stationary multi-axis handling system (“robot”).
SPIRIT will take the step from programming of the robotic inspection task towards configuration of a task by importing a new 3D CAD model of the part, selecting the inspection technology and updating the model of the robotic work-cell. This will reduce the specific engineering costs of an inspection robot by 80% and increase the return on investment, thus creating a total market potential of 600-1.000 robotic installations per year mainly in Europe and Asia. A particular impact will be on SMEs, that can reduce the risk of deploying inspection robots and can thus more easily access a market that goes significantly beyond a regional one.
• Motion programs need to be generated automatically to ensure full coverage of the part while avoiding collisions. Manual or semi-automatic motion planning for the robot proved to be excessively difficult and time-consuming.
• Small deviations of the part shape invalidate previously planned inspection paths and motion programs, because full coverage can no longer be ensured. Robots need to adapt to the actual shape of the part.
• To fully assess the quality of the part, back-projection is needed that maps the single measurements seamlessly to the 3D model of the part. Only a seamless mapping allows proper data analysis.
• The performance of inspection robots in terms of cycle time and accuracy is not well known. There are very little operational data available that would allow e.g. a system integrator to assess whether a certain application will meet its target values.
The SPIRIT project will consider the following class of robotic inspection tasks:
• The task requires that the sensor is moving continuously (“scanning”) over the surface.
• The sensor acquires 2D or 3D patches of data (“images”, “point clouds”) from the part.
• The sensor is handled by a stationary multi-axis handling system (“robot”).
SPIRIT will take the step from programming of the robotic inspection task towards configuration of a task by importing a new 3D CAD model of the part, selecting the inspection technology and updating the model of the robotic work-cell. This will reduce the specific engineering costs of an inspection robot by 80% and increase the return on investment, thus creating a total market potential of 600-1.000 robotic installations per year mainly in Europe and Asia. A particular impact will be on SMEs, that can reduce the risk of deploying inspection robots and can thus more easily access a market that goes significantly beyond a regional one.
Through a series of demonstrations and evaluations the developments in the project were iteratively assessed and the next development steps were identified. In the last stages of the project all developments were evaluated on a total of 17 use cases, including 6 robotic workcells, 7 different inspection technologies and 17 different parts with a size of 20cm up to several meters in length.
The main results include:
Offline coverage planning, which is used to plan inspection paths that ensure full coverage, avoid collision and optimize reachability for the robot. The path is planned based on shape primitives and is then morphed to the shape of the part. This makes the resulting scanning path more predictable and better understandable for the user.
An offline global re-planning module, which is used to adapt a previously planned path to the actual shape of the part, based e.g. on a 3D measurement. This can be relevant if the part position varies and for non-rigid parts.
An inline reactive planning module, which adjusts the path during the inspection process. This helps to inspect parts with small deviations in shape where a prior full 3D measurement is not possible.
A generic calibration method for hand-eye calibration of image-based sensors. The method works for eye-in-hand as well as eye-on-base configurations depending on whether the robot is holding the sensor or the part.
An image backprojection method that maps the acquired 2D images to the 3D model of the part. This creates a fully textured model for better visualization. Backprojection is combined with image stitching, to remove any remaining discontinuities between adjacent images and also to make the illumination of the single images more homogeneous to improve the visual appearance.
All of the developments were integrated into one of two components of the overall software framework. The "offline framework" includes the offline path planning, the modelling of the robot, the workcell, the process model of the inspection process and the global re-planning. The "inline framework" includes the calibration procedures, the execution of the motion program on the robot, the reactive path planning and the data mapping.
Exploitation of the results will be mainly through licensing of the resulting software and through its integration into robotic inspection systems. The two main software components (offline and inline framework) can be exploited jointly, but also separately as they both have standardized interfaces that are suitable for data formats typically used in production environments.
Dissemination included several scientific papers, blog entries, newspaper articles and the presentation of the results at (physical and later virtual) fairs. A data set relating to the single use cases is made available in the Zenodo repository.
The main results include:
Offline coverage planning, which is used to plan inspection paths that ensure full coverage, avoid collision and optimize reachability for the robot. The path is planned based on shape primitives and is then morphed to the shape of the part. This makes the resulting scanning path more predictable and better understandable for the user.
An offline global re-planning module, which is used to adapt a previously planned path to the actual shape of the part, based e.g. on a 3D measurement. This can be relevant if the part position varies and for non-rigid parts.
An inline reactive planning module, which adjusts the path during the inspection process. This helps to inspect parts with small deviations in shape where a prior full 3D measurement is not possible.
A generic calibration method for hand-eye calibration of image-based sensors. The method works for eye-in-hand as well as eye-on-base configurations depending on whether the robot is holding the sensor or the part.
An image backprojection method that maps the acquired 2D images to the 3D model of the part. This creates a fully textured model for better visualization. Backprojection is combined with image stitching, to remove any remaining discontinuities between adjacent images and also to make the illumination of the single images more homogeneous to improve the visual appearance.
All of the developments were integrated into one of two components of the overall software framework. The "offline framework" includes the offline path planning, the modelling of the robot, the workcell, the process model of the inspection process and the global re-planning. The "inline framework" includes the calibration procedures, the execution of the motion program on the robot, the reactive path planning and the data mapping.
Exploitation of the results will be mainly through licensing of the resulting software and through its integration into robotic inspection systems. The two main software components (offline and inline framework) can be exploited jointly, but also separately as they both have standardized interfaces that are suitable for data formats typically used in production environments.
Dissemination included several scientific papers, blog entries, newspaper articles and the presentation of the results at (physical and later virtual) fairs. A data set relating to the single use cases is made available in the Zenodo repository.
The SPIRIT project advanced the state of the art in three key topics that are needed to make the implementation of inspection robots more efficient:
• Coverage planning for complex parts and a variety of inspection processes. This was achieved through a generic process model that covers a wide range of image-based inspection processes.
• Reactive path planning to enable inline adaptation to changes of the part. By using sensor information (e.g. coming directly from the inspection sensor) local changes to the robot’s path can be determined and a local re-planning can be made to ensure full coverage of the part while avoiding collisions.
• Seamless mapping of 2D images to a 3D object. This requires the accurate calibration of the whole robotic system, the precise synchronization of image acquisition and robot movements and advanced methods for image stitching, in particular when mapping the images to parts of complex 3D shape.
These developments have been integrated in a two different frameworks:
• An offline software framework that is used for planning the inspection task. This will include automatic coverage and robot motion planning in a 3D model of the robotic workcell. It will also enable the parameterization of the inspection task itself and the modelling of the robot.
• An inline software framework that is used for the actual execution of the inspection tasks on the robot. It will include calibration procedures, the synchronisation of the data acquisition and robot motion, the reactive path planning to adapt to deviation of the part and the seamless data mapping.
The results will enable companies (including SMEs) to set up complex robotic inspection processes at less risk. This will enable companies to offer to technology to a wider market in terms of regions, but a als in terms of inspection technologies, because automated planning methods were previously used only for comparably simple (stop&go or point-like) inspection processes. There are about 2000 companies in Europe that could possibly benefit from the SPIRIT technologies.
• Coverage planning for complex parts and a variety of inspection processes. This was achieved through a generic process model that covers a wide range of image-based inspection processes.
• Reactive path planning to enable inline adaptation to changes of the part. By using sensor information (e.g. coming directly from the inspection sensor) local changes to the robot’s path can be determined and a local re-planning can be made to ensure full coverage of the part while avoiding collisions.
• Seamless mapping of 2D images to a 3D object. This requires the accurate calibration of the whole robotic system, the precise synchronization of image acquisition and robot movements and advanced methods for image stitching, in particular when mapping the images to parts of complex 3D shape.
These developments have been integrated in a two different frameworks:
• An offline software framework that is used for planning the inspection task. This will include automatic coverage and robot motion planning in a 3D model of the robotic workcell. It will also enable the parameterization of the inspection task itself and the modelling of the robot.
• An inline software framework that is used for the actual execution of the inspection tasks on the robot. It will include calibration procedures, the synchronisation of the data acquisition and robot motion, the reactive path planning to adapt to deviation of the part and the seamless data mapping.
The results will enable companies (including SMEs) to set up complex robotic inspection processes at less risk. This will enable companies to offer to technology to a wider market in terms of regions, but a als in terms of inspection technologies, because automated planning methods were previously used only for comparably simple (stop&go or point-like) inspection processes. There are about 2000 companies in Europe that could possibly benefit from the SPIRIT technologies.