Final Report Summary - THERMOBOT (Autonomous robotic system for thermo-graphic detection of cracks)
Executive Summary:
THERMOBOT investigated the automation of crack detection in parts of complex geometry and large size. The current procedure for crack detection is a process that dates back to the 1920s and is called “magnetic particle inspection”. This method is infamous in industry, because it is a cumbersome, dirty process that is often done manually even in otherwise fully automatic production lines. THERMOBOT aimed at replacing this old method for crack detection with a new technology that is based on autonomous inspection robots using thermography to recognize cracks on parts of complex geometry. The robot scans the whole part with a thermo-camera and analyzes the heat-flow to find cracks and other defects hidden under the surface. To achieve this, the project targeted the following three objectives:
• (A) development of a thermographic process and a process model for the automatic detection of cracks in parts of complex geometry, varying surface structures and for different materials.
• (B) development of an automatic path and motion planning module that uses the thermographic process model of objective (A) to automatically generate a path for the inspection robot from 3D CAD data. Path planning also requires a dynamic (on-line) adaption of the path to account for variations in the thermal properties and surface structures of the part.
• (C) investigation of thermo-image analysis methods that not only aim at detecting cracks and making an accept/reject decision for the whole part, but also have the capability of self-evaluating the performance of the crack detection.
In the project two different demonstrators have been set up for different types of parts and materials. The first demonstrator uses laser-based thermography for metallic parts. The second demonstrator uses flash thermography and is used for inspection of carbon fibre parts.
Project Context and Objectives:
Despite considerable progress in the automation of production processes visual inspection of products and parts is still done manually in a wide variety of inspection tasks. This project aims at the automation of crack detection in parts of complex geometry and large size. The current procedure for crack detection is a process that dates back to the 1920s and is called “magnetic particle inspection”. This method is infamous in industry, because it is a cumbersome, dirty process that is often done manually even in otherwise fully automatic production lines. The component to be tested is magnetized before applying a suspension of finely divided coloured or fluorescent magnetic particles. Cracks or inclusions cause the magnetic flux to break the surface forming free magnetic poles and the magnetic particles will collect at these locations indicating a crack. Using UV-light the fluorescent particles in the suspension are activated to increase the visibility of the cracks (see figure below). Inspection of complex parts is usually done manually, whereas for simple geometries machine vision systems can be used for automatic detection.
This (still) state-of-the-art process is undesirable for a number of reasons:
• the process is time-consuming and involves several non-productive process steps, such as magnetizing and de-magnetizing the parts and spraying them with a suspension that has to be removed later in many applications.
• the liquids used for the process are ecologically undesirable. They are usually petroleum-based or water-based with corrosion-inhibitors added in the latter case. The magnetic particles need to be replaced frequently as they stick to the part.
• the manual inspection process is expensive, time-consuming and unreliable. It also has problems with documentation, which is often a requirement, especially in the aerospace industry.
It should be noted that inspection has to be done not only as a final production step before delivery, but also takes place throughout the lifetime of the product, when repairs have been made or for regular inspection, e.g. in aerospace applications.
THERMOBOT aims at replacing this old method for crack detection with a new technology that is based on autonomous inspection robots using thermography to recognize cracks on parts of complex geometry. The robot will scan the whole part with a thermo-camera and analyse the heat-flow to find cracks and other defects hidden under the surface.
Up to now such robotic inspection systems using thermography cannot be implemented, because thermography is a dynamical process. The thermal properties (conductivity, emission) of the material and the image acquisition process itself need to be considered when defining the trajectory for the robots. Manual offline-programming of the robot’s path or programming by demonstration cannot capture these properties. It thus requires a model-based approach as aimed for in THERMOBOT.
These scientific goals of the project correspond to the technological objectives:
(A) A full, parametric model of the thermographic imaging process has to be developed that is sufficiently quick to be calculated in real-time. The model will consider emission coefficients for different surface properties, reflections from nearby surfaces and artefacts that the image acquisition process may produce. It also has to include the fact that multiple images of the same spot have to be taken at different times to fully analyse the heat flow.
(B) Robot path planning has to be fully autonomous based on 3D CAD data. Using the process model, the path planning module needs to generate a sequence of viewpoints (robot poses) that create a series of overlapping images so that each spot on the part is visible in multiple images. Path planning also has to consider that the robot’s motion is continuous (not a start/stop motion for each viewpoint) and has to reach each viewpoint at a specific time. In addition to this, methods for dynamic adaptation of the robot’s path need to be developed that account for the specific properties of each part and ensure full coverage of the part as well as optimal inspection quality.
(C) Image analysis not only needs to detect the defects (which is standard today), but needs to include generic knowledge about the image acquisition process and learning capabilities so as to distinguish between artefacts such as reflections (that often occur on complex shapes) and real defects. It also needs to include a self-evaluative component that predicts the quality of the detection by analysing the expected contrast based on on-line simulation of the heat flow and the thermographic imaging process. Image analysis will provide the input data for the automatic accept/reject decision.
The overall project plan is structured along a series of integration tests and demonstrators.
As planned in Annex I, two different demonstrators have been set up. The first one is located at Profactor. It uses laser-based active thermography and is mainly used on the crankshaft (metallic) inspection task. The second demonstrator is based at TRIMEK and uses flash thermography for the inspection of the carbon fibre parts. Originally it was planned, that both demonstrators would cover the same (whole) spectrum of parts, but the substantially different material properties led the consortium to use different technologies for the demonstrators so that all test cases can be adequately covered.
In month 36 the demonstration included static path planning as in demonstrator #2. In addition, it provided a dynamic component that modifies the robot’s path for each part so as to locally optimize image quality. This required feedback from the image analysis component that used an on-line simulation model to predict the contrast with which faults would be visible in the images. To demonstrate the autonomy achieved by the robotic inspection system, it was tested on 3 new parts (2 metallic, 1 composite), which demonstrated its capability to perform the inspection tasks fully autonomously. Re-calculation of the path for a new (previously unknown) took less than 15 minutes.
The bold properties indicate the key features that were added to the demonstrator shown and described in month 18. Not all the features are present in both demonstrators, because the carbon fibre parts inspection uses start/stop motion, where online simulation, contrast optimization and dynamic replanning are not possible. These features are only present in the crankshaft inspection demonstrator.
Project Results:
WP1 - Test case definition, general concepts
The objectives of this workpackage are to develop general specifications for a thermographic inspection robot, to define the test cases and test parts and to collect sample parts for testing.
Task 1.1 Selection of test cases
The selection of test cases was completed by the end of month 6. The result of this selection process is documented in deliverable D1.1 prepared by BSCT and BRP. The test samples for these test cases were used by BAM for doing the CT measurements and by Profactor for identifying the right specifications for the thermo-imaging equipment (camera, laser).
Task 1.2 Collection of labelled sample parts
During the first 18 months BRP collected and labelled a set of crankshafts that have been exploited for further testing during the integration tests and the demonstrations. These parts were taken from the “scrap” parts collected during quality control and were documented and labelled to make them available to the other partners.
BSCT produced a set of the small sample parts (CFRP) for the CT measurements and a few samples of the Audi R8 sideblade, which will serve as demonstration part in the ThermoBot project.
After initial experiments have been performed, both BRP and BSCT collected another (final) set of labelled sample parts for the different inspection tasks. Sample parts of BRP were sent to Profactor, while the parts of BSCT were sent to both TRIMEK and Profactor. All the sample parts were assigned a specific ID that allows the tracking of the parts throughout the project. A list of these sample parts is maintained by the project manager that includes the parts, defects to be found, their ID and their current location. In the course of the second half of the project these sets of samples were extended as part of the assessment and demonstration activities in WP5 and WP6.
Task 1.3 Definition of accept/reject criteria
BSCT and BRP provided information about the accept/reject criteria to be used for their particular applications. In general, the specification of the criteria is not made on a tight numerical basis, but leaves room for subjective interpretation, because up to now the inspection has been done manually by experts. The criteria have been integrated in the overall concept by Infratec (D1.2) and were later added to the documentation in of the sample parts in deliverable D1.3.
Task 1.4 Generally applicable concepts for inspection
In this task, Infratec had the lead in preparing a document that identifies generally applicable concepts for thermographic inspection of parts. Many partners contributed to this document, including ITRobotics, TRIMEK, and Profactor. The main results that emerged from these discussions are that:
• Quantum cameras are required for robotic inspection tasks (bolometer is not an option due to the motion and frame rate that is required).
• In many cases the robot will hold the part rather than the thermo-imaging system, because of the cables and cooling system required for the laser and the camera.
• Lasers are the only reasonable heat source for the inspection of whole crankshafts, due to the complex geometry of the parts.
• Flash thermography seems to be more suitable for more or less flat parts such as the sideblade.
• In terms of robotic handling systems high flexibility is needed, and the ThermoBot framework should not be limited to particular kinematic structures.
During the preparation of the document it was also found that it is useful and feasible to design the system software in such a way that a generally applicable concept for inspection robots is obtained, without being limited to thermographic inspection. The general concept includes 6 modules: process model, coverage planning, path planning, synchronisation, image analysis and decision making. Interfaces between these modules can be defined in a quite generic way that allows the exchange of single modules without affecting the others.
WP2 - Thermographic process model
The objectives of this workpackage are to develop a parametric process model of thermographic image acquisition and to perform comparison measurements with CT and a statistical assessment of the reliability.
Task 2.1 Process modelling based on enhanced CAD data
Profactor, with support by UNIPD, developed an initial physical model of the thermo-imaging process. The results are described in deliverable D2.1. While the physics of the thermodynamic process are well known, work also included experiments with lasers and thermocameras to identify optimal operating characteristics. At Profactor a laser was rented for 2 months to perform initial tests. The initial result was a set of specifications for the laser that was bought for the inspection system in the project. A particular challenge was the wide variety of thermal material properties of the test cases. Metallic parts (crankshafts by BRP) differed almost by an order of magnitude from the CFRP parts (R8 sideblade BSCT) in terms of emission coefficients.
The initial process model has been implemented and integrated into the first versions of the path planning module. Initially, the process model used a simple, phenomenological model that consists of a ring-shaped area around the centre of the laser spot. This ring identifies the area that can be checked from a single viewpoint of the camera when looking down perpendicularly on a flat surface. The next step was to map this ring onto the CAD data of the test parts and to reduce the area by considering the visibility, edges and corners on the part. This work was done by Profactor.
Based on the experiments for the POD and ROC characteristics, the model could be further refined, to include a more realistic model of the inspectable region. The dependence of the shape of this region on the speed for the robot was included and the exact area was determined for the surface properties of a crankshaft.
An additional model was required for flash thermography. A series of images have been taken by TRIMEK at the end of the reporting period. These images will provide the basis for the development of the flash-thermography model. Between month 19 and month 24 Profactor and Infratec jointly developed the pulse phase thermography process required for the carbon parts inspection on the TRIMEK demonstrator. The main goal was to inspect an adhesive layer between two components of the sideblade. The adhesive layer is distributed along the edge of the side blade and varies in width, depending on the process. This inspection task was defined as a consequence of the results achieved between month 12 and 18, where the detection of the smaller defects, such as inclusion or gaps turned out to be infeasible within the scope and limitations of ThermoBot.
Development work was mainly focused on first identifying pulse-phase thermography as a good method for solving the inspection task and then tuning the process parameters to optimize the contrast of the adhesive layer in the images. A laboratory setup at Profactor was used to develop a process with substantial support from BAM and Infratec. The most promising process was based on pulse-phase-thermography (PPT) using 4 flashes of 6kJ energy each at a frequency of 0.04Hz. This process was later implemented by Profactor at TRIMEK, but with just a single flash. At the general meeting in month 24 of the project it turned out that no budget was available for additional flashes and that the TRIMEK robot could not carry four flashes. The image acquisition and PPT analysis was implemented by Profactor and transferred to TRIMEK. Infratec substantially supported the task using their IRBIS software.
Task 2.2 Experimental evaluation of tolerances during acquisition
During the setup of the demonstrator at Profactor, two main sources for uncertainties have been identified:
• Temporal uncertainties related to an unknown offset between the recording of the thermoimages and the position of the robot.
• Spatial uncertainties related to the transformation between the many coordinate systems.
For the temporal uncertainties, a significant amount of work was done to acquire time-stamped robot joint angles and high frequency and combined with time-stamped frames coming from the thermo-camera. Given the maximum speed of 100mm/s, the readout rate of 4-8ms for the robot joint angels and acquisition rates of up to 200Hz (with a resolution of 320x240) were found to be sufficient. This also considers that the image processing algorithms will have to adaptively locate edges of the parts in any case.
A second series of experiments has been conducted by Profactor to obtain quantitative data and to further improve the laser-based thermography process with the aim of extending its flexibility with respect to speed, laser power and angle between laser, camera and surface. This additional flexibility will be a key step towards further optimising the path planning.
Task 2.3 CT measurements and matching to thermography data
The selected test samples of BRP and BSCT have been evaluated by BAM according to the needs for CT reference measurements. Using the simulation program aRTist, optimal measurement parameters (X-ray energy, scanning parameters, prefilter, etc.) have been selected to detect the different types of defects. Test samples with a diameter of 15 mm have been prepared for samples made of steel and with a diameter of 120 mm for samples made of composite material. CT reference measurements of 7 sample parts from BRP and 4 sample parts from BSCT have been performed with BAM μ-CT 225 kV X-ray source. 3D-CT volume data have been evaluated according to defect parameters (length, width, etc.) for matching with thermography data.
Furthermore, the thermography group of BAM also performed experiments to generate images from a few selected metallic test parts (crankshaft). By aligning and transforming these images a match to the CT data could be made. This included (after aligning, transforming and thresholding) a pixel-wise comparison between the images. From this comparison the sensitivity (does it detect all defect pixels) and specificity (does it detect only the defects) could be calculated. This initial evaluation done by BAM was the basis for the statistical analysis (task 2.5).
This work was completed before month 18. Experiments were done to compare magnetic particle inspection (Magnaflux), thermography and computed tomography.
Task 2.4 3D scanning measurements and matching to thermography data
The 3D scanning system has been prepared for inspecting the crankshaft pieces provided by BRP and the composites pieces provided by BSCT. The scanning system consists of a high accuracy non-contact optical sensor (OptiScan), specific software to process the information obtained in the scanning process and of the Stäubli robot, all installed and used at TRIMEK. Moreover, the pieces provided have been scanned at high speeds (up to 100.000 points per second) and all types of details on the parts’ surface, from free-form shapes to fine details have been captured.
Matching to thermography data did not reveal any correlation between the defects visible in the thermographic images and in the 3D scans. Therefore, this task was finished. The 3D scans, however, provide a basis for generating an inspection path for the robot in cases where no CAD data exist. This work was completed before month 18.
Matching to thermography data did not reveal any correlation between the defects visible in the thermographic images and in the 3D scans. The 3D scans, however, could provide a basis for generating an inspection path for the robot in cases where no CAD data exist.
Task 2.5 Statistical Analysis: POD and ROC
At Profactor a series of experiments were done to vary process parameters of the thermo-imaging process. Laser power and the robot’s speed were systematically adjusted to identify the range of possible process parameters and to provide a data basis for the POD and ROC characteristics. BAM used the resulting data to generate the respective statistics (Deliverable D2.3). The statistical analysis was made significantly more complex by the fact that thermography also includes software algorithms for processing the images.
A useful procedure for evaluating the algorithm parameters could be developed with the help of UNIPD. The results of the analysis revealed that the probability of detection follows the typical (expected) curve if laser power is chosen as parameter, while it does not show a well-defined behaviour if the processing speed is used as parameter. ROC analysis showed that all 3 methods (CT, magnetic particle inspection and thermography) are able to identify the characteristic properties of a defect.
CT clearly provides the most complete data, but there is no doubt that thermography has a good potential of replacing magnetic particle inspection.
Only one of the crankshaft sample parts with a “microstructural inclusion” produced inconclusive results. It generated a signal in magnetic particle inspection, but neither CT nor thermography were able to reproduce this signal. This particular sample will be further investigated, probably by cutting it and inspecting it under a microscope.
This work was completed before month 18 and the results are documented in deliverable D2.3.
Highlights of most significant results
Thermographic processes for metallic parts inspection (laser-based) and carbon fibre parts inspection (flash thermography) developed and implemented; experiments for improved process model completed; comparison with CT completed; POD and ROC analysis finished.
WP3 - Robot path planning
The objectives of this workpackage are to develop methods for autonomous static path planning based on 3D CAD data, to develop dynamic path planning methods based on visual feedback and to develop on-line simulation tools for collision detection and work-cell simulation.
Task 3.1 Implementation of the process model
The implementation of the initial process model, based on a very rough analysis, consisted of a ring-shaped, circular region around the laser spot. This area was projected onto the CAD considering edges and the local geometry of the part to reduce the ring to the “inspectable” area.
Based on the experiments done for the POD and ROC analysis, the model was further refined. The main parameters that were added to the model were the laser power, the processing speed and the actual, intrinsic camera parameters. The ring-shaped region changes its shape substantially when the laser is moving across the surface. With increasing speed, the “backside” of the ring becomes more elongated, while the front side is compressed. This may proceed until no inspectable area remains on the front side and the ring finally turns into a crescentic shape.
Camera calibration was done to obtain the intrinsic parameters of the camera, such as the centre of the optical system on the chip and the actual focal length (see next section).
This task ended according to plan in month 15.
Task 3.2 Static path planning based on enhanced CAD data
The first step in path planning is to generate a dense 3D point cloud of the part, where each point is augmented with a vector pointing outward. The next step is “coverage planning”, which tries to identify an (ideally) minimal set of viewpoints needed to fully cover the area to be inspected. This is done in parallel with the inverse kinematics solver that checks the reachability of these points. In a further development step the path that was generated also considered the kinematics of the robot in order to avoid that the robot often switches between different configurations thus wasting time during the inspection. Finally, close to the end of the reporting period, the path that covers the whole part (generated by Profactor) was imported into the work cell simulator of IT+Robotics to create collision-free path that fully covers the part.
Tests showed that the overall inspection time was a tough challenge. Given the maximum processing speed (speed with which the laser spot moves across the surface), cycles times of less than 1 minute are very difficult to obtain even with an optimized path. IT+Robotics designed an algorithm that finds the optimal (less time consuming) zone visiting order. The computation finds the trajectories that permit to the robot to pass from a zone to another as well. Regarding optimization, IT+Robotics developed a plan to split the part into different zones and generate optimal paths for each zone. This takes advantage of using optimized solutions for particular geometries such as cylindrical shapes or flat areas. Nevertheless additional measures are required and the partners believe that improvements can be achieved by widening the inspectable area by using the scanning system in the laser that would e.g. allow superimposing quick oscillations in the laser spot to create a larger spot.
In the period up to month 24 the “coverage planning” for the laser-based thermography was finished. Path planning was implemented by Profactor in collaboration with ITRobotics. The first step in the planning process is to generate a dense 3D point cloud of the part, where each point is augmented with a vector pointing outward. The following step is “coverage planning”, which tries to identify an (ideally) minimal set of viewpoints needed to fully cover the area to be inspected. This is done in parallel with the inverse kinematics solver that checks the reachability of these points. It also has to be avoided that the robot often switches between different configurations thus wasting time during the inspection. By splitting the part into different zones that are then processes separately and linked together a substantial simplification of the coverage planning could be achieved and it also gave rise to some optimization potential in terms of inspection speed.
The second part of the path planning is an iterative process, where collisions are checked and the path is adapted in order to ensure that no collisions occur, while still fully covering all relevant areas. This iterative process includes a data exchange between the modules of Profactor and IT+Robotics. The overall iterative process was implemented by IT+Robotics. These activities had strong links to task 3.3.
Tests showed that the overall inspection time is a tough challenge. Despite all the optimizations that were done, the target of 20-30 seconds for the crankshaft remained out of reach (see also assessment WP5). Theoretical calculations using the maximum surface speed of the laser spot (188mm/s), a maximum spot size of 5mm and a spiral-like motion along a cylinder with the size of crankshaft showed a scanning time of 2.4 minutes. More realistic values indicated scanning times in the range of 5-6 minutes.
Task 3.3 Offline Workcell simulation and robot program generation
IT+Robotics set up the workcells of both demonstrators in their simulation software. The workcell contained the main elements (robot, laser, camera, part) and their relative positions and was a virtual duplicate of the demonstrators set up at Profactor and TRIMEK. The final design was finished around Month 15, when the robot had been installed at Profactor and the first demonstration was finished so that the whole setup was defined. The TRIMEK demonstrator was modelled later, at the end of month 17, when the setup was finished after Profactor had completed the first integration at TRIMEK.
The overall software structure that is based on TRIMEK’s existing concept for their measurement systems and robots, was further developed with IT+Robotics, TRIMEK and Profactor. A generic interface to the TRIMEK robot was developed that allows the specification of (a sequence of) robot positions coming out of IT+Robotics’ workcell simulator software. This software was also extended to directly generate robot programs for both types of robots (Fanuc at Profactor, Stäubli at TRIMEK).
In the second period IT Robotics finished the robot program generation for the Fanuc robot that is used in the project for demonstration and the COMAU robot that is used for testing at UNIPD and IT+Robotics. The workcell simulation was update with the latest version of the robotic workcell and linked with Profactor’s coverage planning. Whereas at the beginning of the period these were still separate tools that communicated via file interfaces, they have now been integrated into a single tool that performs path planning, work cell simulation, collision detection and avoidance and robot program generation.
Substantial effort was spent by IT+Robotics to improve the workcell simulation. A fully updated version became available on 27th June 2014 (month 30). This version was also used during the 3rd integration test. Features that have been added that mainly relate to the communication structure between the robot controller (Profactor/TRIMEK) and to processing of the feedback provided by the online contrast optimization (Profactor).
Additional work was required to make transformations between the robot and the part available for each image. These transformations are needed for the projection of the identified defects from the 2D thermo-images onto the 3D CAD model of the part.
Task 3.4 Dynamic path planning and contrast optimization
This task was started in month 14 with a slight delay accounting for the fact that the robot was delivered late. In the following months, up to month 18, a concept was developed about how dynamic path planning could be done. The agreement between IT+Robotics and Profactor was that dynamic path planning would not require the global, on-line re-planning of the path, which is neither necessary nor feasible. As described in the DOW, dynamic path planning accounts for variations in the material, changes in the surface properties and small deviations from the ideal shape of the part. It was therefore decided to use a set of registers in the robot’s control unit to define changes (offsets) for the single joints of the robot. Some higher-level control algorithm deal with the accumulation of these offsets, e.g. if a flat area is inspected and the inspection path is adjusted 0.2mm to the right, then all other elements of the path that relate this area also have to be shifted by that amount. The separation of the part into different zones as described in task 3.2 helps in this respect as it decouples the whole part into a set of smaller, independent regions.
The Fanuc robot at Profactor (and its control unit) has been chosen to allow this kind of on-line adjustment, while the Stäubli robot at TRIMEK does not have that kind of capabilities and the dynamic adjustment of the path could not be implemented there. The dynamic on-line adaptation of the robot path could be tested on the third workcell of the Thermobot project: the one installed at UNIPD. The controller of the COMAU Six industrial robot was improved by adding the COMAU C4G Open feature to the controller. This enabled a 1ms control cycle and allowed a direct control of the robot trajectory from a high-level software application like Workcell Simulator by IT+Robotics. Thus, even if the two robots used in the Thermobot demonstrator are not ready for dynamic on-line adaptation of the robot path (like most of the state of the art industrial robots), this innovative feature of the Thermobot project could be tested using the workcell of UNIPD. This will need to be further developed before being ready to industrial integration; such activity could be performed once the next generation of industrial robots will be fit with the new dynamic path adjustment functionality. These robots are expected to massively hit the market in the next few years: most of the leading industrial robot manufacturers have prototypes or top series robot with this functionality.
The goal of dynamic path (re-)planning and contrast optimizations is to ensure that the inspection process is able to detect defects even when there are variants in the surface properties or geometry of the part. Based on the results achieved during the tests at Profactor, it turned out that the following strategy is most suitable for the inspection process and more likely to resolve issues with the image contrast: the module for contrast optimization (see task 4.5) provides an output indicating whether the contrast in the image is still suitable for inspection. If contrast is too low, the robot stops, acquisition parameters (e.g. laser power) are adjusted and the robot returns to last position, where the contrast was still good enough and continue the inspection from that position. This requires the quick re-planning of a safe and reachable path from the current (stop) position to the previous (re-start) position and online simulation (see task 3.5).
Development work of Profactor was mainly concerned with the synchronisation of the “stop” signal and the robot positions. This synchronization is needed to determine the last “good” position with high accuracy. This is to avoid that the robot unnecessarily scans areas of the part that have already been successfully inspection.
IT+Robotics managed the communication with Profactor libraries using Protocol Buffers. When the trajectory is interrupted, IT+Robotics path planner module plans a feasible path between the current robot position and the last valid one. Then it continues the inspection from here. If possible, the path planning module tries to keep the same robot configuration to minimize the cycle time. Otherwise, the best joint configuration is used, taking in account the robot dynamics.
Task 3.5 On-line simulation of robot work cell
In order to allow dynamic adjustment of the path, collisions and the kinematics of the robot need to be evaluated to check the feasibility of certain adjustments in real-time.
IT+Robotics developed an algorithm of motion planning that permits to calculate a safe robot path. It means that robot motions do not produce a path that passes too close to the obstacles, considering also speeds and accelerations of each joint. In this way the robot has also more space in case of dynamic re-planning in order to optimize the contrast.
Moreover, to speed up the performance of the motion planning in order to satisfy on-line real-time constraints, the collision engine was modified to allow multithreading computation. The parallel approach allows taking full advantage of the new multi-core processor chips technology. In fact, the computer processors technology is increasing the number of CPUs on chip. It means that the system will reduce the computational time over the time.
IT+Robotics made substantial progress with respect to online simulation of the whole workcell and the robot. During the integration and demonstration in month 23 the simulation was able to show a real-time behaviour and to detect collisions in real-time.
This activity led to research work on data structures necessary to parallelize the virtual rendering engine. Moreover, motion and path planning have been revised and tested to improve robot planning performances.
During the last integration week at Profactor (Steyr, month 33), the interaction between the new software modules and the demonstrator was successfully tested.
Highlights of most significant results
Workcell models and simulation finished; process model implemented; coverage planning operative; robot program generator operative for Fanuc and the Stäubli robot.
WP4 - Thermographic image analysis
The objective of this workpackage is to develop thermo-image analysis methods for the segmentation and classification of defects and to develop methods that assess the reliability of the detection based on image contrast.
Task 4.1 Forward/backward projection of thermo images
A key element of this step is the geometric calibration of the camera. The calibration provides the transformation from the camera pixels to the world/part coordinate system (or vice versa). In (standard) machine vision calibration often uses checkerboard patterns in order to identify the projective transformation and any distortions that might be present. For thermo-images a checkerboard pattern has also be used, although with some modifications. A checkerboard pattern of dark and light squares that covers the whole field of view was produced by Profactor. This checkerboard pattern was briefly heated by a lamp, the dark squares heat up more quickly than the other ones. This generated quite clear images and reduced the usually blurred edges in thermo-images, because this process generates the necessary contrast to identify the corners of the squares that serve as main feature points for the calibration. Nevertheless, some post-processing was required to make the images suitable for the use with the standard calibration function of the Open CV library. UNIPD performed also calibration experiments with a new type of checkerboard: a calibration body that consisted of a regular structure of circular holes in a metal part was manufactured by BAM. By putting a diffuse heat source in the other side of the metal square is possible to image with the camera a matrix of hot spots with known positions and use this as the matrix of the corner of the checker board in the usual calibration process. This approach proved to solve the problem of low contrast edges generated by a paper checkerboard in thermographic images. This calibration body has been successfully used by UNIPD for calibrating the thermographic camera. This calibration provides all elements required for calculating the projection of the defect onto the CAD model of the part.
The second element is the extrinsic calibration that localizes the camera in the world and in particular relative to part/robot. This calibration process was done by Profactor by using a stylus (or needle) mounted on the robot in a known position relative to the robot’s tool centre point. By placing the stylus in the corners of the checkerboard pattern that is recorded also by the camera, the coordinate transformations between the robot, the part and the field of view of the camera could be established.
This set of transformations creates the full chain from the pixel coordinated of the image to the 3D CAD coordinates of the part. What has been neglected in this process is to establish the exact position of the part relative to the robot. This, however, is an industrial standard procedure that need not be further investigated in a research project and the consortium decided to skip this step by mounting the part in a fixed position on the robot (no grasping of parts).
The main result of this task is a set of transformations covering the full chain from the pixel coordinates of the image to the 3D CAD coordinates of the part. Using calibration methods that have been described in the previous reporting period, these transformations could be established and the main work related to that task was finished by month 18. However, during the integration on the demonstrators several topics still needed to be addressed by UNIPD in cooperation with Profactor. This included technical issues such as software bugs at the interfaces, but also clarifications about the use of the transformations. Work in this task thus continued with lower intensity up to the final stages of the integration.
Task 4.2 Adaptive and knowledge based image segmentation
Initially, developments in this task were delayed due to the lack of thermo-images, because the laser heating system was not available until month 12. After month 12 several image sequence were acquire with different process parameters that were used as input for the image segmentation. The first step of the segmentation process developed by UNIPD is to classify the whole image as “contains a defect” or is “free of defects”. This was done by analysing the shape of the laser spot. The key observation is that the laser spot loses its slightly elongated, elliptical shape, whenever there is a defect in the vicinity of the spot. Labelling of the images was done based on ground truth data coming from magnetic particle inspection and confirmed by CT measurements. Concerning the evaluation of the cracks, a basic implementation was done that builds upon finding high temperature gradients in the image. Further developments aim at using a sequence of consecutive images to further improve detection also in areas of lower contrast. This will also help lowering the cycle time of the overall inspection process. Also the issue of edges on the part or abrupt changes of surface properties (rough vs. milled) have still need to be included in the knowledge-based segmentation.
Regarding carbon fibre parts, Flash Thermography was chosen as an alternative technique and the necessary hardware (camera) was sent to TRIMEK early in month 13. Due to the fact that a serious defect in the camera (caused during transport) was not discovered until month 16, the TRIMEK system, hardware and software for the inspection of carbon fiber parts was available only in month 18. A preliminary set of image sequences were obtained after generating a heat pulse (generated by a flashlight) impacting the surface of the part. The diffusion of the heat inside the part results in a distribution of temperatures in the time sequence. After initial experiments some sets of images were produced, unfortunately the known defect in the CFRPs did not show up in the thermal images. From the thermographic images acquired in the TRIMEK demonstrator with carbon fiber parts, it turned out that it is not possible to detect defects that are inside the part, but only superficial ones; however, the capability of detecting defects that are inside the inspected parts would be a strong advantage of the Thermobot system over traditional inspection systems. To keep this advantage, pulsed-phase thermography (PPT) was chosen as an alternative inspection method. This choice turned out to be crucial, as PPT images are able to highlight the glue jointing together adjacent layers of carbon fiber.
For the laser-based thermography used in the metallic part inspection demonstrator at Profactor image processing was performed by segmenting the laser spot, analyzing its shape, extracting a set of descriptors to measure the shape itself, and feed a classifier-
The key observation is that the laser spot loses its slightly elongated, elliptical shape, whenever there is a defect in the vicinity of the spot. The detection is based on a classification method that uses a few features to characterize the shape of the laser spot. Cracks and other inhomogenities change the circular shape of the spot and thus enable the detection.
During the project, several different methods for measuring the shape changes of the laser spot were tested and compared. The first algorithm employed, Radial Density Profile (RDP), was thoroughly tested. Further tests were run using other texture measurement features, like Pyramid Local Binary Pattern (PLBP) and Morphological features. The features were also exploited to build an ensemble, which was then used to feed a classifier.
In order to deal with reflections and other heat spots on the part that may occur during the inspection process, information about the expected position of the laser spot in the images is used. This allows the identification of the spot that is coming directly from the laser in contrast to e.g. reflections.
Regarding the detection of defects in CFRP parts, after resolving the open issues related to carbon fibre parts inspection and successfully setting up the acquisition process together with the PPT pre-processing steps, a first set of images was available for the development of the algorithms. A second larger set was originally planned for month 25, but was only acquired in month 31, so that there was the need to substantially revise the algorithms late in the project.
The final algorithms detect defects in the bonding layers by comparing the inspected part with a given reference, that is considered defect-free. The system exploits a set of filters for removing the high texture content and highlight the regions where the gluing layer can be found. The algorithm is capable of highlighting a set of Regions Of Interest (ROIs) on which the following steps of the algorithms will focus. The ROIs of both the inspected part and the reference are compared, and mismatches are highlighted.
Task 4.3 Defect classification and decision making
In D1.3 the end-users BRP and BSCT identified the outline of criteria that are to be used for the automatic classification of the defects and for the final accept/reject decision. Defect classification has been implemented by UNIPD using a linear SVM (Support Vector Machine) classifier exploiting the features developed in Task 4.2 as input. The best results were obtained using the Local Binary Pattern (LBP) feature evaluated on the image region around the laser spot. Such region of interest is rescaled to a fixed dimension in order to keep the dimension of the LBP descriptor constant. A classifier was trained on these images to detect the cracks. The SVM classifier was trained following a leave-on-out protocol, and cross-validation has been performed during the training phase. Experimental results showed an area under the ROC curve of 0.9337 indicating a very good classification performance. Experiments involved the main type of defect of metal sample parts, namely the crack. In principle, this approach can be extended to other kinds of defect by running again the training of the SVM classifier, even though modifications of the segmentation and texture analysis algorithms could lead to performance improvements.
Given the lack of thermographic images showing defects, to further develop image processing algorithms, especially for CFRPs, UNIPD decided to look for other image datasets which could provide images similar to the ones expected by the two Thermobot demonstrators. Similar images were identified in medical and biological datasets and new algorithms for image classifications have been developed on those images. Moreover, UNIPD also explored the possibility to generate thermographic image using induction heating in the workcell installed at UNIPD. Because of security issues, both laser and flash generated images could not be implemented in the lab at UNIPD, but induction heating using small probes does not pose these problems and was considered as a third (or forth considering also pulsed-phase thermography) excitation source.
Additional sample parts were made available by BRP, so that a set of 12 whole crankshafts exists, for which full image sequences have been generated. A classifier was trained by UNIPD on about 6000 frames that were manually labelled to provide the crack/no crack information for the training. The data set was split into 12 sequences, one for each sample part. A leave-one-part cross-validation protocol was exploited: for each fold, 11 parts were used for training, and the last one for testing the system. Detection accuracy per frame was reported to vary widely between 70% for the majority of the sample parts, but dropped down to 25% for a single crankshaft.
The classification software has been integrated into the image analysis module. Active learning techniques were also implemented for reducing the effort needed for collecting the training samples. This is a crucial point, as the correct training of an SVM often requires a very large dataset in order to avoid overfitting.
The presented classification results are a basis for further development that aim at generating a good/bad decision for the whole part. The concept that has been developed is to analyse sequences and remove spurious false classifications. The assumption being that a crack will appear in multiple consecutive frames, while false classifications will appear more or less random.
Task 4.4 Online simulation of thermodynamic process
The first 6 months of the project were dedicated to an initial simulation of a simple thermographic process and to compare this simulation to experimental data. The goal of this comparison is to obtain an early assessment whether online simulation has a chance of being successfully used as a basis for a segmentation algorithm. Work has been done on setting up the theory and a basic simulation task. Once the basic feasibility was established, work on this topic was interrupted until about month 15 of the project, because the implementation of online simulation was planned only for the last year of the project. After month 15 the concepts have been further developed and different methods have been investigated for the real-time solution of the thermodynamic equations. Explicit methods seemed to be more usable because they require less computing power and the stability criterion could be satisfied as we aim for prediction only over a few frames. The initial conditions is an image taken by the camera and the evolution of this image over time is predicted and compared to the real image, checking for substantial deviations.
During the third year, the online simulation of the heat flow was completed. The goal was to develop heat simulation that can run in parallel to the inspection process and generate artificial images. By comparing these images to the actual image, defects can be detected. The key element was to develop finite element methods that are quick enough to be run at the speed of the inspection process. The details of the methods are described in deliverable D4.6. A key method to achieve the necessary speed, was to limit the volume (voxels) for which heat simulation needs to be done. By estimating which volume elements (voxel) will significantly contribute to the heat flow, the number of voxel could be reduced by more than 50%. Also the level of discretization was investigated in order to have as few voxels as possible.
From the heat equations a set of linear equations has been deduced that are then used in the online simulation. Again, the details were presented in deliverable D4.6.
This model has been implemented and integrated into the demonstrator. The online heat simulation is considered to be an alternative option for image analysis and defect detection. Its computational requirements are, however, challenging for the computing equipment typically used in such inspection system.
Task 4.5 Automatic contrast estimation
This task will provide the input for the dynamic adjustment of the path by assessing the image quality and deducing changes in the path of the robot or other process parameters that may be adapted to improve the detection of cracks. The basis for this work has been developed during the experiments done also for the POD and ROC analysis. During this work, knowledge has been generated about the detectability of cracks under certain process parameters.
The main concept that proved to be useful for that purpose was to fit a Gaussian function to the temperature distribution in an image window close to the laser spot. This fit has the advantage that it reduces the whole heat distribution to a few numbers, such as the standard deviations in both directions, that provide an indication about the available contrast.
The standard deviations were calculated along the principal axes in order to be invariant to the direction of motion of the laser spot relative to the image axes. Results indicate that the typical image with good contrast shows standard deviations in the range of 15 to 18, which low contrast images have standard deviation of 33 to 36. The factor of 2 between these values indicates that there is sufficiently fine distinction between high an low contrast images.
These algorithms were implemented as a separate module and included in the demonstrator, where it was linked to the path re-planning module.
Highlights of most significant results
Image analysis for defect detection and classification on metallic and carbon fiber parts. Forward- and backprojection of defect onto the 3D CAD model.
WP5 - Assessment
The objectives of this workpackage are to integrate the single modules into the demonstrators that are set up every 6 months and to assess the technical results of the project on the test cases defined by the end users.
Task 5.1 Evaluation on basic inspection task
At the end of month 6 a first basic integration test was done by Profactor with support by UNIPD and ITRobotics. The goal of this integration test was to plug the single modules together and to check whether there are any problems at the interfaces. The modules that were tested in this first integration test were the communication with the robot’s control unit, the connection to the thermo-cameras, the interface to the image analysis module and the interface to the defect classification module. Additionally, interfaces were developed and testes that related to the offline (static) path planning. Overall the results showed that aside from minor compatibility issues (Win7 32bit vs. 64bit) the integration of the single modules worked well.
A major part of the work at Profactor was dedicated to setting up the demonstrator workcell. This included the design of the workcell (including a protective housing against laser radiation), the setup of the laser and the camera and the integration of various software components. Two major (formal) problems caused significant delays that could only partly be compensated.
The first delay was caused by the fact that for the robot a rental contract that is compatible with EC funding rules had to be set up, which lead the first robot manufacturer (Motoman) to withdraw their offer after 3 months and discussions with the second manufacturer (Fanuc) took further 4 months for completion. The Fanuc robot was finally delivered by the mid of month 12, therefore a smaller Stäubli robot had to be used temporarily as a replacement for the demonstration.
The second delay was caused by the laser that was supposed to be delivered in month 10 but did not arrive until late in month 12, so that it was not possible to generate large sets of test images. This delay have been partly compensated by developing the image processing software using similar images from previous projects and similar image datasets, so that the demonstrator is now roughly 1 month behind schedule.
Nevertheless an evaluation could be done to assess the main properties such as path planning results, accuracy of the synchronisation between camera and robot and defect detection in the recorded image sequences. The results are presented in deliverable D5.1 that has been prepared by TRIMEK.
The second basic integration task was done at the demonstrator for the CFRP parts at TRIMEK. In month 13 TRIMEK started with the work on the demonstrator by building the mechanical integration of the flash, the thermo-camera and the robot. This work was finished in month 17, before an integration week started, where Profactor added software components for acquiring images and controlling the robot via TRIMEK’s standard interface. By the end of month 17 the TRIMEK demonstrator was able to move the robot and to acquire images using the flash. Unfortunately, the set up of the image acquisition parameters revealed to be tricky and took a long time. In the end, it seems it is not possible to image inner defects in the CFRPs other than checking the gluing layer.
Task 5.2 Evaluation on static, autonomous inspection
By month 18 an integration test was done that combined the most recent version of UNIPD’s image segmentation, IT+Robotic’s path planning module and Profactor’s software framework. This integration test was done as planned at the end of month 18.
For the crankshaft inspection demonstrator at Profactor the following results were achieved by the end of month 18. The robot is able to acquire a sequence of images while moving the part. Joint coordinates of the robot are acquired every 8ms, while images are taken every 5ms. Occasionally single joint angles or images are missed, which may be due to real-time processing issues. However, this does not affect the overall performance of the system. Both the joint angles and the images are time-stamped with a synchronized clock to allow a direct correlation between the data. To investigate the extent and the cause of these problems, in the workcell set up at UNIPD based on a COMAU robot was decided to use the COMAU C4G Open feature to the controller. This enabled a hard real-time bidirectional communication with the COMAU robot and will be used to investigate synchronization issues. Full calibration has been performed including the world, part, robot and camera coordinate systems. The overall accuracy in a static setting is within ±1 pixel, which is sufficiently accurate for the application.
Processing of the images works well, however the algorithms have slight problems keeping up with the speed of 200 frames per second, which is mainly due to a time-consuming image conversion that can possibly be avoided. Also it will be possible to slow down the acquisition rate to 50 frames per second without affecting the quality of the inspection.
The sideblade inspection demonstrator at TRIMEK is based on flashlamp heating and oriented to carbon fiber parts. This material cannot be heated using a laser, since very high temperatures generated by the laser can damage it. In order to avoid such a problem a flash lamp has been used to heat the piece for the thermographic inspection. Trimek has assembled the demonstrator and integrated both the hardware and software required. Concretely, the demonstrator set up at Trimek integrates the following hardware components: the Stäubli TX90 robot with the CS8C controller, the flash EH Pro 6000 and the generator TRIA 6000 S from Hensel, the thermographic camera provided by Infratec and a PC. Moreover, Trimek has developed a hand effector for the robot so it could carry the thermocamera and the heating system. Regarding the software, this demonstrator integrates the robot controller, the TBRi interface and the path planning application developed until M18.
After the intermediate integration test developed in month 17 at Trimek, this demonstrator integrates correctly both the hardware and the software and works as required. Path planning tests are correctly developed and thermoimages are captured and saved, ready to be analysed by Profactor. Unfortunately, the captured images proved not to be useful for the software development realised by UNIPD, because no defects are visible. Images obtained show only superficial defects, which are also visible with a standard (visible light) vision system. A careful investigation on the image acquisition parameters and on the new pulsed-phase thermography technology is jointly running from Trimek, Profactor, and Infratec with the contributions of BAM and the feedback of UNIPD.
Between month 19 and 27 numerous tests were done on the demonstrator to assess various aspects. These included mainly path planning and the time needed for the execution of a whole path, tests with the image analysis and also initial tests related to online simulation and contrast optimization. The latter two topics are mainly part of the final assessment in task 5.3.
In month 24, during the general meeting of the consortium, the results of the demonstrations were discussed and further steps have been agreed. BRP also provided substantial feedback, which led to a priority list of properties that an automatic inspection system has to fulfil:
(1) The system has to fully cover all areas on the part that have to be inspected. There shall be no “blind spots”. BRP acknowledged that certain physical limitation cannot be overcome, but also confirmed that the crankshaft is the most complicated part and other parts of simpler geometry could be easily covered by the system
(2) The inspection system shall not require a combination with other methods (such as ultrasound or eddy current, …), because quality control is a non-productive element of the production process.
(3) Defects down to the minimum size as required by quality control instruction have to be detected.
(4) Cycle time has to be kept within reasonable limits. While it is possible to duplicate the system and thus double the throughput, the large investment will make the implementation less likely.
For the crankshaft inspection system the main assessment criterion is the classification accuracy of the crack detection, which was found to be about 70% for most test cases, but very low (~25%) for a single case. Further details are described in task 4.5. The second criterion that has been monitored is the processing time that is needed for inspection a whole crankshaft. Simulation results by IT+Robotics indicated a cycle time of 77s.
Assessment of the CFRP inspection system was mainly done by TRIMEK with support from all partners in their respective field. The evaluation was focused on the image acquisition with pulse-phase-thermography and the pre-processing of the images. Two different methods were compared: On the one hand Infratec’s “standard” implementation in the IRBIS software and one the other hand Profactor’s implementation for the online inspection tasks. After adjusting the parameters to the correct settings, very similar results were achieved for both implementations.
Task 5.3 Evaluation on dynamic, autonomous inspection
The final evaluation included the whole set of features that were developed for both of the demonstrators. The first demonstrator covered 2 different test cases by BRP (crankshaft and piston rod) while the second demonstrator covered the test case by BSCT (sideblade). Tests with parts of companies outside the consortium were also done, these are presented in the description of task 6.2.
For the metallic part inspection the main assessment criteria and their evaluation results are briefly described in the subsections below. Detailed results are presented in D5.3.
Parts inspected (size of a crankshaft) in 20-30s
This goal turned out to be very difficult to achieve and later in the project, when all process parameters were optimized, the following theoretical considerations showed the following limits: Given the optimal laser spot size of 5mm and the maximum speed of the robot, the spot may pass over the surface at 188mm/s. Assuming a cylindrical surface of the size of crankshaft, the ideal motion along the path that covers the whole cylinder would take about 2.4minutes. More realistic estimates that include motions of the robot to change its configuration where necessary showed a limit of about 5 minutes. During the evaluation the best possible path that could be achieved took about 6 minutes for the inspection of the crankshaft.
Detection of defects according to end user specifications
Defects that can be detected are limited to the rugged portions of the crankshaft, as the shiny parts cannot be analyzed. The analysis of the hot spot shape was successfully used to classify the images acquired by the demonstrator. Even though the image-based classification still shows a limited number of false detections, the per-sequence classification mechanism – looking for bursts of images containing a defect – is able to provide satisfactory results.
Autonomous path planning for full coverage
This criterion has been achieved to a very large extent. A process model including camera, laser and acquisition process has been implemented and used in the coverage planning. In an iterative exchange between the coverage planning module and the simulation the path is checked for collisions and reachability, while making sure that it is still fully inspection. Path planning could be shown to work well on two quite different parts in terms of size and complexity.
Contrast estimation and dynamic re-planning if inspection quality is low
A contrast estimation method was implemented that uses the shape of the spot and its intensity to determine whether an accurate inspection is still possible. This estimation was integrated in the demonstrator and could be successfully tuned to detect low contrast. This was demonstrated by inspecting a narrow gap that requires a small angle between surface and laser. Due to this angle the spot induced heat and the spot size decrease. This was successfully detected and robot motion control was triggered to stop the robot. After automatically adjusting the laser power, the robot repeated the path and that same segment then showed sufficient contrast for inspection.
Online simulation for defect detection
An online heat flow simulation was implemented, that performs a simplified version of full finite element calculation. By limiting the volume that was considered in the calculation the computing time could be reduced, so that an online simulation was possible. For the detection of defects a re-projection is required that converts the simulated 3D data into a 2D image. This re-projection could not be made accurate enough to allow a direct comparison between the simulated and the actual image. Also computing time issues for the re-projection did not allow this to be computed in real-time. The final result is that simulation of heat flow is possible in realtime, but additional computing steps are required that need further development and improvement. Applications of the simulation are possible, however, in more simple settings, where the motion of the object is not as complicated.
Second metallic part to be demonstrated
As a second part for testing a piston rod was provided by BRP. This part was chosen, because it differs substantially from the crankshaft in terms of complexity and size, while still being an important part in terms of later use of the technology. It could be demonstrated that the inspection process can be set up for the new test part without any part-specific tuning or adjustments.
Planning a new part < 15 minutes
Path planning consists of several steps, starting from the coverage planning over the robot motion planning to the robot program generation. An iterative process was needed to ensure that all relevant criteria are fulfilled (full coverage, no collisions, all points reachable). It was found that the computation of a path for a crankshaft takes about 4 minutes, which is less than one third compared to the original goal. The remaining time could thus be used to run further iterations in order to continue minimizing the time needed to execute the path.
Carbon Fibre parts inspection
The demonstrator set up at Trimek has been evaluated performing several experiments. The tests have been done with BSCT composite parts. The main objective of the demonstration phase is to test if the demonstrator is able to capture useful images of the bonding areas of the BSCT carbon fiber components. Several images have been acquired, using different exposition times and capturing with different frames per second. The results obtained show that the optimum acquisition parameters are: 25 seconds of acquisition time and a frame rate of 8 frames per second.
Regarding the path-planning sub-system, the evaluation was performed using the Trimek machine and the OptiScan sensor. The objective was to scan the positions obtained with the positioning system to test its accuracy. The main conclusion is that path planning sub-system is considered precise enough for thermographic inspection operations.
Highlights of most significant results
Both of the demonstrators were tested and evaluated relative to the technical goals foreseen in the description of work and also in comparison to additional specifications of the end users. The results are documented in D5.3
WP6 - Demonstration
The objective of this workpackage is to demonstrate the capabilities of the THERMOBOT system on the use cases and additional test cases provided by the business interest group.
Task 6.1 Demonstration on the parts of the end users
Originally the consortium tried to have the demonstration for the end users already at the end of month 11. But due to the problems with the delivery of the robot and the laser, the demonstration for the end users had to be postponed to month 14 (Feb. 2013). At that time BRP and BSCT visited Profactor, and the integrated system consisting of laser heating system, camera and robot was demonstrated. It could be shown to process a segment of a crankshaft, to record a series of images and to process these images with the image segmentation method of UNIPD that was available at that time. The key elements of the demonstrator (pre-defined path, recording a sequence of images, basic functionality for crack detection) could be successfully demonstrated.
The sideblade could not be demonstrated as at that time it was already decided that the inspection of the sideblade will be done on the TRIMEK demonstrator, whose construction just started in month 13.
After the set up of the TRIMEK demonstrator in month 18, a demonstration was performed to all the partners showing how the flash thermography system was able to acquire the required thermographic images. During this demonstration it was also shown how the carbon fiber parts can be adequately fixtured, as well as the optimal use of the camera flash to perform the inspection of the full part. End users BRP and BSCT provided also their views to this solution while the demonstration was performed.
In month 21 a demonstration was organised for a larger group of BRP, where the current status of the inspection robot was shown and additional test parts were investigated. The discussion mainly focused on the practical implementation of a ThermoBot system in a real-world production line and the question of how a certain area on the part, that often has cracks, can best be inspected. The key challenge is that the area is difficult to access and hard to see for the camera/laser system.
A specific demonstration was done for BRP in month 35, close to the end of the project during the final evaluation.
Task 6.2 Demonstration on parts of the BIG
At Profactor the following parts of companies outside the consortium were tested: a section of a large shell of ball bearings used in wind power applications. Artificial notches were introduced in the parts, but tests with the ThermoBot acquisition method showed that these artificial notches could not be detected. The second test part was a corrugated tube, where a weld seam had to be inspected. These tests showed positive results and high contrast between crack and background could be achieved for the samples. The third test past was a printed circuit board, where cracks in the board had to be detected. The results were positive and the crack generated clearly visible distortions in the heat distribution. All the results are given in detail in deliverable D6.1 and are not reproduced here.
At TRIMEK parts belonging to companies from outside the consortium were tested. The experiments performed are based on the creation of air spaces, placing foam between the layers before being glued. The objective is to test the capability of the Trimek’s demonstrator to detect layers with different thicknesses that have being glued as first layer. The results obtained show that the Trimek’s demonstrator is able to acquire accurate images that are useful to detect the presence or absence of air between composite layers, until a certain thickness (7 mm) where the demonstrator is unable to obtain clear images. So the experiments prove that there is an uncertainty area between 3 mm (air between layers identified) and 7 mm (air between layers not identified). This technical limitation provokes that the ThermoBot demonstrator set up in TRIMEK is certainly a useful defect detector for carbon fibre parts with a layer distance of 3 mm or less. From 3 mm to thicker parts the demonstrator cannot assure the detection of defects between layers in carbon fibre parts.
Highlights of most significant results
Demonstrations with parts of the end users and a set of parts from companies outside the consortium.
Potential Impact:
Impact
THERMOBOT aims at the development of a trans-sectoral inspection technology that can be used for metallic and composite parts and is relevant in many industrial sectors such as aerospace, automotive, and recreational vehicles but also construction and all other areas that use high-performance mechanical components. It thus addresses a large part of the European manufacturing industry that represents 21% of the EU GDP and provides more than 30 million jobs in Europe.
Developing the necessary enabling technologies to support EU manufacturing
THERMOBOT aims at the development of a technology that replaces a quality control method (magnetic particle inspection) that has been in existence for almost 100 years and is still executed manually even in otherwise fully automatic production plants. Automatic inspection systems for parts of complex geometry are rarely used in industry because of the costs of setting up the system (especially the robot program) for a specific type of part. THERMOBOT demonstrated that a technology for automatic path planning for complex inspection tasks is able to automatically and largely autonomously adapt to new parts (geometries, materials, surface structures) without the need for manual robot programming.
The different software modules developed for THERMOBOT show that the European industry can benefit from the flexibility of robot inspection systems not only in thermography, but also in classical vision systems, eddy-current systems or ultrasonic systems.
THERMOBOT provides a solution by reducing labour costs for manual vision inspection (in case of magnetic particle inspection by more than 90%, i.e.120k vs. 5k), in particular by creating an autonomous system that can adapt to different parts in very short time and does not need extended manual programming. As a side-effect it also solves the problem of documentation and of human fatigue which is always an issue in manual inspection processes. Moreover, THERMOBOT can lover also the maintenance costs. At the moment, the costs of maintenance in the European aerospace industry are substantial and most of the maintenance costs are labour costs. Airlines try to reduce these costs by having maintenance done in countries such as Pakistan, Libya or India. This is undesirable not only because highly qualified jobs are transferred to non-European countries, but it is also a safety issue. Even though these companies are certified by EASA (European Aviation Safety Agency) under Part-145 it is much harder to monitor companies in such distant countries than European countries.
Exploitation plans (ESS results, confidential)
The list below shows the exploitable results. The two demonstrators (No. 1,7) are the main results of the project. All other results (No. 2-6) are modules that are part of the “ThermoBot system” and are used in at least one of the demonstrators. The main route of exploitation for these results is inside the “ThermoBot system”, but some of them can also be exploited independently.
1. Robotic inspection system for CFRP parts (Lead partner: TRIMEK)
2. Module 1: Image Analysis (Lead Partner: UNIPD)
3. Module 2: Online Simulation (Lead Partner: Profactor)
4. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
5. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
6. Module 5: Inspection Software (Lead Partner: TRIMEK)
7. Robotic inspection system for metallic parts (Lead Partner: Profactor)
Description of exploitable results
No. 1 : Robotic inspection system (the “ThermoBot” system)
Innovativeness introduced compared to already existing Products/Services: The system developed is able to detect internal defects in composite parts, specifically defects in the glue layer (absence or presence) between sheets in carbon fibre parts. This is a great advance in inspection systems.
Product/Service Market Size: A conservative estimation could be of 1000 units for automotive industry, wind energy equipment manufacturers and other part manufacturers.
Competitors: Nowadays it seems that this solution has no direct competitors, but in the middle term producers of robotic inspection and of thermographic solutions could develop a solution adapted to the composite parts market.
Prospects/Customers: Potential users will be in those manufacturing sectors where the use of composite parts is increasingly important: automotive industry, wind energy components, aeronautics and, maybe, aerospace industry.
Cost of Implementation (before Exploitation): 350k EUR of additional development costs for the whole system.
Time to market: 2 years after the end of the project.
1. Which partner contributes to what (main contributions in terms of know how, patents, etc.): See Module 1: Image Analysis (Lead Partner: UNIPD)
2. Module 2: Online Simulation (Lead Partner: Profactor)
3. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
4. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
5. Module 5: Inspection Software (Lead Partner: TRIMEK)
No. 2 : Module 1: Image Analysis
Innovativeness introduced compared to already existing Products/Services: The module deals with the peculiarities of robotic thermography and is particularly made for this type of inspection.
Product/Service Market Size: 1 module per robot; see result No. 1
Competitors: Developers of “standard” software libraries such as HalCon, VisionBlox or similar might consider developing thermography modules.
Prospects/Customers. See result No. 1
Cost of Implementation (before Exploitation): About 80k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): UNIPD uses images provided by Profactor and TRIMEK. There are also links to motion planning (ITRobotics) because image analysis needs to know about the current position of the robot.
Partner/s involved expectations: Image analysis should perform, robustly on the demonstrators and on the test parts considered in ThermoBot. Further development will be required to turn it into a product.
No. 3: Module 2: Online Simulation
Innovativeness introduced compared to already existing Products/Services: The module deals with online simulation of thermographic processes. It uses a model to make short-term prediction about what the image should look like.
Product/Service Market Size: Due to the complexity of the developments, the module will first be used in more simple application, e.g. those where parts are only transported along a linear motion and where the geometry is simple. There could be a use for about 10 such modules per year in crack detection systems in the steel industry.
Competitors: The module is an alternative to exploitable result No.2 (image analysis).
Developers of “standard” software libraries such as HalCon, VisionBlox or similar might consider developing thermography modules (but not very likely).
Prospects/Customers: Large companies that integrate thermography system in high-value applications, such as steel industry. Potential customers include SMS Meer or Inst. Dr. Foerster.
Cost of Implementation (before Exploitation): About 250k EUR. This increased over earlier expectations, because the level of complexity is higher than expected.
Time to market: 3 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor uses information provided by BAM.
Partner/s involved expectations: The module should be integrated and working in the demonstrator.
No. 4 : Module 3: Robot path planning
Innovativeness introduced compared to already existing Products/Services: Robotic process planning exists only for simple processes. Thermography is much more complex, because it involves temporal (not only spatial) processes.
Product/Service Market Size: 1 module per robot for metallic part inspection; see result No. 7
Competitors: There are companies developing (semi-) automatic tools for path planning (e.g. in Germany and Austria.), however, thermography requires a quite complex, thermodynamic process model for path planning.
Prospects/Customers: Potential customers will be in the automotive, transport and ship-building industry and those that deliver parts for the automotive, transport and ship-building industry. Additionally, technologies used in the path planning module may be used for a much wider range of inspection tasks, that require continuous scanning.
Cost of Implementation (before Exploitation): About 100k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor contributes the process model and coverage planning, while ITRobotics contributes collision detection and path/motion planning.
Partner/s involved expectations: Path planning is an important element of the ThermoBot system for the metallic part inspection. At the end of ThermoBot, the robot should be able to autonomously plan a path for the inspection of a complex object (e.g. crankshaft).
No. 5 : Module 4: Workcell simulator
Innovativeness introduced compared to already existing Products/Services: Robotic simulation uses the dynamical models of the robot and is able to model time constraints.
Product/Service Market Size: 1 module per installation; see result No. 1 and No. 7
Competitors: Robot manufacturers offer simulation tools that also include 3D modelling of workcells.
Competitors are brand -oriented and they do not handle perception in the path planning loop and solution of time- constrained tasks.
Prospects/Customers: See result No. 1
Cost of Implementation (before Exploitation): About 80k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor and TRIMEK contributes by providing models and testing.
Partner/s involved expectations: … to extend the range of possible applications of the workcell simulator with the modules developed within the project
No 6: Module 5:Inspection software
Innovativeness introduced compared to already existing Products/Services: The inspection software developed will be adapted to the peculiarities of robotic thermography.
Product/Service Market Size: 1 module per robot; see result No. 1
Competitors: Inspection software developers.
Prospects/Customers: See result No. 1
Cost of Implementation (before Exploitation): About 200k EUR
Time to market: About 2 years after the end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Image analysis provided by UNIPD and Profactor. Path planning software provided by IT Robotics.
Partner/s involved expectations: The software should be integrated and working in the demonstrator.
No. 7: Robotic inspection system for metallic parts
Innovativeness introduced compared to already existing Products/Services: The robotic system is able to plan the inspection task for a complex metallic part with a high degree of autonomy.
Product/Service Market Size: About 1000 units (automotive and parts manufacturers).
Competitors: There are companies that offer thermographic inspection (e.g. ThermoSensorik) and others that offer robotic inspection systems (ISRA, Vitronic), but we had not found companies that do both in a single system.
Prospects/Customers: Potential customers will be in the automotive, transport and ship-building industry and those that deliver parts for the automotive, transport and ship-building industry. Additionally, technologies used in the path planning module may be used for a much wider range of inspection tasks, that require continuous scanning.
Cost of Implementation (before Exploitation): About 400k EUR of additional development costs for the whole system.
Time to market: About 2 years after the end of the project.
Which partner contributes to what (main contributions in terms of know how, patents, etc.):
1. Module 1: Image Analysis (Lead Partner: UNIPD)
2. Module 2: Online Simulation (Lead Partner: Profactor)
3. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
4. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
Dissemination Activities
At the start of the project a web page (www.thermobot.eu) has been set up. The project web page (www.thermobot.eu) was updated on a regular basis with deliverables and news.
Further information about particular events is provided in deliverable D8.4
Industrial dissemination
In the first 6 months members for the business interest group were being sought. The business interest group is a selection of companies, typically end users, who provide additional test cases. IT+Robotics has been very active and successful in acquiring members for the business interest group (CIMM SPA, ZML SPA). Also UNIPD contacted for the BIG several companies: Tecnogamma (Mermec Group), Starmatik srl, Rea Robotics srl. Profactor contacted: MIBA, SKF, Schaeffler. Three companies agreed to join. Another, three companies provided test parts (see task 6.2) for test on the metallic parts inspection, but these companies chose not to become full members of the BIG. Nevertheless, their parts provided useful insights into the capabilities and limits of the developed technology.
Furthermore, Profactor took part to an exhibition at the Control fair 2013, where first results of ThermoBot were presented. IT+Robotics, UNIPD, and Profactor were present in an exhibition at the Automatica fair 2013 to disseminate Thermobot results.
Moreover, UNIPD and IT+Robotics disseminated the Thermobot activity both in 2012, in 2013, in 2014 in the “Corso Nazionale Automazione Industriale e Robotica” organized yearly by “SIRI Associazione Italiana di Robotica e Automazione”.
TRIMEK has been actively disseminating the Thermobot activity among potencial users: NECO,EPC, ITP, Renault Valladolid. Trimek also disseminated Thermobot at Metromeet, and other exhibitions such us BIEMH.
For carbon fibre parts inspection a number of companies have been contacted by Trimek during the project. Some of these companies provided parts to perform additional tests undertaken as part of WP6 (see D6.1). The agreements signed between Trimek and these companies specify clearly the confidential nature of this collaboration, so unfortunately they cannot be named or listed in this report. However, their contribution was essential to keep on deepening on the potentials and limitations of the developed technology.
General public dissemination
The project was presented by Profactor at the “CE Colloquium” held at the University of Linz. This presentation was attended by about 20 students from the field of computational engineering. At the end of May again Profactor presented the project at the annual Workshop of the Austrian Association for Pattern Recognition. The invited talk in the industrial part of the workshop was attended by about 40 participants. UNIPD took the opportunity to present the project at two large public events in Italy. The first on was the “NEXT fest” organized by the WIRED magazine in Milano, and the second one was the RoboVal event in Verona.
Scientific dissemination
UNIPD presented the results of the project in 4 journal papers and Profactor and BAM in 1 additional journal paper.
BAM and BRP prepared a presentation for the annual meeting 2013 of the German Society for NDT, title and abstract have been submitted in month 11 and were selected for publication. Two papers were presented at the HES-13 conference (Heating by Electromagnetic Sources) in Padua. The first paper was a joint paper by UNIPD and Profactor describing the approach implemented in the Thermobot project, the second one was a paper by UNIPD describing the image processing and image classification algorithms developed for Thermobot.
Further results of the ThermoBot project were presented at the following scientific conferences: ECNDT 2014 (European Conference on Non-Destructive Testing), QIRT 2014 (International Conference on Quantitative InfraRed Thermography) and the workshop of the DGZfP (Deutsche Gesellschaft für Zerstörungsfreie Prüfung).
A full list of publications resulting from the project is included in deliverable D8.4.
A major dissemination event that was related to ThermoBot was the IAS-13 conference that was held from 15th to 19th of July 2014. The conference was organized by UNIPD and the ThermoBot project was presented on 3 occasions: an Industrial Workshop, an Industrial Forum focused on Horizon 2020, and a technological and scientific exhibit.
UNIPD & Profactor organized the IAS-13 “Industrial Forum” with specific emphasis on the technologies and applications of autonomous systems in industry. Smaller lot sizes, a high number of product variants and complex requirements in terms of flexibility require autonomous systems that can quickly adapt to changes and also interact with users. Also research directions at the European level will be addressed that are currently aimed for in the “Factories of the Future” topics of Horizon 2020. IT+Robotics presented “Flexible visual inspection systems” with a focus on Thermobot technologies
The conference and its workshops were attended by about 350 participants, mainly from Europe, but also with a strong participation from Asia and the US in the field of robotics and artificial intelligence. The special motto was: “Bringing autonomous robots into industrial production”, in order to highlight the most recent results of autonomous robots and intelligent systems which are now sufficiently mature and robust to operate in industrial production..
In addition to this, UNIPD presented twice the Thermobot project in the FoF workshop organized by the EC in Brussels disseminating the project and its achievements among the other project leaders and coordinators of EU funded projects.
Clustering with other projects
On two occasions in March 2013 and March 2014 the ThermoBot project, represented by UNIPD, participated in cluster events organized by the European Commission to improve the Impact that the project will have. After the second workshop, a collaboration with the other three coordinators of the projects funded in the same call of Thermobot was started to implement project cluster activities for further dissemination, cross-fertilization, and knowledge transferm among the four project in the cluster. ThermoBot was part of a cluster, with a group of projects that have all been submitted to the topic related “robots for post-production and auxiliary processes”. Other projects included “MiRoR”, “CableBot”, and “MainBot”. The project were quite diverse, so it was decided to focus on joint dissemination activities.
A first event was a large web conference call attended (almost) by each partner of the four projects in which the four projects were presented in details. After this a mutual linking of the project in each project web site was implemented.
The second cluster event was a joint workshop co-located with IAS-13 in Padua (Italy) in July 2014. This workshop was organized by UNIPD under the lead of Stefano Ghidoni and included presentations of the single projects and a discussion about challenges that were encountered in these projects. The workshop was attended by about 25 persons.
The third was the participation of UNIPD as coordinator of Thermobot in a new EU Coordination and Support Action called “FOCUS: Factory of the Future Clusters”. The project builds upon the fundaments of five existing FoF Clusters, Zero Defect Manufacturing (4ZDM), Robotics, Clean factory, Precision Micro Production Technologies (High Micro) and Maintenance and support. The outcome of FOCUS will be a methodology for clustering and for industrial exploitation & take-up.
List of Websites:
Project web page:
www.thermobot.eu
Contacts:
Project Coordination
Prof. Emanuele Menegatti, Ph.D.
Intelligent Autonomous Systems
Department of Information Engineering
The University of Padua
via g. gradenigo 6/A
35131 Padova, Italy
Phone: +39 049 827 7651
E-mail: emg@dei.unipd.it
Technical Management
Dr. Christian Eitzinger
Profactor GmbH
Im Stadtgut A2
4407 Steyr-Gleink, Austria
Phone: +43 7252 885 250
E-mail: christian.eitzinger@profactor.at
THERMOBOT investigated the automation of crack detection in parts of complex geometry and large size. The current procedure for crack detection is a process that dates back to the 1920s and is called “magnetic particle inspection”. This method is infamous in industry, because it is a cumbersome, dirty process that is often done manually even in otherwise fully automatic production lines. THERMOBOT aimed at replacing this old method for crack detection with a new technology that is based on autonomous inspection robots using thermography to recognize cracks on parts of complex geometry. The robot scans the whole part with a thermo-camera and analyzes the heat-flow to find cracks and other defects hidden under the surface. To achieve this, the project targeted the following three objectives:
• (A) development of a thermographic process and a process model for the automatic detection of cracks in parts of complex geometry, varying surface structures and for different materials.
• (B) development of an automatic path and motion planning module that uses the thermographic process model of objective (A) to automatically generate a path for the inspection robot from 3D CAD data. Path planning also requires a dynamic (on-line) adaption of the path to account for variations in the thermal properties and surface structures of the part.
• (C) investigation of thermo-image analysis methods that not only aim at detecting cracks and making an accept/reject decision for the whole part, but also have the capability of self-evaluating the performance of the crack detection.
In the project two different demonstrators have been set up for different types of parts and materials. The first demonstrator uses laser-based thermography for metallic parts. The second demonstrator uses flash thermography and is used for inspection of carbon fibre parts.
Project Context and Objectives:
Despite considerable progress in the automation of production processes visual inspection of products and parts is still done manually in a wide variety of inspection tasks. This project aims at the automation of crack detection in parts of complex geometry and large size. The current procedure for crack detection is a process that dates back to the 1920s and is called “magnetic particle inspection”. This method is infamous in industry, because it is a cumbersome, dirty process that is often done manually even in otherwise fully automatic production lines. The component to be tested is magnetized before applying a suspension of finely divided coloured or fluorescent magnetic particles. Cracks or inclusions cause the magnetic flux to break the surface forming free magnetic poles and the magnetic particles will collect at these locations indicating a crack. Using UV-light the fluorescent particles in the suspension are activated to increase the visibility of the cracks (see figure below). Inspection of complex parts is usually done manually, whereas for simple geometries machine vision systems can be used for automatic detection.
This (still) state-of-the-art process is undesirable for a number of reasons:
• the process is time-consuming and involves several non-productive process steps, such as magnetizing and de-magnetizing the parts and spraying them with a suspension that has to be removed later in many applications.
• the liquids used for the process are ecologically undesirable. They are usually petroleum-based or water-based with corrosion-inhibitors added in the latter case. The magnetic particles need to be replaced frequently as they stick to the part.
• the manual inspection process is expensive, time-consuming and unreliable. It also has problems with documentation, which is often a requirement, especially in the aerospace industry.
It should be noted that inspection has to be done not only as a final production step before delivery, but also takes place throughout the lifetime of the product, when repairs have been made or for regular inspection, e.g. in aerospace applications.
THERMOBOT aims at replacing this old method for crack detection with a new technology that is based on autonomous inspection robots using thermography to recognize cracks on parts of complex geometry. The robot will scan the whole part with a thermo-camera and analyse the heat-flow to find cracks and other defects hidden under the surface.
Up to now such robotic inspection systems using thermography cannot be implemented, because thermography is a dynamical process. The thermal properties (conductivity, emission) of the material and the image acquisition process itself need to be considered when defining the trajectory for the robots. Manual offline-programming of the robot’s path or programming by demonstration cannot capture these properties. It thus requires a model-based approach as aimed for in THERMOBOT.
These scientific goals of the project correspond to the technological objectives:
(A) A full, parametric model of the thermographic imaging process has to be developed that is sufficiently quick to be calculated in real-time. The model will consider emission coefficients for different surface properties, reflections from nearby surfaces and artefacts that the image acquisition process may produce. It also has to include the fact that multiple images of the same spot have to be taken at different times to fully analyse the heat flow.
(B) Robot path planning has to be fully autonomous based on 3D CAD data. Using the process model, the path planning module needs to generate a sequence of viewpoints (robot poses) that create a series of overlapping images so that each spot on the part is visible in multiple images. Path planning also has to consider that the robot’s motion is continuous (not a start/stop motion for each viewpoint) and has to reach each viewpoint at a specific time. In addition to this, methods for dynamic adaptation of the robot’s path need to be developed that account for the specific properties of each part and ensure full coverage of the part as well as optimal inspection quality.
(C) Image analysis not only needs to detect the defects (which is standard today), but needs to include generic knowledge about the image acquisition process and learning capabilities so as to distinguish between artefacts such as reflections (that often occur on complex shapes) and real defects. It also needs to include a self-evaluative component that predicts the quality of the detection by analysing the expected contrast based on on-line simulation of the heat flow and the thermographic imaging process. Image analysis will provide the input data for the automatic accept/reject decision.
The overall project plan is structured along a series of integration tests and demonstrators.
As planned in Annex I, two different demonstrators have been set up. The first one is located at Profactor. It uses laser-based active thermography and is mainly used on the crankshaft (metallic) inspection task. The second demonstrator is based at TRIMEK and uses flash thermography for the inspection of the carbon fibre parts. Originally it was planned, that both demonstrators would cover the same (whole) spectrum of parts, but the substantially different material properties led the consortium to use different technologies for the demonstrators so that all test cases can be adequately covered.
In month 36 the demonstration included static path planning as in demonstrator #2. In addition, it provided a dynamic component that modifies the robot’s path for each part so as to locally optimize image quality. This required feedback from the image analysis component that used an on-line simulation model to predict the contrast with which faults would be visible in the images. To demonstrate the autonomy achieved by the robotic inspection system, it was tested on 3 new parts (2 metallic, 1 composite), which demonstrated its capability to perform the inspection tasks fully autonomously. Re-calculation of the path for a new (previously unknown) took less than 15 minutes.
The bold properties indicate the key features that were added to the demonstrator shown and described in month 18. Not all the features are present in both demonstrators, because the carbon fibre parts inspection uses start/stop motion, where online simulation, contrast optimization and dynamic replanning are not possible. These features are only present in the crankshaft inspection demonstrator.
Project Results:
WP1 - Test case definition, general concepts
The objectives of this workpackage are to develop general specifications for a thermographic inspection robot, to define the test cases and test parts and to collect sample parts for testing.
Task 1.1 Selection of test cases
The selection of test cases was completed by the end of month 6. The result of this selection process is documented in deliverable D1.1 prepared by BSCT and BRP. The test samples for these test cases were used by BAM for doing the CT measurements and by Profactor for identifying the right specifications for the thermo-imaging equipment (camera, laser).
Task 1.2 Collection of labelled sample parts
During the first 18 months BRP collected and labelled a set of crankshafts that have been exploited for further testing during the integration tests and the demonstrations. These parts were taken from the “scrap” parts collected during quality control and were documented and labelled to make them available to the other partners.
BSCT produced a set of the small sample parts (CFRP) for the CT measurements and a few samples of the Audi R8 sideblade, which will serve as demonstration part in the ThermoBot project.
After initial experiments have been performed, both BRP and BSCT collected another (final) set of labelled sample parts for the different inspection tasks. Sample parts of BRP were sent to Profactor, while the parts of BSCT were sent to both TRIMEK and Profactor. All the sample parts were assigned a specific ID that allows the tracking of the parts throughout the project. A list of these sample parts is maintained by the project manager that includes the parts, defects to be found, their ID and their current location. In the course of the second half of the project these sets of samples were extended as part of the assessment and demonstration activities in WP5 and WP6.
Task 1.3 Definition of accept/reject criteria
BSCT and BRP provided information about the accept/reject criteria to be used for their particular applications. In general, the specification of the criteria is not made on a tight numerical basis, but leaves room for subjective interpretation, because up to now the inspection has been done manually by experts. The criteria have been integrated in the overall concept by Infratec (D1.2) and were later added to the documentation in of the sample parts in deliverable D1.3.
Task 1.4 Generally applicable concepts for inspection
In this task, Infratec had the lead in preparing a document that identifies generally applicable concepts for thermographic inspection of parts. Many partners contributed to this document, including ITRobotics, TRIMEK, and Profactor. The main results that emerged from these discussions are that:
• Quantum cameras are required for robotic inspection tasks (bolometer is not an option due to the motion and frame rate that is required).
• In many cases the robot will hold the part rather than the thermo-imaging system, because of the cables and cooling system required for the laser and the camera.
• Lasers are the only reasonable heat source for the inspection of whole crankshafts, due to the complex geometry of the parts.
• Flash thermography seems to be more suitable for more or less flat parts such as the sideblade.
• In terms of robotic handling systems high flexibility is needed, and the ThermoBot framework should not be limited to particular kinematic structures.
During the preparation of the document it was also found that it is useful and feasible to design the system software in such a way that a generally applicable concept for inspection robots is obtained, without being limited to thermographic inspection. The general concept includes 6 modules: process model, coverage planning, path planning, synchronisation, image analysis and decision making. Interfaces between these modules can be defined in a quite generic way that allows the exchange of single modules without affecting the others.
WP2 - Thermographic process model
The objectives of this workpackage are to develop a parametric process model of thermographic image acquisition and to perform comparison measurements with CT and a statistical assessment of the reliability.
Task 2.1 Process modelling based on enhanced CAD data
Profactor, with support by UNIPD, developed an initial physical model of the thermo-imaging process. The results are described in deliverable D2.1. While the physics of the thermodynamic process are well known, work also included experiments with lasers and thermocameras to identify optimal operating characteristics. At Profactor a laser was rented for 2 months to perform initial tests. The initial result was a set of specifications for the laser that was bought for the inspection system in the project. A particular challenge was the wide variety of thermal material properties of the test cases. Metallic parts (crankshafts by BRP) differed almost by an order of magnitude from the CFRP parts (R8 sideblade BSCT) in terms of emission coefficients.
The initial process model has been implemented and integrated into the first versions of the path planning module. Initially, the process model used a simple, phenomenological model that consists of a ring-shaped area around the centre of the laser spot. This ring identifies the area that can be checked from a single viewpoint of the camera when looking down perpendicularly on a flat surface. The next step was to map this ring onto the CAD data of the test parts and to reduce the area by considering the visibility, edges and corners on the part. This work was done by Profactor.
Based on the experiments for the POD and ROC characteristics, the model could be further refined, to include a more realistic model of the inspectable region. The dependence of the shape of this region on the speed for the robot was included and the exact area was determined for the surface properties of a crankshaft.
An additional model was required for flash thermography. A series of images have been taken by TRIMEK at the end of the reporting period. These images will provide the basis for the development of the flash-thermography model. Between month 19 and month 24 Profactor and Infratec jointly developed the pulse phase thermography process required for the carbon parts inspection on the TRIMEK demonstrator. The main goal was to inspect an adhesive layer between two components of the sideblade. The adhesive layer is distributed along the edge of the side blade and varies in width, depending on the process. This inspection task was defined as a consequence of the results achieved between month 12 and 18, where the detection of the smaller defects, such as inclusion or gaps turned out to be infeasible within the scope and limitations of ThermoBot.
Development work was mainly focused on first identifying pulse-phase thermography as a good method for solving the inspection task and then tuning the process parameters to optimize the contrast of the adhesive layer in the images. A laboratory setup at Profactor was used to develop a process with substantial support from BAM and Infratec. The most promising process was based on pulse-phase-thermography (PPT) using 4 flashes of 6kJ energy each at a frequency of 0.04Hz. This process was later implemented by Profactor at TRIMEK, but with just a single flash. At the general meeting in month 24 of the project it turned out that no budget was available for additional flashes and that the TRIMEK robot could not carry four flashes. The image acquisition and PPT analysis was implemented by Profactor and transferred to TRIMEK. Infratec substantially supported the task using their IRBIS software.
Task 2.2 Experimental evaluation of tolerances during acquisition
During the setup of the demonstrator at Profactor, two main sources for uncertainties have been identified:
• Temporal uncertainties related to an unknown offset between the recording of the thermoimages and the position of the robot.
• Spatial uncertainties related to the transformation between the many coordinate systems.
For the temporal uncertainties, a significant amount of work was done to acquire time-stamped robot joint angles and high frequency and combined with time-stamped frames coming from the thermo-camera. Given the maximum speed of 100mm/s, the readout rate of 4-8ms for the robot joint angels and acquisition rates of up to 200Hz (with a resolution of 320x240) were found to be sufficient. This also considers that the image processing algorithms will have to adaptively locate edges of the parts in any case.
A second series of experiments has been conducted by Profactor to obtain quantitative data and to further improve the laser-based thermography process with the aim of extending its flexibility with respect to speed, laser power and angle between laser, camera and surface. This additional flexibility will be a key step towards further optimising the path planning.
Task 2.3 CT measurements and matching to thermography data
The selected test samples of BRP and BSCT have been evaluated by BAM according to the needs for CT reference measurements. Using the simulation program aRTist, optimal measurement parameters (X-ray energy, scanning parameters, prefilter, etc.) have been selected to detect the different types of defects. Test samples with a diameter of 15 mm have been prepared for samples made of steel and with a diameter of 120 mm for samples made of composite material. CT reference measurements of 7 sample parts from BRP and 4 sample parts from BSCT have been performed with BAM μ-CT 225 kV X-ray source. 3D-CT volume data have been evaluated according to defect parameters (length, width, etc.) for matching with thermography data.
Furthermore, the thermography group of BAM also performed experiments to generate images from a few selected metallic test parts (crankshaft). By aligning and transforming these images a match to the CT data could be made. This included (after aligning, transforming and thresholding) a pixel-wise comparison between the images. From this comparison the sensitivity (does it detect all defect pixels) and specificity (does it detect only the defects) could be calculated. This initial evaluation done by BAM was the basis for the statistical analysis (task 2.5).
This work was completed before month 18. Experiments were done to compare magnetic particle inspection (Magnaflux), thermography and computed tomography.
Task 2.4 3D scanning measurements and matching to thermography data
The 3D scanning system has been prepared for inspecting the crankshaft pieces provided by BRP and the composites pieces provided by BSCT. The scanning system consists of a high accuracy non-contact optical sensor (OptiScan), specific software to process the information obtained in the scanning process and of the Stäubli robot, all installed and used at TRIMEK. Moreover, the pieces provided have been scanned at high speeds (up to 100.000 points per second) and all types of details on the parts’ surface, from free-form shapes to fine details have been captured.
Matching to thermography data did not reveal any correlation between the defects visible in the thermographic images and in the 3D scans. Therefore, this task was finished. The 3D scans, however, provide a basis for generating an inspection path for the robot in cases where no CAD data exist. This work was completed before month 18.
Matching to thermography data did not reveal any correlation between the defects visible in the thermographic images and in the 3D scans. The 3D scans, however, could provide a basis for generating an inspection path for the robot in cases where no CAD data exist.
Task 2.5 Statistical Analysis: POD and ROC
At Profactor a series of experiments were done to vary process parameters of the thermo-imaging process. Laser power and the robot’s speed were systematically adjusted to identify the range of possible process parameters and to provide a data basis for the POD and ROC characteristics. BAM used the resulting data to generate the respective statistics (Deliverable D2.3). The statistical analysis was made significantly more complex by the fact that thermography also includes software algorithms for processing the images.
A useful procedure for evaluating the algorithm parameters could be developed with the help of UNIPD. The results of the analysis revealed that the probability of detection follows the typical (expected) curve if laser power is chosen as parameter, while it does not show a well-defined behaviour if the processing speed is used as parameter. ROC analysis showed that all 3 methods (CT, magnetic particle inspection and thermography) are able to identify the characteristic properties of a defect.
CT clearly provides the most complete data, but there is no doubt that thermography has a good potential of replacing magnetic particle inspection.
Only one of the crankshaft sample parts with a “microstructural inclusion” produced inconclusive results. It generated a signal in magnetic particle inspection, but neither CT nor thermography were able to reproduce this signal. This particular sample will be further investigated, probably by cutting it and inspecting it under a microscope.
This work was completed before month 18 and the results are documented in deliverable D2.3.
Highlights of most significant results
Thermographic processes for metallic parts inspection (laser-based) and carbon fibre parts inspection (flash thermography) developed and implemented; experiments for improved process model completed; comparison with CT completed; POD and ROC analysis finished.
WP3 - Robot path planning
The objectives of this workpackage are to develop methods for autonomous static path planning based on 3D CAD data, to develop dynamic path planning methods based on visual feedback and to develop on-line simulation tools for collision detection and work-cell simulation.
Task 3.1 Implementation of the process model
The implementation of the initial process model, based on a very rough analysis, consisted of a ring-shaped, circular region around the laser spot. This area was projected onto the CAD considering edges and the local geometry of the part to reduce the ring to the “inspectable” area.
Based on the experiments done for the POD and ROC analysis, the model was further refined. The main parameters that were added to the model were the laser power, the processing speed and the actual, intrinsic camera parameters. The ring-shaped region changes its shape substantially when the laser is moving across the surface. With increasing speed, the “backside” of the ring becomes more elongated, while the front side is compressed. This may proceed until no inspectable area remains on the front side and the ring finally turns into a crescentic shape.
Camera calibration was done to obtain the intrinsic parameters of the camera, such as the centre of the optical system on the chip and the actual focal length (see next section).
This task ended according to plan in month 15.
Task 3.2 Static path planning based on enhanced CAD data
The first step in path planning is to generate a dense 3D point cloud of the part, where each point is augmented with a vector pointing outward. The next step is “coverage planning”, which tries to identify an (ideally) minimal set of viewpoints needed to fully cover the area to be inspected. This is done in parallel with the inverse kinematics solver that checks the reachability of these points. In a further development step the path that was generated also considered the kinematics of the robot in order to avoid that the robot often switches between different configurations thus wasting time during the inspection. Finally, close to the end of the reporting period, the path that covers the whole part (generated by Profactor) was imported into the work cell simulator of IT+Robotics to create collision-free path that fully covers the part.
Tests showed that the overall inspection time was a tough challenge. Given the maximum processing speed (speed with which the laser spot moves across the surface), cycles times of less than 1 minute are very difficult to obtain even with an optimized path. IT+Robotics designed an algorithm that finds the optimal (less time consuming) zone visiting order. The computation finds the trajectories that permit to the robot to pass from a zone to another as well. Regarding optimization, IT+Robotics developed a plan to split the part into different zones and generate optimal paths for each zone. This takes advantage of using optimized solutions for particular geometries such as cylindrical shapes or flat areas. Nevertheless additional measures are required and the partners believe that improvements can be achieved by widening the inspectable area by using the scanning system in the laser that would e.g. allow superimposing quick oscillations in the laser spot to create a larger spot.
In the period up to month 24 the “coverage planning” for the laser-based thermography was finished. Path planning was implemented by Profactor in collaboration with ITRobotics. The first step in the planning process is to generate a dense 3D point cloud of the part, where each point is augmented with a vector pointing outward. The following step is “coverage planning”, which tries to identify an (ideally) minimal set of viewpoints needed to fully cover the area to be inspected. This is done in parallel with the inverse kinematics solver that checks the reachability of these points. It also has to be avoided that the robot often switches between different configurations thus wasting time during the inspection. By splitting the part into different zones that are then processes separately and linked together a substantial simplification of the coverage planning could be achieved and it also gave rise to some optimization potential in terms of inspection speed.
The second part of the path planning is an iterative process, where collisions are checked and the path is adapted in order to ensure that no collisions occur, while still fully covering all relevant areas. This iterative process includes a data exchange between the modules of Profactor and IT+Robotics. The overall iterative process was implemented by IT+Robotics. These activities had strong links to task 3.3.
Tests showed that the overall inspection time is a tough challenge. Despite all the optimizations that were done, the target of 20-30 seconds for the crankshaft remained out of reach (see also assessment WP5). Theoretical calculations using the maximum surface speed of the laser spot (188mm/s), a maximum spot size of 5mm and a spiral-like motion along a cylinder with the size of crankshaft showed a scanning time of 2.4 minutes. More realistic values indicated scanning times in the range of 5-6 minutes.
Task 3.3 Offline Workcell simulation and robot program generation
IT+Robotics set up the workcells of both demonstrators in their simulation software. The workcell contained the main elements (robot, laser, camera, part) and their relative positions and was a virtual duplicate of the demonstrators set up at Profactor and TRIMEK. The final design was finished around Month 15, when the robot had been installed at Profactor and the first demonstration was finished so that the whole setup was defined. The TRIMEK demonstrator was modelled later, at the end of month 17, when the setup was finished after Profactor had completed the first integration at TRIMEK.
The overall software structure that is based on TRIMEK’s existing concept for their measurement systems and robots, was further developed with IT+Robotics, TRIMEK and Profactor. A generic interface to the TRIMEK robot was developed that allows the specification of (a sequence of) robot positions coming out of IT+Robotics’ workcell simulator software. This software was also extended to directly generate robot programs for both types of robots (Fanuc at Profactor, Stäubli at TRIMEK).
In the second period IT Robotics finished the robot program generation for the Fanuc robot that is used in the project for demonstration and the COMAU robot that is used for testing at UNIPD and IT+Robotics. The workcell simulation was update with the latest version of the robotic workcell and linked with Profactor’s coverage planning. Whereas at the beginning of the period these were still separate tools that communicated via file interfaces, they have now been integrated into a single tool that performs path planning, work cell simulation, collision detection and avoidance and robot program generation.
Substantial effort was spent by IT+Robotics to improve the workcell simulation. A fully updated version became available on 27th June 2014 (month 30). This version was also used during the 3rd integration test. Features that have been added that mainly relate to the communication structure between the robot controller (Profactor/TRIMEK) and to processing of the feedback provided by the online contrast optimization (Profactor).
Additional work was required to make transformations between the robot and the part available for each image. These transformations are needed for the projection of the identified defects from the 2D thermo-images onto the 3D CAD model of the part.
Task 3.4 Dynamic path planning and contrast optimization
This task was started in month 14 with a slight delay accounting for the fact that the robot was delivered late. In the following months, up to month 18, a concept was developed about how dynamic path planning could be done. The agreement between IT+Robotics and Profactor was that dynamic path planning would not require the global, on-line re-planning of the path, which is neither necessary nor feasible. As described in the DOW, dynamic path planning accounts for variations in the material, changes in the surface properties and small deviations from the ideal shape of the part. It was therefore decided to use a set of registers in the robot’s control unit to define changes (offsets) for the single joints of the robot. Some higher-level control algorithm deal with the accumulation of these offsets, e.g. if a flat area is inspected and the inspection path is adjusted 0.2mm to the right, then all other elements of the path that relate this area also have to be shifted by that amount. The separation of the part into different zones as described in task 3.2 helps in this respect as it decouples the whole part into a set of smaller, independent regions.
The Fanuc robot at Profactor (and its control unit) has been chosen to allow this kind of on-line adjustment, while the Stäubli robot at TRIMEK does not have that kind of capabilities and the dynamic adjustment of the path could not be implemented there. The dynamic on-line adaptation of the robot path could be tested on the third workcell of the Thermobot project: the one installed at UNIPD. The controller of the COMAU Six industrial robot was improved by adding the COMAU C4G Open feature to the controller. This enabled a 1ms control cycle and allowed a direct control of the robot trajectory from a high-level software application like Workcell Simulator by IT+Robotics. Thus, even if the two robots used in the Thermobot demonstrator are not ready for dynamic on-line adaptation of the robot path (like most of the state of the art industrial robots), this innovative feature of the Thermobot project could be tested using the workcell of UNIPD. This will need to be further developed before being ready to industrial integration; such activity could be performed once the next generation of industrial robots will be fit with the new dynamic path adjustment functionality. These robots are expected to massively hit the market in the next few years: most of the leading industrial robot manufacturers have prototypes or top series robot with this functionality.
The goal of dynamic path (re-)planning and contrast optimizations is to ensure that the inspection process is able to detect defects even when there are variants in the surface properties or geometry of the part. Based on the results achieved during the tests at Profactor, it turned out that the following strategy is most suitable for the inspection process and more likely to resolve issues with the image contrast: the module for contrast optimization (see task 4.5) provides an output indicating whether the contrast in the image is still suitable for inspection. If contrast is too low, the robot stops, acquisition parameters (e.g. laser power) are adjusted and the robot returns to last position, where the contrast was still good enough and continue the inspection from that position. This requires the quick re-planning of a safe and reachable path from the current (stop) position to the previous (re-start) position and online simulation (see task 3.5).
Development work of Profactor was mainly concerned with the synchronisation of the “stop” signal and the robot positions. This synchronization is needed to determine the last “good” position with high accuracy. This is to avoid that the robot unnecessarily scans areas of the part that have already been successfully inspection.
IT+Robotics managed the communication with Profactor libraries using Protocol Buffers. When the trajectory is interrupted, IT+Robotics path planner module plans a feasible path between the current robot position and the last valid one. Then it continues the inspection from here. If possible, the path planning module tries to keep the same robot configuration to minimize the cycle time. Otherwise, the best joint configuration is used, taking in account the robot dynamics.
Task 3.5 On-line simulation of robot work cell
In order to allow dynamic adjustment of the path, collisions and the kinematics of the robot need to be evaluated to check the feasibility of certain adjustments in real-time.
IT+Robotics developed an algorithm of motion planning that permits to calculate a safe robot path. It means that robot motions do not produce a path that passes too close to the obstacles, considering also speeds and accelerations of each joint. In this way the robot has also more space in case of dynamic re-planning in order to optimize the contrast.
Moreover, to speed up the performance of the motion planning in order to satisfy on-line real-time constraints, the collision engine was modified to allow multithreading computation. The parallel approach allows taking full advantage of the new multi-core processor chips technology. In fact, the computer processors technology is increasing the number of CPUs on chip. It means that the system will reduce the computational time over the time.
IT+Robotics made substantial progress with respect to online simulation of the whole workcell and the robot. During the integration and demonstration in month 23 the simulation was able to show a real-time behaviour and to detect collisions in real-time.
This activity led to research work on data structures necessary to parallelize the virtual rendering engine. Moreover, motion and path planning have been revised and tested to improve robot planning performances.
During the last integration week at Profactor (Steyr, month 33), the interaction between the new software modules and the demonstrator was successfully tested.
Highlights of most significant results
Workcell models and simulation finished; process model implemented; coverage planning operative; robot program generator operative for Fanuc and the Stäubli robot.
WP4 - Thermographic image analysis
The objective of this workpackage is to develop thermo-image analysis methods for the segmentation and classification of defects and to develop methods that assess the reliability of the detection based on image contrast.
Task 4.1 Forward/backward projection of thermo images
A key element of this step is the geometric calibration of the camera. The calibration provides the transformation from the camera pixels to the world/part coordinate system (or vice versa). In (standard) machine vision calibration often uses checkerboard patterns in order to identify the projective transformation and any distortions that might be present. For thermo-images a checkerboard pattern has also be used, although with some modifications. A checkerboard pattern of dark and light squares that covers the whole field of view was produced by Profactor. This checkerboard pattern was briefly heated by a lamp, the dark squares heat up more quickly than the other ones. This generated quite clear images and reduced the usually blurred edges in thermo-images, because this process generates the necessary contrast to identify the corners of the squares that serve as main feature points for the calibration. Nevertheless, some post-processing was required to make the images suitable for the use with the standard calibration function of the Open CV library. UNIPD performed also calibration experiments with a new type of checkerboard: a calibration body that consisted of a regular structure of circular holes in a metal part was manufactured by BAM. By putting a diffuse heat source in the other side of the metal square is possible to image with the camera a matrix of hot spots with known positions and use this as the matrix of the corner of the checker board in the usual calibration process. This approach proved to solve the problem of low contrast edges generated by a paper checkerboard in thermographic images. This calibration body has been successfully used by UNIPD for calibrating the thermographic camera. This calibration provides all elements required for calculating the projection of the defect onto the CAD model of the part.
The second element is the extrinsic calibration that localizes the camera in the world and in particular relative to part/robot. This calibration process was done by Profactor by using a stylus (or needle) mounted on the robot in a known position relative to the robot’s tool centre point. By placing the stylus in the corners of the checkerboard pattern that is recorded also by the camera, the coordinate transformations between the robot, the part and the field of view of the camera could be established.
This set of transformations creates the full chain from the pixel coordinated of the image to the 3D CAD coordinates of the part. What has been neglected in this process is to establish the exact position of the part relative to the robot. This, however, is an industrial standard procedure that need not be further investigated in a research project and the consortium decided to skip this step by mounting the part in a fixed position on the robot (no grasping of parts).
The main result of this task is a set of transformations covering the full chain from the pixel coordinates of the image to the 3D CAD coordinates of the part. Using calibration methods that have been described in the previous reporting period, these transformations could be established and the main work related to that task was finished by month 18. However, during the integration on the demonstrators several topics still needed to be addressed by UNIPD in cooperation with Profactor. This included technical issues such as software bugs at the interfaces, but also clarifications about the use of the transformations. Work in this task thus continued with lower intensity up to the final stages of the integration.
Task 4.2 Adaptive and knowledge based image segmentation
Initially, developments in this task were delayed due to the lack of thermo-images, because the laser heating system was not available until month 12. After month 12 several image sequence were acquire with different process parameters that were used as input for the image segmentation. The first step of the segmentation process developed by UNIPD is to classify the whole image as “contains a defect” or is “free of defects”. This was done by analysing the shape of the laser spot. The key observation is that the laser spot loses its slightly elongated, elliptical shape, whenever there is a defect in the vicinity of the spot. Labelling of the images was done based on ground truth data coming from magnetic particle inspection and confirmed by CT measurements. Concerning the evaluation of the cracks, a basic implementation was done that builds upon finding high temperature gradients in the image. Further developments aim at using a sequence of consecutive images to further improve detection also in areas of lower contrast. This will also help lowering the cycle time of the overall inspection process. Also the issue of edges on the part or abrupt changes of surface properties (rough vs. milled) have still need to be included in the knowledge-based segmentation.
Regarding carbon fibre parts, Flash Thermography was chosen as an alternative technique and the necessary hardware (camera) was sent to TRIMEK early in month 13. Due to the fact that a serious defect in the camera (caused during transport) was not discovered until month 16, the TRIMEK system, hardware and software for the inspection of carbon fiber parts was available only in month 18. A preliminary set of image sequences were obtained after generating a heat pulse (generated by a flashlight) impacting the surface of the part. The diffusion of the heat inside the part results in a distribution of temperatures in the time sequence. After initial experiments some sets of images were produced, unfortunately the known defect in the CFRPs did not show up in the thermal images. From the thermographic images acquired in the TRIMEK demonstrator with carbon fiber parts, it turned out that it is not possible to detect defects that are inside the part, but only superficial ones; however, the capability of detecting defects that are inside the inspected parts would be a strong advantage of the Thermobot system over traditional inspection systems. To keep this advantage, pulsed-phase thermography (PPT) was chosen as an alternative inspection method. This choice turned out to be crucial, as PPT images are able to highlight the glue jointing together adjacent layers of carbon fiber.
For the laser-based thermography used in the metallic part inspection demonstrator at Profactor image processing was performed by segmenting the laser spot, analyzing its shape, extracting a set of descriptors to measure the shape itself, and feed a classifier-
The key observation is that the laser spot loses its slightly elongated, elliptical shape, whenever there is a defect in the vicinity of the spot. The detection is based on a classification method that uses a few features to characterize the shape of the laser spot. Cracks and other inhomogenities change the circular shape of the spot and thus enable the detection.
During the project, several different methods for measuring the shape changes of the laser spot were tested and compared. The first algorithm employed, Radial Density Profile (RDP), was thoroughly tested. Further tests were run using other texture measurement features, like Pyramid Local Binary Pattern (PLBP) and Morphological features. The features were also exploited to build an ensemble, which was then used to feed a classifier.
In order to deal with reflections and other heat spots on the part that may occur during the inspection process, information about the expected position of the laser spot in the images is used. This allows the identification of the spot that is coming directly from the laser in contrast to e.g. reflections.
Regarding the detection of defects in CFRP parts, after resolving the open issues related to carbon fibre parts inspection and successfully setting up the acquisition process together with the PPT pre-processing steps, a first set of images was available for the development of the algorithms. A second larger set was originally planned for month 25, but was only acquired in month 31, so that there was the need to substantially revise the algorithms late in the project.
The final algorithms detect defects in the bonding layers by comparing the inspected part with a given reference, that is considered defect-free. The system exploits a set of filters for removing the high texture content and highlight the regions where the gluing layer can be found. The algorithm is capable of highlighting a set of Regions Of Interest (ROIs) on which the following steps of the algorithms will focus. The ROIs of both the inspected part and the reference are compared, and mismatches are highlighted.
Task 4.3 Defect classification and decision making
In D1.3 the end-users BRP and BSCT identified the outline of criteria that are to be used for the automatic classification of the defects and for the final accept/reject decision. Defect classification has been implemented by UNIPD using a linear SVM (Support Vector Machine) classifier exploiting the features developed in Task 4.2 as input. The best results were obtained using the Local Binary Pattern (LBP) feature evaluated on the image region around the laser spot. Such region of interest is rescaled to a fixed dimension in order to keep the dimension of the LBP descriptor constant. A classifier was trained on these images to detect the cracks. The SVM classifier was trained following a leave-on-out protocol, and cross-validation has been performed during the training phase. Experimental results showed an area under the ROC curve of 0.9337 indicating a very good classification performance. Experiments involved the main type of defect of metal sample parts, namely the crack. In principle, this approach can be extended to other kinds of defect by running again the training of the SVM classifier, even though modifications of the segmentation and texture analysis algorithms could lead to performance improvements.
Given the lack of thermographic images showing defects, to further develop image processing algorithms, especially for CFRPs, UNIPD decided to look for other image datasets which could provide images similar to the ones expected by the two Thermobot demonstrators. Similar images were identified in medical and biological datasets and new algorithms for image classifications have been developed on those images. Moreover, UNIPD also explored the possibility to generate thermographic image using induction heating in the workcell installed at UNIPD. Because of security issues, both laser and flash generated images could not be implemented in the lab at UNIPD, but induction heating using small probes does not pose these problems and was considered as a third (or forth considering also pulsed-phase thermography) excitation source.
Additional sample parts were made available by BRP, so that a set of 12 whole crankshafts exists, for which full image sequences have been generated. A classifier was trained by UNIPD on about 6000 frames that were manually labelled to provide the crack/no crack information for the training. The data set was split into 12 sequences, one for each sample part. A leave-one-part cross-validation protocol was exploited: for each fold, 11 parts were used for training, and the last one for testing the system. Detection accuracy per frame was reported to vary widely between 70% for the majority of the sample parts, but dropped down to 25% for a single crankshaft.
The classification software has been integrated into the image analysis module. Active learning techniques were also implemented for reducing the effort needed for collecting the training samples. This is a crucial point, as the correct training of an SVM often requires a very large dataset in order to avoid overfitting.
The presented classification results are a basis for further development that aim at generating a good/bad decision for the whole part. The concept that has been developed is to analyse sequences and remove spurious false classifications. The assumption being that a crack will appear in multiple consecutive frames, while false classifications will appear more or less random.
Task 4.4 Online simulation of thermodynamic process
The first 6 months of the project were dedicated to an initial simulation of a simple thermographic process and to compare this simulation to experimental data. The goal of this comparison is to obtain an early assessment whether online simulation has a chance of being successfully used as a basis for a segmentation algorithm. Work has been done on setting up the theory and a basic simulation task. Once the basic feasibility was established, work on this topic was interrupted until about month 15 of the project, because the implementation of online simulation was planned only for the last year of the project. After month 15 the concepts have been further developed and different methods have been investigated for the real-time solution of the thermodynamic equations. Explicit methods seemed to be more usable because they require less computing power and the stability criterion could be satisfied as we aim for prediction only over a few frames. The initial conditions is an image taken by the camera and the evolution of this image over time is predicted and compared to the real image, checking for substantial deviations.
During the third year, the online simulation of the heat flow was completed. The goal was to develop heat simulation that can run in parallel to the inspection process and generate artificial images. By comparing these images to the actual image, defects can be detected. The key element was to develop finite element methods that are quick enough to be run at the speed of the inspection process. The details of the methods are described in deliverable D4.6. A key method to achieve the necessary speed, was to limit the volume (voxels) for which heat simulation needs to be done. By estimating which volume elements (voxel) will significantly contribute to the heat flow, the number of voxel could be reduced by more than 50%. Also the level of discretization was investigated in order to have as few voxels as possible.
From the heat equations a set of linear equations has been deduced that are then used in the online simulation. Again, the details were presented in deliverable D4.6.
This model has been implemented and integrated into the demonstrator. The online heat simulation is considered to be an alternative option for image analysis and defect detection. Its computational requirements are, however, challenging for the computing equipment typically used in such inspection system.
Task 4.5 Automatic contrast estimation
This task will provide the input for the dynamic adjustment of the path by assessing the image quality and deducing changes in the path of the robot or other process parameters that may be adapted to improve the detection of cracks. The basis for this work has been developed during the experiments done also for the POD and ROC analysis. During this work, knowledge has been generated about the detectability of cracks under certain process parameters.
The main concept that proved to be useful for that purpose was to fit a Gaussian function to the temperature distribution in an image window close to the laser spot. This fit has the advantage that it reduces the whole heat distribution to a few numbers, such as the standard deviations in both directions, that provide an indication about the available contrast.
The standard deviations were calculated along the principal axes in order to be invariant to the direction of motion of the laser spot relative to the image axes. Results indicate that the typical image with good contrast shows standard deviations in the range of 15 to 18, which low contrast images have standard deviation of 33 to 36. The factor of 2 between these values indicates that there is sufficiently fine distinction between high an low contrast images.
These algorithms were implemented as a separate module and included in the demonstrator, where it was linked to the path re-planning module.
Highlights of most significant results
Image analysis for defect detection and classification on metallic and carbon fiber parts. Forward- and backprojection of defect onto the 3D CAD model.
WP5 - Assessment
The objectives of this workpackage are to integrate the single modules into the demonstrators that are set up every 6 months and to assess the technical results of the project on the test cases defined by the end users.
Task 5.1 Evaluation on basic inspection task
At the end of month 6 a first basic integration test was done by Profactor with support by UNIPD and ITRobotics. The goal of this integration test was to plug the single modules together and to check whether there are any problems at the interfaces. The modules that were tested in this first integration test were the communication with the robot’s control unit, the connection to the thermo-cameras, the interface to the image analysis module and the interface to the defect classification module. Additionally, interfaces were developed and testes that related to the offline (static) path planning. Overall the results showed that aside from minor compatibility issues (Win7 32bit vs. 64bit) the integration of the single modules worked well.
A major part of the work at Profactor was dedicated to setting up the demonstrator workcell. This included the design of the workcell (including a protective housing against laser radiation), the setup of the laser and the camera and the integration of various software components. Two major (formal) problems caused significant delays that could only partly be compensated.
The first delay was caused by the fact that for the robot a rental contract that is compatible with EC funding rules had to be set up, which lead the first robot manufacturer (Motoman) to withdraw their offer after 3 months and discussions with the second manufacturer (Fanuc) took further 4 months for completion. The Fanuc robot was finally delivered by the mid of month 12, therefore a smaller Stäubli robot had to be used temporarily as a replacement for the demonstration.
The second delay was caused by the laser that was supposed to be delivered in month 10 but did not arrive until late in month 12, so that it was not possible to generate large sets of test images. This delay have been partly compensated by developing the image processing software using similar images from previous projects and similar image datasets, so that the demonstrator is now roughly 1 month behind schedule.
Nevertheless an evaluation could be done to assess the main properties such as path planning results, accuracy of the synchronisation between camera and robot and defect detection in the recorded image sequences. The results are presented in deliverable D5.1 that has been prepared by TRIMEK.
The second basic integration task was done at the demonstrator for the CFRP parts at TRIMEK. In month 13 TRIMEK started with the work on the demonstrator by building the mechanical integration of the flash, the thermo-camera and the robot. This work was finished in month 17, before an integration week started, where Profactor added software components for acquiring images and controlling the robot via TRIMEK’s standard interface. By the end of month 17 the TRIMEK demonstrator was able to move the robot and to acquire images using the flash. Unfortunately, the set up of the image acquisition parameters revealed to be tricky and took a long time. In the end, it seems it is not possible to image inner defects in the CFRPs other than checking the gluing layer.
Task 5.2 Evaluation on static, autonomous inspection
By month 18 an integration test was done that combined the most recent version of UNIPD’s image segmentation, IT+Robotic’s path planning module and Profactor’s software framework. This integration test was done as planned at the end of month 18.
For the crankshaft inspection demonstrator at Profactor the following results were achieved by the end of month 18. The robot is able to acquire a sequence of images while moving the part. Joint coordinates of the robot are acquired every 8ms, while images are taken every 5ms. Occasionally single joint angles or images are missed, which may be due to real-time processing issues. However, this does not affect the overall performance of the system. Both the joint angles and the images are time-stamped with a synchronized clock to allow a direct correlation between the data. To investigate the extent and the cause of these problems, in the workcell set up at UNIPD based on a COMAU robot was decided to use the COMAU C4G Open feature to the controller. This enabled a hard real-time bidirectional communication with the COMAU robot and will be used to investigate synchronization issues. Full calibration has been performed including the world, part, robot and camera coordinate systems. The overall accuracy in a static setting is within ±1 pixel, which is sufficiently accurate for the application.
Processing of the images works well, however the algorithms have slight problems keeping up with the speed of 200 frames per second, which is mainly due to a time-consuming image conversion that can possibly be avoided. Also it will be possible to slow down the acquisition rate to 50 frames per second without affecting the quality of the inspection.
The sideblade inspection demonstrator at TRIMEK is based on flashlamp heating and oriented to carbon fiber parts. This material cannot be heated using a laser, since very high temperatures generated by the laser can damage it. In order to avoid such a problem a flash lamp has been used to heat the piece for the thermographic inspection. Trimek has assembled the demonstrator and integrated both the hardware and software required. Concretely, the demonstrator set up at Trimek integrates the following hardware components: the Stäubli TX90 robot with the CS8C controller, the flash EH Pro 6000 and the generator TRIA 6000 S from Hensel, the thermographic camera provided by Infratec and a PC. Moreover, Trimek has developed a hand effector for the robot so it could carry the thermocamera and the heating system. Regarding the software, this demonstrator integrates the robot controller, the TBRi interface and the path planning application developed until M18.
After the intermediate integration test developed in month 17 at Trimek, this demonstrator integrates correctly both the hardware and the software and works as required. Path planning tests are correctly developed and thermoimages are captured and saved, ready to be analysed by Profactor. Unfortunately, the captured images proved not to be useful for the software development realised by UNIPD, because no defects are visible. Images obtained show only superficial defects, which are also visible with a standard (visible light) vision system. A careful investigation on the image acquisition parameters and on the new pulsed-phase thermography technology is jointly running from Trimek, Profactor, and Infratec with the contributions of BAM and the feedback of UNIPD.
Between month 19 and 27 numerous tests were done on the demonstrator to assess various aspects. These included mainly path planning and the time needed for the execution of a whole path, tests with the image analysis and also initial tests related to online simulation and contrast optimization. The latter two topics are mainly part of the final assessment in task 5.3.
In month 24, during the general meeting of the consortium, the results of the demonstrations were discussed and further steps have been agreed. BRP also provided substantial feedback, which led to a priority list of properties that an automatic inspection system has to fulfil:
(1) The system has to fully cover all areas on the part that have to be inspected. There shall be no “blind spots”. BRP acknowledged that certain physical limitation cannot be overcome, but also confirmed that the crankshaft is the most complicated part and other parts of simpler geometry could be easily covered by the system
(2) The inspection system shall not require a combination with other methods (such as ultrasound or eddy current, …), because quality control is a non-productive element of the production process.
(3) Defects down to the minimum size as required by quality control instruction have to be detected.
(4) Cycle time has to be kept within reasonable limits. While it is possible to duplicate the system and thus double the throughput, the large investment will make the implementation less likely.
For the crankshaft inspection system the main assessment criterion is the classification accuracy of the crack detection, which was found to be about 70% for most test cases, but very low (~25%) for a single case. Further details are described in task 4.5. The second criterion that has been monitored is the processing time that is needed for inspection a whole crankshaft. Simulation results by IT+Robotics indicated a cycle time of 77s.
Assessment of the CFRP inspection system was mainly done by TRIMEK with support from all partners in their respective field. The evaluation was focused on the image acquisition with pulse-phase-thermography and the pre-processing of the images. Two different methods were compared: On the one hand Infratec’s “standard” implementation in the IRBIS software and one the other hand Profactor’s implementation for the online inspection tasks. After adjusting the parameters to the correct settings, very similar results were achieved for both implementations.
Task 5.3 Evaluation on dynamic, autonomous inspection
The final evaluation included the whole set of features that were developed for both of the demonstrators. The first demonstrator covered 2 different test cases by BRP (crankshaft and piston rod) while the second demonstrator covered the test case by BSCT (sideblade). Tests with parts of companies outside the consortium were also done, these are presented in the description of task 6.2.
For the metallic part inspection the main assessment criteria and their evaluation results are briefly described in the subsections below. Detailed results are presented in D5.3.
Parts inspected (size of a crankshaft) in 20-30s
This goal turned out to be very difficult to achieve and later in the project, when all process parameters were optimized, the following theoretical considerations showed the following limits: Given the optimal laser spot size of 5mm and the maximum speed of the robot, the spot may pass over the surface at 188mm/s. Assuming a cylindrical surface of the size of crankshaft, the ideal motion along the path that covers the whole cylinder would take about 2.4minutes. More realistic estimates that include motions of the robot to change its configuration where necessary showed a limit of about 5 minutes. During the evaluation the best possible path that could be achieved took about 6 minutes for the inspection of the crankshaft.
Detection of defects according to end user specifications
Defects that can be detected are limited to the rugged portions of the crankshaft, as the shiny parts cannot be analyzed. The analysis of the hot spot shape was successfully used to classify the images acquired by the demonstrator. Even though the image-based classification still shows a limited number of false detections, the per-sequence classification mechanism – looking for bursts of images containing a defect – is able to provide satisfactory results.
Autonomous path planning for full coverage
This criterion has been achieved to a very large extent. A process model including camera, laser and acquisition process has been implemented and used in the coverage planning. In an iterative exchange between the coverage planning module and the simulation the path is checked for collisions and reachability, while making sure that it is still fully inspection. Path planning could be shown to work well on two quite different parts in terms of size and complexity.
Contrast estimation and dynamic re-planning if inspection quality is low
A contrast estimation method was implemented that uses the shape of the spot and its intensity to determine whether an accurate inspection is still possible. This estimation was integrated in the demonstrator and could be successfully tuned to detect low contrast. This was demonstrated by inspecting a narrow gap that requires a small angle between surface and laser. Due to this angle the spot induced heat and the spot size decrease. This was successfully detected and robot motion control was triggered to stop the robot. After automatically adjusting the laser power, the robot repeated the path and that same segment then showed sufficient contrast for inspection.
Online simulation for defect detection
An online heat flow simulation was implemented, that performs a simplified version of full finite element calculation. By limiting the volume that was considered in the calculation the computing time could be reduced, so that an online simulation was possible. For the detection of defects a re-projection is required that converts the simulated 3D data into a 2D image. This re-projection could not be made accurate enough to allow a direct comparison between the simulated and the actual image. Also computing time issues for the re-projection did not allow this to be computed in real-time. The final result is that simulation of heat flow is possible in realtime, but additional computing steps are required that need further development and improvement. Applications of the simulation are possible, however, in more simple settings, where the motion of the object is not as complicated.
Second metallic part to be demonstrated
As a second part for testing a piston rod was provided by BRP. This part was chosen, because it differs substantially from the crankshaft in terms of complexity and size, while still being an important part in terms of later use of the technology. It could be demonstrated that the inspection process can be set up for the new test part without any part-specific tuning or adjustments.
Planning a new part < 15 minutes
Path planning consists of several steps, starting from the coverage planning over the robot motion planning to the robot program generation. An iterative process was needed to ensure that all relevant criteria are fulfilled (full coverage, no collisions, all points reachable). It was found that the computation of a path for a crankshaft takes about 4 minutes, which is less than one third compared to the original goal. The remaining time could thus be used to run further iterations in order to continue minimizing the time needed to execute the path.
Carbon Fibre parts inspection
The demonstrator set up at Trimek has been evaluated performing several experiments. The tests have been done with BSCT composite parts. The main objective of the demonstration phase is to test if the demonstrator is able to capture useful images of the bonding areas of the BSCT carbon fiber components. Several images have been acquired, using different exposition times and capturing with different frames per second. The results obtained show that the optimum acquisition parameters are: 25 seconds of acquisition time and a frame rate of 8 frames per second.
Regarding the path-planning sub-system, the evaluation was performed using the Trimek machine and the OptiScan sensor. The objective was to scan the positions obtained with the positioning system to test its accuracy. The main conclusion is that path planning sub-system is considered precise enough for thermographic inspection operations.
Highlights of most significant results
Both of the demonstrators were tested and evaluated relative to the technical goals foreseen in the description of work and also in comparison to additional specifications of the end users. The results are documented in D5.3
WP6 - Demonstration
The objective of this workpackage is to demonstrate the capabilities of the THERMOBOT system on the use cases and additional test cases provided by the business interest group.
Task 6.1 Demonstration on the parts of the end users
Originally the consortium tried to have the demonstration for the end users already at the end of month 11. But due to the problems with the delivery of the robot and the laser, the demonstration for the end users had to be postponed to month 14 (Feb. 2013). At that time BRP and BSCT visited Profactor, and the integrated system consisting of laser heating system, camera and robot was demonstrated. It could be shown to process a segment of a crankshaft, to record a series of images and to process these images with the image segmentation method of UNIPD that was available at that time. The key elements of the demonstrator (pre-defined path, recording a sequence of images, basic functionality for crack detection) could be successfully demonstrated.
The sideblade could not be demonstrated as at that time it was already decided that the inspection of the sideblade will be done on the TRIMEK demonstrator, whose construction just started in month 13.
After the set up of the TRIMEK demonstrator in month 18, a demonstration was performed to all the partners showing how the flash thermography system was able to acquire the required thermographic images. During this demonstration it was also shown how the carbon fiber parts can be adequately fixtured, as well as the optimal use of the camera flash to perform the inspection of the full part. End users BRP and BSCT provided also their views to this solution while the demonstration was performed.
In month 21 a demonstration was organised for a larger group of BRP, where the current status of the inspection robot was shown and additional test parts were investigated. The discussion mainly focused on the practical implementation of a ThermoBot system in a real-world production line and the question of how a certain area on the part, that often has cracks, can best be inspected. The key challenge is that the area is difficult to access and hard to see for the camera/laser system.
A specific demonstration was done for BRP in month 35, close to the end of the project during the final evaluation.
Task 6.2 Demonstration on parts of the BIG
At Profactor the following parts of companies outside the consortium were tested: a section of a large shell of ball bearings used in wind power applications. Artificial notches were introduced in the parts, but tests with the ThermoBot acquisition method showed that these artificial notches could not be detected. The second test part was a corrugated tube, where a weld seam had to be inspected. These tests showed positive results and high contrast between crack and background could be achieved for the samples. The third test past was a printed circuit board, where cracks in the board had to be detected. The results were positive and the crack generated clearly visible distortions in the heat distribution. All the results are given in detail in deliverable D6.1 and are not reproduced here.
At TRIMEK parts belonging to companies from outside the consortium were tested. The experiments performed are based on the creation of air spaces, placing foam between the layers before being glued. The objective is to test the capability of the Trimek’s demonstrator to detect layers with different thicknesses that have being glued as first layer. The results obtained show that the Trimek’s demonstrator is able to acquire accurate images that are useful to detect the presence or absence of air between composite layers, until a certain thickness (7 mm) where the demonstrator is unable to obtain clear images. So the experiments prove that there is an uncertainty area between 3 mm (air between layers identified) and 7 mm (air between layers not identified). This technical limitation provokes that the ThermoBot demonstrator set up in TRIMEK is certainly a useful defect detector for carbon fibre parts with a layer distance of 3 mm or less. From 3 mm to thicker parts the demonstrator cannot assure the detection of defects between layers in carbon fibre parts.
Highlights of most significant results
Demonstrations with parts of the end users and a set of parts from companies outside the consortium.
Potential Impact:
Impact
THERMOBOT aims at the development of a trans-sectoral inspection technology that can be used for metallic and composite parts and is relevant in many industrial sectors such as aerospace, automotive, and recreational vehicles but also construction and all other areas that use high-performance mechanical components. It thus addresses a large part of the European manufacturing industry that represents 21% of the EU GDP and provides more than 30 million jobs in Europe.
Developing the necessary enabling technologies to support EU manufacturing
THERMOBOT aims at the development of a technology that replaces a quality control method (magnetic particle inspection) that has been in existence for almost 100 years and is still executed manually even in otherwise fully automatic production plants. Automatic inspection systems for parts of complex geometry are rarely used in industry because of the costs of setting up the system (especially the robot program) for a specific type of part. THERMOBOT demonstrated that a technology for automatic path planning for complex inspection tasks is able to automatically and largely autonomously adapt to new parts (geometries, materials, surface structures) without the need for manual robot programming.
The different software modules developed for THERMOBOT show that the European industry can benefit from the flexibility of robot inspection systems not only in thermography, but also in classical vision systems, eddy-current systems or ultrasonic systems.
THERMOBOT provides a solution by reducing labour costs for manual vision inspection (in case of magnetic particle inspection by more than 90%, i.e.120k vs. 5k), in particular by creating an autonomous system that can adapt to different parts in very short time and does not need extended manual programming. As a side-effect it also solves the problem of documentation and of human fatigue which is always an issue in manual inspection processes. Moreover, THERMOBOT can lover also the maintenance costs. At the moment, the costs of maintenance in the European aerospace industry are substantial and most of the maintenance costs are labour costs. Airlines try to reduce these costs by having maintenance done in countries such as Pakistan, Libya or India. This is undesirable not only because highly qualified jobs are transferred to non-European countries, but it is also a safety issue. Even though these companies are certified by EASA (European Aviation Safety Agency) under Part-145 it is much harder to monitor companies in such distant countries than European countries.
Exploitation plans (ESS results, confidential)
The list below shows the exploitable results. The two demonstrators (No. 1,7) are the main results of the project. All other results (No. 2-6) are modules that are part of the “ThermoBot system” and are used in at least one of the demonstrators. The main route of exploitation for these results is inside the “ThermoBot system”, but some of them can also be exploited independently.
1. Robotic inspection system for CFRP parts (Lead partner: TRIMEK)
2. Module 1: Image Analysis (Lead Partner: UNIPD)
3. Module 2: Online Simulation (Lead Partner: Profactor)
4. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
5. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
6. Module 5: Inspection Software (Lead Partner: TRIMEK)
7. Robotic inspection system for metallic parts (Lead Partner: Profactor)
Description of exploitable results
No. 1 : Robotic inspection system (the “ThermoBot” system)
Innovativeness introduced compared to already existing Products/Services: The system developed is able to detect internal defects in composite parts, specifically defects in the glue layer (absence or presence) between sheets in carbon fibre parts. This is a great advance in inspection systems.
Product/Service Market Size: A conservative estimation could be of 1000 units for automotive industry, wind energy equipment manufacturers and other part manufacturers.
Competitors: Nowadays it seems that this solution has no direct competitors, but in the middle term producers of robotic inspection and of thermographic solutions could develop a solution adapted to the composite parts market.
Prospects/Customers: Potential users will be in those manufacturing sectors where the use of composite parts is increasingly important: automotive industry, wind energy components, aeronautics and, maybe, aerospace industry.
Cost of Implementation (before Exploitation): 350k EUR of additional development costs for the whole system.
Time to market: 2 years after the end of the project.
1. Which partner contributes to what (main contributions in terms of know how, patents, etc.): See Module 1: Image Analysis (Lead Partner: UNIPD)
2. Module 2: Online Simulation (Lead Partner: Profactor)
3. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
4. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
5. Module 5: Inspection Software (Lead Partner: TRIMEK)
No. 2 : Module 1: Image Analysis
Innovativeness introduced compared to already existing Products/Services: The module deals with the peculiarities of robotic thermography and is particularly made for this type of inspection.
Product/Service Market Size: 1 module per robot; see result No. 1
Competitors: Developers of “standard” software libraries such as HalCon, VisionBlox or similar might consider developing thermography modules.
Prospects/Customers. See result No. 1
Cost of Implementation (before Exploitation): About 80k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): UNIPD uses images provided by Profactor and TRIMEK. There are also links to motion planning (ITRobotics) because image analysis needs to know about the current position of the robot.
Partner/s involved expectations: Image analysis should perform, robustly on the demonstrators and on the test parts considered in ThermoBot. Further development will be required to turn it into a product.
No. 3: Module 2: Online Simulation
Innovativeness introduced compared to already existing Products/Services: The module deals with online simulation of thermographic processes. It uses a model to make short-term prediction about what the image should look like.
Product/Service Market Size: Due to the complexity of the developments, the module will first be used in more simple application, e.g. those where parts are only transported along a linear motion and where the geometry is simple. There could be a use for about 10 such modules per year in crack detection systems in the steel industry.
Competitors: The module is an alternative to exploitable result No.2 (image analysis).
Developers of “standard” software libraries such as HalCon, VisionBlox or similar might consider developing thermography modules (but not very likely).
Prospects/Customers: Large companies that integrate thermography system in high-value applications, such as steel industry. Potential customers include SMS Meer or Inst. Dr. Foerster.
Cost of Implementation (before Exploitation): About 250k EUR. This increased over earlier expectations, because the level of complexity is higher than expected.
Time to market: 3 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor uses information provided by BAM.
Partner/s involved expectations: The module should be integrated and working in the demonstrator.
No. 4 : Module 3: Robot path planning
Innovativeness introduced compared to already existing Products/Services: Robotic process planning exists only for simple processes. Thermography is much more complex, because it involves temporal (not only spatial) processes.
Product/Service Market Size: 1 module per robot for metallic part inspection; see result No. 7
Competitors: There are companies developing (semi-) automatic tools for path planning (e.g. in Germany and Austria.), however, thermography requires a quite complex, thermodynamic process model for path planning.
Prospects/Customers: Potential customers will be in the automotive, transport and ship-building industry and those that deliver parts for the automotive, transport and ship-building industry. Additionally, technologies used in the path planning module may be used for a much wider range of inspection tasks, that require continuous scanning.
Cost of Implementation (before Exploitation): About 100k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor contributes the process model and coverage planning, while ITRobotics contributes collision detection and path/motion planning.
Partner/s involved expectations: Path planning is an important element of the ThermoBot system for the metallic part inspection. At the end of ThermoBot, the robot should be able to autonomously plan a path for the inspection of a complex object (e.g. crankshaft).
No. 5 : Module 4: Workcell simulator
Innovativeness introduced compared to already existing Products/Services: Robotic simulation uses the dynamical models of the robot and is able to model time constraints.
Product/Service Market Size: 1 module per installation; see result No. 1 and No. 7
Competitors: Robot manufacturers offer simulation tools that also include 3D modelling of workcells.
Competitors are brand -oriented and they do not handle perception in the path planning loop and solution of time- constrained tasks.
Prospects/Customers: See result No. 1
Cost of Implementation (before Exploitation): About 80k EUR
Time to market: 2 years after end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Profactor and TRIMEK contributes by providing models and testing.
Partner/s involved expectations: … to extend the range of possible applications of the workcell simulator with the modules developed within the project
No 6: Module 5:Inspection software
Innovativeness introduced compared to already existing Products/Services: The inspection software developed will be adapted to the peculiarities of robotic thermography.
Product/Service Market Size: 1 module per robot; see result No. 1
Competitors: Inspection software developers.
Prospects/Customers: See result No. 1
Cost of Implementation (before Exploitation): About 200k EUR
Time to market: About 2 years after the end of the project
Which partner contributes to what (main contributions in terms of know how, patents, etc.): Image analysis provided by UNIPD and Profactor. Path planning software provided by IT Robotics.
Partner/s involved expectations: The software should be integrated and working in the demonstrator.
No. 7: Robotic inspection system for metallic parts
Innovativeness introduced compared to already existing Products/Services: The robotic system is able to plan the inspection task for a complex metallic part with a high degree of autonomy.
Product/Service Market Size: About 1000 units (automotive and parts manufacturers).
Competitors: There are companies that offer thermographic inspection (e.g. ThermoSensorik) and others that offer robotic inspection systems (ISRA, Vitronic), but we had not found companies that do both in a single system.
Prospects/Customers: Potential customers will be in the automotive, transport and ship-building industry and those that deliver parts for the automotive, transport and ship-building industry. Additionally, technologies used in the path planning module may be used for a much wider range of inspection tasks, that require continuous scanning.
Cost of Implementation (before Exploitation): About 400k EUR of additional development costs for the whole system.
Time to market: About 2 years after the end of the project.
Which partner contributes to what (main contributions in terms of know how, patents, etc.):
1. Module 1: Image Analysis (Lead Partner: UNIPD)
2. Module 2: Online Simulation (Lead Partner: Profactor)
3. Module 3: Robot path planning (Lead Partner: Profactor, IT-Robotics)
4. Module 4: Workcell simulator (Lead Partner: IT-Robotics)
Dissemination Activities
At the start of the project a web page (www.thermobot.eu) has been set up. The project web page (www.thermobot.eu) was updated on a regular basis with deliverables and news.
Further information about particular events is provided in deliverable D8.4
Industrial dissemination
In the first 6 months members for the business interest group were being sought. The business interest group is a selection of companies, typically end users, who provide additional test cases. IT+Robotics has been very active and successful in acquiring members for the business interest group (CIMM SPA, ZML SPA). Also UNIPD contacted for the BIG several companies: Tecnogamma (Mermec Group), Starmatik srl, Rea Robotics srl. Profactor contacted: MIBA, SKF, Schaeffler. Three companies agreed to join. Another, three companies provided test parts (see task 6.2) for test on the metallic parts inspection, but these companies chose not to become full members of the BIG. Nevertheless, their parts provided useful insights into the capabilities and limits of the developed technology.
Furthermore, Profactor took part to an exhibition at the Control fair 2013, where first results of ThermoBot were presented. IT+Robotics, UNIPD, and Profactor were present in an exhibition at the Automatica fair 2013 to disseminate Thermobot results.
Moreover, UNIPD and IT+Robotics disseminated the Thermobot activity both in 2012, in 2013, in 2014 in the “Corso Nazionale Automazione Industriale e Robotica” organized yearly by “SIRI Associazione Italiana di Robotica e Automazione”.
TRIMEK has been actively disseminating the Thermobot activity among potencial users: NECO,EPC, ITP, Renault Valladolid. Trimek also disseminated Thermobot at Metromeet, and other exhibitions such us BIEMH.
For carbon fibre parts inspection a number of companies have been contacted by Trimek during the project. Some of these companies provided parts to perform additional tests undertaken as part of WP6 (see D6.1). The agreements signed between Trimek and these companies specify clearly the confidential nature of this collaboration, so unfortunately they cannot be named or listed in this report. However, their contribution was essential to keep on deepening on the potentials and limitations of the developed technology.
General public dissemination
The project was presented by Profactor at the “CE Colloquium” held at the University of Linz. This presentation was attended by about 20 students from the field of computational engineering. At the end of May again Profactor presented the project at the annual Workshop of the Austrian Association for Pattern Recognition. The invited talk in the industrial part of the workshop was attended by about 40 participants. UNIPD took the opportunity to present the project at two large public events in Italy. The first on was the “NEXT fest” organized by the WIRED magazine in Milano, and the second one was the RoboVal event in Verona.
Scientific dissemination
UNIPD presented the results of the project in 4 journal papers and Profactor and BAM in 1 additional journal paper.
BAM and BRP prepared a presentation for the annual meeting 2013 of the German Society for NDT, title and abstract have been submitted in month 11 and were selected for publication. Two papers were presented at the HES-13 conference (Heating by Electromagnetic Sources) in Padua. The first paper was a joint paper by UNIPD and Profactor describing the approach implemented in the Thermobot project, the second one was a paper by UNIPD describing the image processing and image classification algorithms developed for Thermobot.
Further results of the ThermoBot project were presented at the following scientific conferences: ECNDT 2014 (European Conference on Non-Destructive Testing), QIRT 2014 (International Conference on Quantitative InfraRed Thermography) and the workshop of the DGZfP (Deutsche Gesellschaft für Zerstörungsfreie Prüfung).
A full list of publications resulting from the project is included in deliverable D8.4.
A major dissemination event that was related to ThermoBot was the IAS-13 conference that was held from 15th to 19th of July 2014. The conference was organized by UNIPD and the ThermoBot project was presented on 3 occasions: an Industrial Workshop, an Industrial Forum focused on Horizon 2020, and a technological and scientific exhibit.
UNIPD & Profactor organized the IAS-13 “Industrial Forum” with specific emphasis on the technologies and applications of autonomous systems in industry. Smaller lot sizes, a high number of product variants and complex requirements in terms of flexibility require autonomous systems that can quickly adapt to changes and also interact with users. Also research directions at the European level will be addressed that are currently aimed for in the “Factories of the Future” topics of Horizon 2020. IT+Robotics presented “Flexible visual inspection systems” with a focus on Thermobot technologies
The conference and its workshops were attended by about 350 participants, mainly from Europe, but also with a strong participation from Asia and the US in the field of robotics and artificial intelligence. The special motto was: “Bringing autonomous robots into industrial production”, in order to highlight the most recent results of autonomous robots and intelligent systems which are now sufficiently mature and robust to operate in industrial production..
In addition to this, UNIPD presented twice the Thermobot project in the FoF workshop organized by the EC in Brussels disseminating the project and its achievements among the other project leaders and coordinators of EU funded projects.
Clustering with other projects
On two occasions in March 2013 and March 2014 the ThermoBot project, represented by UNIPD, participated in cluster events organized by the European Commission to improve the Impact that the project will have. After the second workshop, a collaboration with the other three coordinators of the projects funded in the same call of Thermobot was started to implement project cluster activities for further dissemination, cross-fertilization, and knowledge transferm among the four project in the cluster. ThermoBot was part of a cluster, with a group of projects that have all been submitted to the topic related “robots for post-production and auxiliary processes”. Other projects included “MiRoR”, “CableBot”, and “MainBot”. The project were quite diverse, so it was decided to focus on joint dissemination activities.
A first event was a large web conference call attended (almost) by each partner of the four projects in which the four projects were presented in details. After this a mutual linking of the project in each project web site was implemented.
The second cluster event was a joint workshop co-located with IAS-13 in Padua (Italy) in July 2014. This workshop was organized by UNIPD under the lead of Stefano Ghidoni and included presentations of the single projects and a discussion about challenges that were encountered in these projects. The workshop was attended by about 25 persons.
The third was the participation of UNIPD as coordinator of Thermobot in a new EU Coordination and Support Action called “FOCUS: Factory of the Future Clusters”. The project builds upon the fundaments of five existing FoF Clusters, Zero Defect Manufacturing (4ZDM), Robotics, Clean factory, Precision Micro Production Technologies (High Micro) and Maintenance and support. The outcome of FOCUS will be a methodology for clustering and for industrial exploitation & take-up.
List of Websites:
Project web page:
www.thermobot.eu
Contacts:
Project Coordination
Prof. Emanuele Menegatti, Ph.D.
Intelligent Autonomous Systems
Department of Information Engineering
The University of Padua
via g. gradenigo 6/A
35131 Padova, Italy
Phone: +39 049 827 7651
E-mail: emg@dei.unipd.it
Technical Management
Dr. Christian Eitzinger
Profactor GmbH
Im Stadtgut A2
4407 Steyr-Gleink, Austria
Phone: +43 7252 885 250
E-mail: christian.eitzinger@profactor.at