Final Report Summary - PICKNPACK (Flexible robotic systems for automated adaptive packaging of fresh and processed food products)
Executive Summary:
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
Functionality that was achieved:
- Flexible tray sizes by a digital mould system, including a fast changeover system
- Bin picking robot module with cameras in the working range of the robot to assist the motion and failures
- Adaptive grippers to grab a wide range and fragile food products
- QAS module with (3D colour cameras, hyperspectral image analysis, 3D x-ray, microwave scanning)
- Flexible sealing and cutting with a laser
- Flexible heating system for convenient baking and cooking
- Flexible decoration by direct printing
- Integrity check camera system of the seal
- Cable robot to pick and place packages in a flexible working range
- Full track and trace system including RFID antennas and flexible software to keep track when products and line configurations are changing
- Software for flexible systems integration
- Cleaning robot and systems for cleaning the line from the inside (only food contact parts)
Functionality was performed on the focus products: vine tomatoes, grapes and ready meals/chicken breasts. Also other products are tested by specific modules.
The demonstration line operated at a cycle time of 17 packages per minute. The goal was to achieve 30 packages per minute, which is the industrial standard. The bottleneck was now in the robot handling and laser sealing. By increasing the capacity (motion planning) of the robot and increase the power of the laser, 30 packages per minute should be possible.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations. A life cycle assessment was carried out to assess the impact of the developed technology towards the industry and society. This all looks very positive for the PicknPack systems.
The line is demonstrated in a video and two live demonstration sessions at Wageningen (NL) and Holbeach (UK). Exploitation plans were made for all exploitable results.
Project Context and Objectives:
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
To meet these current and future challenges of the European food industry the PicknPack consortium will develop three types of modules that can package a wide variety of food products in packages that easily and quickly can be adapted to specific markets.
These modules work closely together:
1. a sensing module that assesses quality and other critical parameters of the individual or small batch products before and/or after packaging;
2. a vision controlled robotic handling module that picks up and separates the product from a harvest bin or conveyor belt and places it in the right position and/or correctly in a package; and
3. an adaptive packaging module that can accommodate various types of packaging with flexibility in terms of package shape, size, product environment, sealing and printing.
In the first year all specifications and concept designs of modules (functionality) were discussed and fixed. These specifications resulted in detailed designs of the line and modules. In the second year partners started to build their modules and line integration functionality. In the third year this work continued and modules were tested first in standalone operation and later in line operation. In the fourth year modules were placed in the line and line tests were done to achieve a high level, flexible and autonomous food packaging system. This line was demonstrated at the end of the project to the food equipment and food industrial sector. During this project a high level of progress was made. In parallel research was done on hygienic issues, societal impact, life cycle assessment and exploitation opportunities.
Functionality that was achieved during this project:
- Flexible tray sizes by a digital mould system, including a fast changeover system
- Bin picking robot module with cameras in the working range of the robot to assist the motion and failures
- Adaptive grippers to grab a wide range and fragile food products
- QAS module with (3D colour cameras, hyperspectral image analysis, 3D x-ray, microwave scanning)
- Flexible sealing and cutting with a laser
- Flexible heating system for convenient baking and cooking
- Flexible decoration by direct printing
- Integrity check camera system of the seal
- Cable robot to pick and place packages in a flexible working range
- Full track and trace system including RFID antennas and flexible software to keep track when products and line configurations are changing
- Software for flexible systems integration
- Cleaning robot and systems for cleaning the line from the inside (only food contact parts)
Functionality was performed on the focus products: vine tomatoes, grapes and ready meals/chicken breasts. Also other products are tested by specific modules.
The demonstration line operated at a cycle time of 17 packages per minute. The goal was to achieve 30 packages per minute, which is the industrial standard. The bottleneck was now in the robot handling and laser sealing. By increasing the capacity (motion planning) of the robot and increase the power of the laser, 30 packages per minute should be possible.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations. A life cycle assessment was carried out to assess the impact of the developed technology towards the industry and society. This all looks very positive for the PicknPack systems.
The line is demonstrated in a video and two live demonstration sessions at Wageningen (NL) and Holbeach (UK). Exploitation plans were made for all exploitable results.
Project Results:
1. Introduction
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations.
2. A flexible food production line
A work package to setup the line was committed to the following main objectives:
- To develop and test a fresh food production line, focused on quality assessment, separation tasks, handling and packaging of vine fruits and vegetables (case focus: vine tomatoes and grapes).
- To develop and test a processed food production line focused on quality assessment of a variety of processed food components, arranging these components in a package and closing the package by sealing (case focus ready meals)
- To develop and evaluate generic concepts and control within the production line that can perform on other products within fresh and processed food applications.
In the first half of the project, this work package focused on drafting the requirements of the first two objectives. This was defined in the first task where the definition of the lines would take shape in an iterative manner in cooperation with all the partners involved.
This task included a system analysis and bottleneck assessment. However, early on it became apparent that most requirements were hard to set in stone. Instead we focused on a range of values for each requirement and fall-back scenarios in case some could not be met. The design included requirements for hygiene standards in the food industry and should adhere to the economic viability.
To obtain the desired minimum level of performance of these requirements, multiple site visits were performed with a part of the consortium. For the fresh food lines, a tomato packaging facility was visited. To get an idea about the processed food production lines, members of the projectteam went to a ready meal factory,
Team members responsible for the software integration, adjustments to the design requirements were made for allow a flexible systems integration. The hierarchical concept of modules was introduced and templates were written for each partner to implement the communication protocols. Furthermore, overviews were made of each module’s input and output and integrated in the grant design of the lines.
It became apparent after a few iterations of requirements that modules for both lines could be integrated in a single flexible line. The design was then fixated on this single integrated food processing line for both fresh and processed food products.
Furthermore, a very important design principle was fixated. Instead of having conveyor belts for each module and attach them into a single line, an integrated backbone of sectional frames was chosen. The main benefits of this choice is that the web of trays does not have to be separated in the beginning and that at each point in time it is known where each product is located. This information is shared knowledge through the software. The modules could then be placed over the sectional frames and if necessary moved in a different order to accommodate different food processing functions. However, the downside of such a setup was later realized that the development was a major dependency for each module, slowing the overall process significantly.
After the requirements for the line were fixated as much as possible, each individual module could start fixating their module, moving from proven prototype concepts. WP7 supported in this stage the test trials of critical functions and working principles. Also here an iterative process took place; after modules became more finalized, more and more requirement parameters could be updated. Dependencies between modules crystallize. Teammembers of Work Package 7 were responsible for the integrated overview of the module and their parameters.
Part of creating this overview included quantitative simulations and mathematical models to calculate the effect of all parameters on the throughput of the product. The main performance measure of the line in both food products was set on the amount of packages produced per minute. Because the line is a serial process, the slowest modules would dictate the maximum speed.
In the second part of the project, modules started to emerge from assembly and came out of prototype status. To support integration of all the modules, a large food grade hall was found and leased at Wageningen University and Research to enable close support. The hall provided power, water and for cleanable a drainage system.
It became apparent that not all module owners could deliver the hardware in time and although some modules were delivered early, not all functionality was present upon delivery. WP7 supported the integration efforts by extending help also in finalizing the modules and connecting them to the sectional frames, which could not be performed before arrival to the food hall.
The thermoformer was the first module to arrive, after which it was connected to the sectional frames for package transport. Finding the right foil composition and thickness proved a challenge, putting a potential delay on the project as other modules required trays as input to perform integration tests. Preformed packages were made as a contingency plan.
Each partner visited the facility often to do their part for the integration efforts, usually for a week period at a time. A team of hard- and software engineers from Wageningen DLO supported their efforts, although often tasks were taken over to speed up the integration.
The final deadline was set at May 1st, 2016. At that point the line should be fully functional and working. However, this was not met for all functionalities, e.g. printer line controller software and automatic mould exchange of the thermoformer. At the deadline the functionality was frozen and all efforts were focused to create demonstrations for the demo day. Still, the main functionality was able to be shown and physical integration was completed, except for the cable pick and place robot which was demonstrated separately. The line was fully operational during demonstration time.
Improvements were continuously suggested by WP7 and implemented by the partners during the third period. A taskforce for the software was formed in February 2016 to ensure all modules were connected through software.
The cable pick and place robot at the end of the line was demonstrated separately. It functioned stand-alone and was connected to the line controller. This is a deviation from what was promised, mainly due to the partner’s struggle with hardware sub functions in this period.
The printer module was not integrated with the line underneath that provided the packages. The top foil was moved over the web of packages and rolled up thereafter, but did not connect to the packages with the planed motor system in the sectional frames. This was due to the limit in software engineering capability under supervision of DTI and could not be solved by other partners.
The laser was demonstrated separately in another room. The laser was not able to weld and cut the required PET foils in a robust and fast way. PET foils were too transparent for this laser and other foils as PP were not possible to mould in the thermeformer.
The line was integrated with the automated cleaning system. However, not all modules supported this. The second half of the line (X-ray, Printer and Cable Picking Robot) was not automatically cleanable. The first part was successfully implemented and tested on site several times without damage to the modules. This was tested after the first demonstration at Wageningen, due to the risk of possible breakdown involved. A plastic cover was implemented over the sectional frames to prevent water distribution to other places.
After all modules are integrated, the total line performance was measured. The values were inherently the minimum of all modules’ performances, because the sequential nature of the line; hence the bottleneck module will determine the values of the overall performance. The following indicators for the complete line were previously distinguished:
• Packages created per minute.
• Number of fresh and processed food types.
The number of fresh and processed food types the line could handle was 2: tomato and chicken breasts. The second indicator depends on how many trays per cycle the thermoformer creates. For tomato, this was 3 trays per 11 seconds, or 16 packages per minute. For the chicken breasts this was 6 trays per 21 seconds, or 17 packages per minute. The objective for the demonstrator was set at 30 packs per minute, which is considered state-of-the-art. The bottleneck for both lines was the pick and place robot for the food products. Other modules were able to speed up to the set goal of 30 packages per minute. For each individual module, it’s current performance is reported in deliverble 7.5 Report of prototype test under laboratory (and industrial) conditions, as well as suggestions for future improvements.
After the line was integrated, it was successfully demonstrated to the public, picking and placing tomatoes and switching to chicken breasts after a 5 minute change-over.
3. S&T-results on flexible systems integration
3.1 Pick-n-Pack data bus for improved integration and data exchange
The Pick-n-Pack data bus allowed all partners to interconnect to each other, share data in a generic decoupled way and interface to legacy software and hardware in a flexible and easy manner. The step change here with respect to traditional food packaging lines was the addition of auto-discovery and auto-configuration. This was achieved by exploiting state of the art solutions from the IT and networking community. The Pick-n-Pack data bus defined the communication backbone and improved flexibility and dynamism in the Pick-n-Pack line by allowing software entities to dynamically join and leave while all connections to other entities are configured or reconfigured automatically. Two communication primitives were implememted: whispering and shouting. The former was used in peer to peer messaging, while the latter was designed for group messaging.
3.2 JSON message models and 5P production model for improved traceability
The Pick-n-Pack messages were represented in JSON and this format was chosen, since it has a lot of traction in the world wide web community and allowed us to efficiently integrate and interpret the messages in web applications. The model includes a header and payload to first identify the type of message, its sender and receiver. Additionally, a URI of the data model and meta model can be supplied in the message to make it self-descriptive. The step change here was also the introduction of Universally Unique Identifiers or UUIDs at all levels in the Pick-n-Pack line, resulting in a much smaller footprint, in the form of meta-data, in the tracing database and keeping all other raw data locally at the Module that created this data. Additionally, the 5P generic production model described what (meta) data we need to store to effectively trace back to a particular production, which is a composition of a product handled by processors that executed a set of processes with a set of policies.
3.3 Queries and dialogs for improved robustness in data exchange
The rationale behind stateful queries is to move away from stateless data exchange which does not contain context and to introduce dialogs. The bidirectionality of a dialog makes a trade-off between scalability and reliability in a sense that you need to be aware of which peer you are sending messages to or receiving messages from but it allows handshakes to ensure important data is sent and received. In Pick-n-Pack queries were used to communicate with the ontology that was defined in the project around tomato trusses and to query the semantic database that was used to store data from the Pick-n-Pack data bus.
3.4 Life-cycle state machine for improved flexibility in deployment & reconfiguration
All Pick-n-Pack Modules and their devices have to work together to accomplish the food packaging task. The Line Controller was introduced to manage this process. To increase flexibility here with respect to identifying problems in one or more modules and knowing when the line can proceed, the Life-Cycle State Machine was designed. This Life-Cycle State Machine defines a set of states each Pick-n-Pack Module has to go through before it can safely join the food packaging task. This consists of allocating the right resources, setting up communication, configuring the correct software for the task at hand and running its particular task. It provides a homogeneous startup procedure for the wholePick-n-Pack line at different levels of abstraction. The Line Controller exploits this information to confirm if it is safe to continue operation or not. This behaviour was captured in the Stop light protocol. Modules communicate their state as ready (green), busy (orange) or not ready (red) and the Line Controller only sends a go signal to all Modules in the Pick-n-Pack line if it received a green light from all Modules. Additionally, adoption of the Life Cycle State Machine facilitates reconfiguration, both in software and hardware. This can now be done by emiting a reconfiguration signal, either by the Line Controller or directly by an operator from within the Flexible GUI. This signal is transmitted onto the Pick-n-Pack data bus and each Module change their state accordingly. For some Modules this can be done purely in software, while others require manual intervention. The same stop light protocol and Life Cycle State Machine come into play here.
3.5 Flexible GUI for improved visualisation and operator feedback
The architecture of the Flexib GUI is distributed, running partly on a Web-enabled server and partly in the client browser. The Flexible GUI receives data from the Pick-n-Pack databus through the Mediator pattern, which listens to the databus, converts this data to useable data for the web visualisation and adds this data to the semantic realtime database. The user interface itself is built in a modular way, meaning that it will visualise active Modules on the Pick-n-Pack databus and dynamically update events it receives from them through the Mediator. In addition, a Query interface was made availabe, to query the semantic database and visualise, for instance, statistics on the quality assessment or generate labels for the Printer Module. The design of the web application was further modularised by adding a hierarchy and creating a generic visualisation framework which can be reused independent of Pick-n-Pack.
3.6 Mediator design pattern for improved flexibility in software
The Mediator pattern was developed to decouple software entities which share data but not necessarily need to be aware of each others knowledge domain. The Mediator has knowledge about both the agreed data format on the Pick-n-Pack databus and data format used by the visualisation framework.Another way of mediation was done at the Module level, where internal specific knowledge of the Module developer needed to be converted to comply with the agreed Pick-n-Pack language. The same design pattern are applicable at a higher level of abstraction such as a Plant with several Lines, or a finer grained level such as individual Modules, Devices or even Algorithms. This is not only what makes these patterns flexible as required by the Pick-n-Pack objectives, but also generic such that they can be used outside the context of this project.
4. S&T-results on information and traceability
The RFID traceability system integrates the RFID technology for production process monitoring and data integration of the automated PicknPack line. The application of RFID technology and Information and communication technologies allows the comprehensive monitoring and data collection of products for data tracing or other further uses such as process optimisation and vendor managed inventory.
The design and development of RFID traceability system as a module of PicknPack line follows the new concept of digital manufacturing, which features smarter machines, machine collaboration, and data based system optimization. The results of the RFID traceability system are summarized as follows:
4.1 RFID enabled traceability system hardware integration
The system integrates the state of art RFID systems in the production process for product information tracking and tracing. The RFID reader/antennas with Internet connection can talk to the traceability software to notify the sensed RFID tags. The operation can be based on wired Internet or Wi-Fi. The system contains the following devices:
• An RFID reader with four antennas
• A computer with traceability software
• A handheld RFID reader for RFID and QR code reading
• A RFID applicator to place the RFID tags on product units
• A video camera for QR code reading
• A router to build the connection of the devices
One RFID antenna is integrated in the ‘pickrobot’ to monitor the input crate with raw materials. One is installed between ‘laser cutter’ and ‘packrobot’ to monitor the individual product units. Another antenna is installed at the end of the line to monitor the output crates. In addition, one extra antenna is used for information tracing purpose.
Since each product unit in the production process has a Universal Unique Identity (UUID), the data generated by each module in the line can be integrated with this UUID. The UUID can then linked to the current valid RFID labelled input crate and output crate. The UUID can also be linked to a RFID when an RFID is placed. Therefore, with a particular product unit, its associated input crate (source information), output crate (outgoing information), RFID number, and printed QR code are all linked to its UUID (online generated product information), and product information can be traced back from database with an RFID or QR code reader.
The integration of RFID recognition, data communication, and database technologies has successfully led to the real-time online monitoring of the PicknPack line production process.
4.2 RFID traceability system software
Traceability software is developed to deal with user interaction, hardware events, communication events, and database handling, etc. The development of this software focuses on the efficiency, reliability, and Simplicity in operation. Since the software handles the line communication, RFID reader, video camera, and database access, its efficiency and reliability are critical parameters to evaluate the system. The multi-thread technology of visual C# platform is employed to deal with the multi-task events handling.
Software GUI is designed to simplify the operations and enhance the efficiency. The user can navigate to the functions with top level navigation icons, and the selected functions are displayed in the centre workspace of the interface. The current running production job information is displayed on the right column of the interface. Then, communication messages, hardware events, and recognized RFID tag information is displayed at the bottom of the interface.
4.3 Custom-designed database for PicknPack automated food manufacturing system
A flexible database model is developed for the traceability system of PicknPack automated production line. The database model integrates source material information and outgoing information with online generated information. By using UUID as identification of online product unit and RFID as a machine readable ID for offline use, the relationship is built for data integration and information tracing.
Considering the specificity of PackNPack line, the flexibility in food categories and production strategies may result in different product process and different parameters. To deal with this problem, the data modelling integrates relational database with JSON descriptions. The SQL server tables are simplified by using text format to store data of some dynamic structures in JSON string.
With customized design, the database model therefore satisfies the requirements of PicknPack line in terms of flexibility.
4.4. Flexible inter-module communication
Inter-module communication is a critical enabling technology for the automated digital manufacturing systems like PicknPack line. The traceability system communicates to the modules in the line with a communication protocol based on Zyre. It can initiate a Peer-to-Peer (P2P) communication and one-to-multiple broadcast. Therefore, it can communicate to each module in the line for data communication and event notification.
There are two important communication tasks for the traceability system: (1) talk to ‘pickrobot’ to notify the current valid input crate, and (2) listen to ‘linecontroller/worldmodel’ to obtain the online generated data. With this flexible communication protocol, the traceability can collaborate with peer modules in the line for data integration.
4.5 Multiple product information tracing methods
The product information tracing is an essential task for the RFID traceability system. This system provides four approaches to trace information of a product: (1) Fixed RFID reader/antenna; (2) Video camera for QR coder reading; (3) Handheld device RFID reader; (4) Handheld device QR coder reader.
The system supports information tracing of products during production process and post-production. The handheld RFID/QR code reader with WIFI connection to the database can be used for information tracing anywhere, and information of the product item being traced is displayed on the GUI of the device interface.
The capability of the traceability system has been demonstrated in Wageningen and Holbeach in May and September 2016 respectively. The developed traceability system could be extended and customised for applications in practical food manufacturing environments and other industrial sectors.
5. S&T-results on Quality Assessment and Sensing
The main objective of the design and development of a Quality Assessment and Sensing (QAS) module which provides information on various quality aspects of all the individual products on the line. The collected information can then be used by the rest of the line for product sorting, robotic handling, traceability or adaptive packaging.
5.1 Product ontology
In the first stage of the project, a food (quality) ontology was created for the food products to be processed on the PicknPack food line, including vine tomatoes, table grapes and ready meals. This was done based on a listing of key quality parameters for each of the products as well as the acceptable range for these parameters.
The most important quality parameters which were identified, are:
- For fresh fruits: maturation stage, external defects, internal damages, colour, size and shape
- Ready meals: colour, topping, weight, size and composition of the meal
The most important marking and packaging parameters which were identified, are:
- For fresh fruits: shelflife, weight, packaging and traceability
- Ready meals: traceability
The product description provided by the ontology was then used as a basis for the design and development of the sensor technology and capabilities required in the QAS module. For more details the reader is referred to deliverable reports D4.1 and D4.2.
5.2 Design and functionality of the QAS module
The QAS module is designed as a system comprised of multipurpose sensors that each can assess individual products in a unique way. The sensors are divided over two submodules which were designed to fit on the sectional frame transport system which was designed in the PicknPack project. The first submodule was constructed in a collaboration between KUL, WUR and MU and contains an RGB camera, a 3D laser scanner, a microwave line scanner and a hyperspectral camera system. The second submodule was designed and constructed by KUL and InnoS and contains an X-ray unit. A more detailed overview along with further references can be found in deliverable report D4.10.
5.3 Module design and topology
Following the design guidelines and topology of the PicknPack line concept, the QAS module consists of a central module controller and a number of sensor units, called “devices”. Each device consists of a sensor, appropriate illumination and a matching data acquisition and processing unit. The module controller interacts with the devices as well as with the PicknPack line. All communication between the various actors in the QAS module is established via the PicknPack communication protocol which was developed in WP2 & WP7 based on the zyre protocol, even allowing different devices to use different programming languages.
All sensors in the QAS module are connected to a hardware controller that translates the real-time speed of the line as captured by encoders into a trigger signal that drives the data acquisition. This made it possible to collect data at sub-mm resolution at the speed of the line, during all stages of the stop and creep regime.
The overall design of the module makes it possible to add or remove sensors and functionality in the module in a flexible and efficient way, as desired. Furthermore, hygienic requirements were taken into account and the submodules were equipped with a tunnel system, making them fully cleanable.
5.4 Sensor technology
Based on the ontology and key parameters of the products, each sensor system was first developed in stand-alone units to realize the basic functionality and start the development of data processing algorithms. Hereafter, the sensors were integrated into the submodules. The following paragraphs give an overview of the different sensors.
5.4.1 HSI camera
The hyperspectral camera is sensitive in the visible and near infrared region of the electromagnetic spectrum (600 – 1000 nm) and achieves a spatial resolution of 0.4 mm/pixel. This system can accurately measure the unique spectral fingerprint of products. To optimize the illumination, ray tracing software was applied and in the later stages of the project, a cross-polarization system was developed to avoid that specular reflections would disturb the recorded images, thus increasing the quality of the data. The HSI system was used for the following applications and products:
• Truss tomatoes: sweetness (brix), ripeness and presence of skin damage on individual tomatoes in a truss
• Table grapes: sweetness (brix)
• Poultry: discrimination between cooked and uncooked chicken and turkey breasts.
5.4.2 RGB camera
The RGB camera is a line scanning unit that achieves a high spatial resolution of 0.25 mm/pixel. It was used to assess the ripeness of individual tomatoes based on their colour. Furthermore, data recorded with this sensor was merged with that of the 3D sensor to produce accurate 3D colour models of the individual products.
5.4.3 3D sensor
The 3D sensor applies laser triangulation to create 3D models with a resolution of 1 mm/pixel. These models are then used for the contactless estimation of the size, shape and weight of products. This was demonstrated for tomatoes in a truss.
5.4.4 Microwave sensor
The microwave sensor comprises of a series of 22 transmitter-receiver antenna pairs that can achieve a resolution of 2.1 mm in the direction of motion of the line. The sensor was used to determine the composition of a ready meal and to determine the degree of cooking (rawness) of chicken breasts.
5.4.5 X-ray sensor
The X-ray unit applies a novel technique to produce 2.5D images of the internal structures of products with a resolution of 0.135 mm/pixel. The setup consist of two X-ray source-detector pairs with crossing beams. This arrangement allows to determine the 3D location of internal features. The setup was used for foreign body detection and assessment of the internal structure of food products for defects.
5.5 Advanced data processing techniques
Besides the algorithms that were used to process the data of individual sensors, other flexible software tools were developed:
- Sensor fusion: Algorithms were developed that combine the information from multiple sensors to calculate new product features. This was demonstrated in two different way: (1) by mapping the data of the RGB and 3D sensor to create full colour 3D models, (2) by clustering quality features measured by the individual sensors to automatically determine the (user-specified) quality class of each product.
- Semi-supervised segmentation: An algorithm was created that helps a non-expert user to quickly train the QAS system for new types of products in an intuitive way .The algorithm was demonstrated for segmentation of hyperspectral images.
- Automated inspection of ready meal components: An algorithm was developed that can assess the composition of ready meals based on a database of pure component information.
- Deep learning: This technique applies artificial neural networks to perform complex tasks such as identifying cuts and puncture damage in tomatoes or worm damage and bruising in apples. The deep learning algorithms can be trained by non-experts.
5.6 Demonstration of QAS
The QAS module was demonstrated successfully during the two major demonstration events of the project. In May 2016, the full QAS module was demonstrated as part of the entire PicknPack line in Wageningen (NL). The module inspected tomatoes for ripeness, shape, weight, sweetness, the presence of internal damage or foreign bodies. Besides this, chicken breasts were scanned for moisture content, shape and degree of cooking (rawness).
For the demonstration in Holbeach (UK) in September 2016, a smaller stand-alone version of the QAS module was constructed that assessed tomatoes for sweetness, ripeness, weight and the presence of surface and subsurface defects. This module ran on the original software of the main QAS module, together with a simulated version of the line, demonstrating the flexibility of the concepts developed in the PicknPack project.
6. S&T results in robotic handling
Two (three) robotic modules were researched end developed in this project. The former called cable-robot (Pickable) and the latter called delta-robot. Thus this chapter will be also divided in two sections, one for each robot. The (third) robot that can clean the line form inside out is described in the S&T results for hygienic design of the line.
6.1 S&T Results for the cable-robot
As a summary of this chapter, it could be said that a complete innovative robot concept has been built and tested. It goes one step beyond the state of the art as certifies its patent (PCT/EP2014/078932) and no previous robots with all the characteristics of this one were previously developed.
This new robot is based on the same principles as industrial parallel robots to manipulate and place objects at high speed, but the rigid bars are replaced by metallic cables. The cables are connected to a mobile platform at one side and enrolled on winches that control the length and tension of the cables for the controlled movement of the platform on the other. Thus, a rectangular workspace is obtained suitable to grasp foodstuff from conveyors.
The novel robot can work in high speed pick and place applications keeping the same dynamics and cost of commercial parallel robots and making work volume more efficient, allowing re-configurability and improving the required footprint.
Parallel kinematics manipulators or delta robots, used nowadays in food industry, are not reconfigurable in workspace, have a no optimized footprint, which is a limitation in food industry to transfer the products from one conveyor to another one, and have spherical or cylindrical inefficient workspaces in which part of the robot reach is not utilized.
Specific added value:
• Reconfigurability of the robot: Only modifying the cable length and the upper plate, the cable robot can adapt its shape (x-y). This can provide to robot manufacturers a wide range of robots regarding to standard dimensions of conveyors and so on. It is possible to manufacture new robots with the specific necessary workspace, something not possible until now with commercial robots. That is an important advantage for robot manufacturers and for food industry too, because the robots will be easier to manufacture and therefore cheaper.
• Work space-footprint ratio reduced: The difference between the total dimensions (x-y) of the cable robot including the frame and its workspace is lower than commercial parallel robots. It provides a better exploitation of the available space in the plant (it needs only 200 mm more than the work space). This is an important advantage for the industry because the space is optimized. If other conventional ceiling robots are analyzed and compared with Pickable, taking into account that these conventional robot usually need a frame so as to stand their structure, the food print ratio is a valuable more competitive characteristic of this novel cable robot.
• Integrated with the PnP line. It is able to receive and send communications signals/events and data, and thanks to that, it is able to accomplish these expected functionalities:
• Direct and “clean” connection to the PnP line, without changes needed, as long as the subnet accomplishes the Zyre based network requirements: UDP and broadcast allowed, and port 5760 opened.
• Flexible line configuration changes implemented by Cable Robot in realtime thanks to world data model reception, parse and process. This allows real time changes related to batch, tray and punnet types and their dimensions.
• Cable Robot data shipment to the Line in order to be shown in general line GUI. This data includes for example information about final crates used
• More efficient workspace: The rectangular shape of the workspace takes advantage of a more efficient way of the reach of the robot. The delta robots have cylindrical working ranges.
This shape is not the best one so as to grasp stuffs from conveyor belts as is the most common situation for the food industry. However Pickable has a working range with a rectangular shape. It is the most suitable configuration for Pick and Place applications when moving food stuff from input to output conveyors. Taking into account that the working reach of the robot fits in such a suitable way to the target area, the selections of the robot is optimal. Moreover bigger than desired robots must not be selected anymore.
• Optimized conveyor tracking: Several additional elements were added to Pickable so as to reach a more feasible and robust system. Those elements were a specific conveyor and a machine vision system. These peripherals can improve the behaviour of Pickable since they allowed to get and individual operative system. The machine vision system on the conveyor helps Pickable to identify the position and orientation of the input crates. This information can be used, once Pick-cable will be a fully calibrated system, to reach any input stuff in an accurate way.
• Innovative torque control strategy for the cables: This is probably the main scientific output of Pickable. Its control modelling and design is quite challenging. For the control of Pickable, an innovative control strategy developed by Tecnalia has been developed and takes into account the necessity of always keeping in tension the cables. The use of the standard control of the drives is not possible. That is why a crucial step in this case is to totally open the controllers that can directly send torque control references. Once this parametrization of torque control achieved in the engine drives, the first step is to control the robot in position and tune the gains in order to have a good performance in terms of settling time, overshoot and static error. The real time environment used for the control of Pickable is RTX. Its main advantage is its ability to transform a hard real-time system on a standard PC Hardware, transforming Windows into a Real-Time Operating System (RTOS).
6.2 S&T Results for the delta-robot
In the third period of the project the pick-and-place robot module was fully built up and integrated in the PicknPack line. The complete system consists of the following components in the order as they appear during processing:
• Reading of the RFID tags on the harvest crates for tracking and tracing
• Automatic transportation of crates with foodstuff into the robot cell
• Detection of the crate and of the products in the crates by the 3D+colour vision system
• Planning of grasping actions to get the products out of the crates and into the trays
• Weighing of the grasped product
• Transportation of empty crates out of the robot cell
• Flexible software integration and synchronization with the rest of the line by means of p2p internet protocols as developed in WP2
• Integration with the cleaning robot for automatic cleaning of the robot cell
In the following, a short description of the achievements of the three main objectives is given, followed by a description of the performance in the line:
Development of a food product handling module that is flexible and fulfils the criteria of the food industry regarding hygiene, economy and safety
The hardware of the delta robot is largely based on a system that Marel sells to the industry and which is completely up to the standards of the food industry regarding hygiene, safety and economy. Added components meet the same standards. The grippers developed by Lacquey have been thoroughly examined on hygiene by Fraunhofer. Improvements have been made to full fill the necessary standards, which are high, as the grippers are in contact with the food. The vision system consists of off-the-shelve components and are housed in an enclosed vision box for hygienic reasons and for cleanability. All components meet standards of safety and economy.
Development of end-effectors that allow handling of a large variety of (non-uniform, delicate) food products
Two end effectors have been developed by Lacquey, one for grasping vine tomato and grapes and one for grasping chicken filet. In the third period, the fresh-produce gripper has been further developed with different fingertips that can easily be placed on and off the gripper to deal with variation in produce shape between different products and cultivars. Results of grasping are detailed below.
Development of a reprogramming method that allows fast change-overs to other products by non-specialized workers
Physically, quick change over is facilitated by a quick gripper-release system, allowing switches under 1 minute. The software is able to change to other products on the fly. To allow flexible switching to other products, the vision and control software uses parameter files which describe parameters of the products, the gripper and the grasping process. By putting the software in a (re-)configuration state, the required new information can be loaded and the system can be brought into running state again once the line is ready.
Performance in the line – Quantitative analysis
A quantitative and a qualitative analysis of the performance of the module in the line has been reported in deliverable D7.5. In this section, we give a conclusion of these analyses. For a more detailed description, we refer to D7.5.
Software communication: Internal communication between vision system and robot works reliably, as well as the communication of the module with the line. Flexibility with respect to the layout of the web of trays should be improved and a check if the right gripper is mounted should be included.
Grasping: The vine tomatoes and the chicken filets are grasped and removed from the crates successfully and reliably given sufficient space around the products. The dragging action to move products to empty space when they are touching does not work as anticipated, as too many products are damaged in the process. Grasping table grapes showed to be too difficult because of the deform-ability of the product.
Vision: The vision methods to determine grasping actions for vine tomatoes of different colours both non-touching and touching works well. Also for non-touching chicken filet and grapes, the methods are robust. For touching filets, the methods need further improvement to correctly segment all pieces. Entangled grapes are too difficult to ever be able to correctly detect with vision. Detection of the position of the crate works robustly.
Speed: Total average time for filling a tray with vine-tomato, including transport is 4 seconds. For chicken filets, this is 3.5 seconds. This is not yet up to desired 30 picks per second. Most of the time is spend on the mechanical actions of grasping (5%), transportation (70%) and releasing (5%). The robot can move much faster. However, no path optimisation has been performed to test what the effects on higher accelerations are on the product and the stability of the product in the gripper.
7. Adaptive packaging system
For the adaptive packaging system a range of functions was researched and developed. The main functions were:
• Flexible tray sizes by a digital mould system
• Flexible sealing and cutting with a laser
• Flexible heating system for convenient baking and cooking
• Flexible decoration by direct printing
• Integrity check
7.1 Digital Mould
PicknPack has developed several flexible and digital mould systems. A pin mould system with many small pins that together create a mould can change shape all the time and make each pack different. Moulds can be produced by rapid prototyping can create moulds very fast. PicknPack ended up using a brick mould system because this technology fit best to the food and packaging industry. The advantage with the brick mould system is that worker unskilled in computing can create and produce new moulds in seconds just as assembling LEGO. These moulds can automatic be digitalised and digitalised data can be used all over the flexible packaging line. Only 3 types of bricks are needed to each packaging height plus an extra for unused area. In case of four different heights all needed packaging sizes can be created using only 13 different kind of bricks. As in LEGO many special bricks can be added to the system in order to create logos, round or oval shaped packaging.
The thermoformer was also equipped with an automatic exchange system. The moulds are placed on a placed on a plate with the same foot print as the forming chamber of the thermoformer. Several of these mould plates are stored in the automatic exchange system. After designing and producing the moulds on the plates these are slided into the thermoformer. An automated system moves the mould plates into form chamber in seconds. This movement shall best be performed as the film is moved from one position to another. This can make each packaging different to any needs.
In the demonstrations both the brick mould and exchange systems documented. The other two moulding system was demonstrated off-line.
7.2 Flexible sealing and cutting
PicknPack has selected laser cutting and sealing in order to seal the top and bottom plastic films together to one packaging. The same lased was used to each individual packaging out from the long web with filled trays. This system work flexible as the laser beam is controlled by a mirror and a scanner to focus the beam over the packaging films. PicknPack was able to control the process. But PicknPack had problems to integrate the laser system to the total line. The problem was that type of laser need to be adapted to the type of polymer used in the packaging. PicknPack purchased early in the project a laser for PP plastic because this was a normal material when PicknPack started. Over the last few years a price drop in PET has out competed PP totally from the market. It was impossible to find PP to use for the demonstration.
The advantage using laser is that the laser can seal and cut any shape created in the design process. The laser can also create small perforations in the packaging film to be used for eMAP in packaging of fruits and vegetables.
7.3 Flexible heating
PicknPack also developed a microwave active printing layer to be printed on the top film of the packaging. The print was done with current ink printed in different patterns on the top film. The current layer works as shielding in the microwave own. Printed in specific patterns the energy can be directed into the food components needed to be heated and also shield other components from the energy. In this way it is possible to control the heating process of ready meals with different food components. The technology can also remove all problems about hot and cold spots in microwave heating.
PicknPack also developed a technology to print susceptors using the same print. PicknPack made many small dots of the same shielding ink. The result was a susceptor absorbing the energy and heating up the packaging. This technology can only be used for packaging materials able to operate under high temperatures as CPET or paper.
7.4 Flexible decoration
PicknPack designed a flexible decoration system based on ink-jet technology. Ink was sprayed on the packaging in 360 dpi resolution. Each colour need a line of print heads. The flexible printer receives files from the central control system and print the package just before sealing the pack. The food and packaging industry has been very interested in this flexible printing technology because shifting decorations creates many shifts on the packaging lines. Also the delivery of printed films to the industry is a practical daily challenge as the lead time will be increased with 1-4 months. PicknPack demonstrated the flexible printing technology in two colours. The system has room for full five to six colours.
7.5Integrity checking
PicknPack has developed a flexible integrity system using a multispectral imaging system that is easy to adapt. It covers the Visible (VIS) as well as a part of the Near Infra Red (NIR) part of the electromagnetic spectrum (400 – 1000 nm). The camera with sensitivity in the above-mentioned range was mounted over the area for welding and cutting the packaging out of the web. The camera lens looks through a tailor-made filter wheel that houses a total of 6 optical filters. As such, each package is imaged with great spatial resolution at 6 different wavelengths, and a multivariate image analysis (MIA) algorithm is used to find sealing issues. The system can audit all packaging to secure full integrity of each pack. If a pack has problems the integrity system can send a message to the welding laser to upgrade the welding in specific areas.
7.6 Other functions
The PicknPack system will be covered with shields in order to prevent spraying from the hygienic robot. The area of the laser and quality assessment module also need to be protected as the laser, X-ray and microwave all involves risks. If the whole system is covered it is possible to flush this covered volume with a packaging gas. This demand glove boxes on the sides in order to intervene failures. The system can create MAP (=Modified Atmosphere Packaging) and has the extra advantage that the products will be merged together in the special atmosphere which prevent “pockets” of oxygen inside the food. As the PicknPack line never was assembled a complete line this system was only illustrated.
8. S&T results hygienic food handling
During the project the focus of work package on hygienic food handling was to implement the Cleaning System into the PicknPack line and determining its performance parameters. Therefore the concept of the Mobile Cleaning Device which was developed in the first part of the project was put into operation after serval amendments and optimization steps. After this it was integrated into the line for demonstration and experimental purposes. The concept was validated so that in the end a functioning cleaning system was available which can drive automatically through the whole line in order to clean all modules consecutively with individual cleaning programs.
In addition, for the Delta Robot module a conventional CIP-system was integrated in order to compare its performance parameters with the ones of the novel Mobile Cleaning Device. To assess those performance parameters cleaning tests with regard to cleaning efficiency were performed exemplary on the Delta Robot. Therefore, parts of the robot were reproducibly coated with a fluorescent food model soil. Then those areas were cleaned with the different cleaning systems and different cleaning parameters. The cleaning progress was monitored with a camera system in combination with a UV lamp which made the fluorescent soil visible for the camera. In this way it was possible to quantify the cleaning rate of the cleaning procedures time-resolved and to compare their efficiency. The tests showed that the Mobile Cleaning Device is able to reduce water consumption and also to improve process safety with regard to hygiene e.g. by reducing spray shadow areas.
After completion of the cleaning test a kind of feasibility study was conducted in order to develop a concept of the Mobile Cleaning Device, how it could look like in the future and which additional features can make it still more adaptive and flexible. So this future concept has an even more product-like look and contains an optical camera sensor which can automatically detect soiled areas.
8.1 Mobile cleaning device (robot)
In the first part of the project the concept of the Mobile Cleaning Device was developed. It is an automated and self-driving machine which is moving through the whole PicknPack line and at the same time cleaning all modules consecutively with individual cleaning progress. Therefore, it has different nozzle types on-board and is able to spray several cleaning agents such as foam and water. This Mobile Cleaning Device was now integrated into the PicknPack line. Therefore, it uses the bars which support the product trays during production as rails on which it can drive. For the supply of cleaning agent it is connected to an automated hose drum. The hose drum is also driven by an engine and its rotation is synchronized with the speed of the Mobile Cleaning Device. In addition, the hose drum is standing on a movable table with wheels and with a docking station for the Mobile Cleaning. Hereby, the whole system can be moved easily through the whole factory so that the Mobile Cleaning Device can be used on several machineries.
Furthermore, the Software to control the Mobile Cleaning Device was also integrated into the line control system. Hereby, it is able to communicate with all the other modules of the PicknPack line. So when the position sensor of the cleaning device recognizes that it enters a module e.g. the Delta Robot it can send a message to the robot to start moving with a special pattern in order to support the cleaning progress by reducing spray shadows and moving relevant parts like the gripper closer to the cleaning nozzles.
8.2 Cleaning Efficiency Tests
After the line integration of the Mobile Cleaning Device several cleaning tests were performed to assess its cleaning efficiency in comparison the conventional CIP-systems and to determine ideal operating parameters. Those tests were performed exemplary for the PicknPack Delta Robot and the tunnels of the Sectional Frames.
For the tests in the Delta Robot the whole inner rear cover of the robot was coated with a fluorescent food model soil. In addition, a UV lamp and a camera (both IP69) were placed inside the robot to make the soil visible and to monitor the whole cleaning process. By this the cleaning process can be quantified. After the soil had dried for 20 hours the robot was cleaned with the different systems and with different operating parameters:
- Mobile Cleaning Device vs. Conventional CIP-System
- Cleaning agent (Foam) + water vs. water only
- Operating pressure: 3 bars and 4.5 bars
- Speed of the Mobile Cleaning Device: 2 mm/s (water only), 5 mm/s (foam + water), 10 mm/s (foam + water) speed values based on first lab scale tests with rotating spray heads
From the analysis of recorded images of the monitoring system the cleaning time was determined. This time was defined as the time which is needed to remove 95 % of the soil. Together with the flow rate generated by the two different cleaning systems in combination with the operating pressure also the water consumption was calculated.
The tests showed that in comparison to the conventional CIP-system with static nozzles the Mobile Cleaning Device needs more time to clean the robotic cell. This is due to the fact that the CIP-system consisted to two rotating spray heads mounted on the side covers of the robot while the cleaning device carries only one rotating spray head. But in return the Mobile Cleaning Device is able to reduce the water consumption by around 20 %. In addition, the tests showed that the Mobile Cleaning Device is able to eliminate spray shadow area due to its movement. Thereby, it is able to clean areas which cannot be reached by the static nozzles of the conventional CIP-system. So not only water consumption is reduced but also food safety is improved significantly.
Furthermore, the tests showed that using foam and water as cleaning agents instead of water only can reduce water consumption by around 60%.
8.3 Future Concept Design of the Mobile Cleaning Device:
The current version of the Mobile Cleaning Device is a prototype to show the feasibility and its advantages in comparison to conventional cleaning methods. The module will be developed further. Size will be reduced and the hygienic design will be improved to increase its suitability for the use in product contact areas. There will also be a version without wheels which is only carried by conveyors.
To make the device also more adaptive, it is planned to add an optical sensor system for automated soil detection. Since most food products contain fluorescent ingredients, it is possible to make them visible for a camera with a UV light. Both components will be integrated into the Mobile Cleaning Device. With this sensor system it will be possible to improve adaptivity during the cleaning process. It will be possible to determine which areas are really soiled and require cleaning and which areas don’t need to be cleaned. And it will also be possible to determine if all surfaces were successfully cleaned or if further cleaning is required.
Potential Impact:
1. Life cycle assessment
A life cycle assessment was carried out to investigate the impact of the PicknPack systems. In this assessment PicknPack focuses on the following objectives
- The development of a full life cycle picture of the automated flexible systems developed in the project.
- A sustainability assessment of the effects of flexible automation in packaging of fresh and processed food, including aspects like waste minimization, quality increase and energy/logistic optimisation.
- Use of well-stablished metrics to measure the sustainability performance: Life Cycle Assessment (LCA), Life Cycle Costing (LCC), qualitative social assessment and eco-efficiency.
- Demonstrate the environmental, economic and social benefits of the PicknPack automated/flexible systems in comparison with current conventional packaging equipment
Four assessment techniques used within environmental management and technology were implemented here. Life Cycle Assessment (LCA) was implemented at a first stage for the assessment of the potential environmental impacts of the PicknPack line vs. current benchmark packaging systems. The LCA was fully ILCD compliant since all the requirements were checked (see D9.1 for further information). The LCA considered a total of 24 combinations of tray size and type of machine. The amount of energy and plastic packaging material for each packaging size was obtained by extrapolation calculation, while the amount of packaging scrap was estimated in AutoCAD. Additionally, power consumption in the line was gathered through a combination of on-site of power measurements, both for conventional thermoforming/packaging equipment and the PicknPack lines. In the specific case of the PicknPack line, the power consumption was accompanied of a desktop analysis with all the partners. The consumption of water and cleaning agents in was considered as well. With all these information life cycle diagrams and specific life cycle inventories were prepared and modelled in SimaPro 8.3 software. Then, the potential life cycle impacts were calculated for all the systems in accordance with nine impact categories, based on the recommended life cycle impact assessment methods by ILCD. The LCA results were provided with and without the impacts of the food. Additionally, a sensitivity analysis was made in the LCA in order to consider the effects of the geographic location (UK, NL, ES) because of the differences on the Country energy mixes.
Based on the previous data collection and LCA work, the economic assessment through LCC technique was also made. The LCC rely on the LCA structure, so all the background data on flows of energy and materials was considered. These amounts were then multiplied by the economic cost related to them, although other cost categories beyond materials and energy consumption were included (i.e.: labour costs, capital charges, etc.). The cost of each packaging size/machine and type of food contained were subsequently obtained.
In a third step, a social analysis was carried out. Even though it was originally planned as a packer acceptance study in order to get a quantitative value for the Willingness.to Pay (WTP) this was no further possible because of the delays on the final layout of the line and the integration problems between the modules. Because of that it was decided to change the strategy to a qualitative social analysis aimed at checking the potential improvements in terms of (1) human resources, (2) productivity of the employees and (3) expectations on the of qualification of the employees.
Finally, an eco-efficiency analysis was developed in order to analyse if the PicknPack line was able to provide the lowest environmental impact and cost compared to conventional packaging equipment. Spider and scattered dispersion diagrams were created for such purposes.
2. Main dissemination effort
2.1 Workshops
A key deliverable was to organize six dissemination workshops with a wide geographical distribution of venues targeted essentially at industry to maximise the final impact of the project. This was achieved with three of the workshops being associated with significant industrial exhibitions and two being associated with the final demonstration of the project technical outputs. A total of about 300 attendees were registered mainly senior personnel from industry or commercial organisations. A list is included in deliverable 10.3. A comprehensive set of high quality publicity leaflets were designed and produced together with a professional level conference folder for workshop delegates.
2.2 Project website
A public project website was designed and set up to provide a major communication channel to disseminate our activities and outputs. All workshop presentations were posted on the website as well as regular newsletters, details of consortium members and their roles, technical presentations, journal publications and a regular stream of videos. In October 2015 the website had 3837 hits which increased in April 2016 to 19117 and finally to 55450 in September 2016.
2.3 Newsletter
A Mail Chimp tool used for disseminating newsletters has also assisted driving engagement and increased traffic to the site. Campaigns have an above average open rate and a low bounce rate indicating that the content is engaging and thus retaining visitors. Top locations have been the UK, Netherlands and the USA.
2.4 Social and professional online media
Twitter, Facebook and YouTube have been set up to disseminate the technical outputs of the project. The pages were continuously maintained. However despite repeated requests many partners are still not connected to social media so they cannot join the PicknPack channels.
You Tube channel analytics indicates that 83% visitors are male and remaining 17% female. Top watch locations are Netherlands, UK and Spain.
2.5 Industrial Advisory Board
An Industrial Advisory Board was established and formal meetings arranged. Support for travel and accommodation was provided for members to encourage attendance. Their input proved to be valuable in determining the direction of the project.
2.6 Other dissemination activities
Strong communication links were established with a number of organisations including the UK’s Food Manufacturing Engineering Group, Chilled Food Association, Food and Drink Federation and industrial companies including Kuka, ABB, Siemens, Omron, Nestle, Bakkavor and Unilever as well as the UK’s Robotics and Autonomous Systems, Knowledge Transfer Network the Northern Robotics Network a Consortium of some 50 companies from various sectors including nuclear and aerospace that cosponsored our 4th project workshop.
The IML Publishing Group which hosts the industrial / commercial A4E Trade Fair published full details of our workshop in their Process Engineering Journal, invited a series of presentations at their events and are planning an article on expected impact of the technology. New Food has published one paper and suggested a special issue on all aspects of the project. Full details of all published articles and presentations are given below.
2.7 Training
Two training events were planned in association with the final two workshops in the series. The Project Board however recognised the need to focus our attention on delivering a full working demonstrator at the 5th workshop so the training event was postponed to the September 2016 event which was held at the Holbeach campus at the University of Lincoln.
The event attracted 83 student registrants with an additional nine academic / industrial supervisors from a range of companies including Bakkavor, Nestle, Greenvale, Dalehead and the ADDO Food Group. An added feature was the demonstration of a large (100kg payload) robot cell moving bulk food material provided courtesy of O.L Systems.
Over the span of the project we developed a liaison with the E.C project Smart-E which has focused on soft robotics for manufacturing.
2.8 Publications
All partners were continually polled to generate material for the website and provide information on all publications, industrial presentations etc. so that we could generate a comprehensive archive of disseminated materials. Similar requests were made to archive all patent applications and the results are tabled in deliverable 10.4.
2.9 Demonstration
The PicknPack was demonstrated to 349 companies and students as follows:
Denmark 23 February 2015 the thermoformer under construction was presented to 40 companies from the Danish Food and packaging industry.
The Netherlands 26 and 27 May 2016 the PicknPack line was demonstrated as an almost complete functioning line. The first day with 48 and the second day with 54 companies as participants. The demonstrations in The Netherlands was done with the thermoformer, gripper, pick-n-place-robot and the quality systems all integrated. As the laser was unable to work with PET films the laser, printer, integrity and the cable robots was demonstrated as individual units. In another room a number of other PicknPack functions was demonstrated. These functions were the design process using the three different flexible mould systems. The Printer, exchange system, quality evaluation, integrity system, and the cable robot systems was explained although also demonstrated in the other room. The heating system, line integration, software systems, cleaning system and the RFID systems had own booths in this room.
Spain in June 2016 Tecnalia demonstrated the cable robot to several delegations with about 90 participants. The visitor was from: 1. 60 experts from RoboTT-net OpenLab; 2. Visitors from 4 different universities in USA and; about 30 Australian companies also participated in the demonstration of the cable robot.
England 13 September 2016 a special demonstration was organised for 36 companies. The demonstration in England included presentations of all modules in PicknPack. The demonstration was done as demonstrations of each single module.
England 14 September 2016 PicknPack together with University of Lincoln organised a student’s day with 81 students as participants. On the Students day some of the PicknPack modules was selected for a special presentation. These modules was Robot with grippers, Hygiene robot, RFID, Printer, Quality sensing, Laser and heating systems.
2.10 Project video
A professional project video was made to demonstrate the functionality of the line and its modules. Special animation was added to explain the flexible software and line configurations. Specific modules were also produces to explain in more detail the benefits of each module.
3 Exploitable results
PicknPack dedicated a work package to exploitation of the project results. The objective of this work package was to generate useful tools in order to orientate the exploitation plans of the project results. Moreover, these tools were shared among the partners to give them useful information about the market needs and requirements in order to guide their developments in a more realistic, adjusted and optimized way and to give them clues about opportunities in the sector. The work was divided in different tasks.
As a first task, evaluation of the acceptance of the robotic products in the food packaging sector was performed. The objective was to know the real profile of the market, in order to understand the real needs of the industry and the factors influencing the acceptance and implementation of automatic systems in fresh food processing plants in Europe. This was the first step to develop adjusted technology to the market and oriented exploitation plans. For reaching this aim, a wide survey at European level was done to food industries’ decision makers. In summary, the study investigated the acceptability of innovative automatic systems by the FOOD and POSTHARVEST industry. The investment intention by companies of both sectors, and the factors that push them to declare a positive or negative intention were analyzed. The study also provided a context to these factors. Thus the knowledge of the particular and common characteristics of both sectors helped to orient the approaching strategies to design the adequate exploitation plan (in task 12.4) for automation suppliers to facilitate a greater implementation in the food industry of today. The study helped the exploitation task by identifying the helpful (OPPORTUNITIES) and harmful (THREATS) factors that could affect the business in the current situation of the sector. The discussion and conclusions from this study helped to define main recommendations for the technology suppliers in order to orientate their business in function of the factors influencing the implementation decisions of the food companies. The results and conclusions of this work were useful to contextualize the exploitation plans described in Deliverable 12.5 “Technology Exploitation plan”.
Secondly, the analysis of factors impacting the Economic viability of automated food packaging systems was carried out. Moreover, a bibliographic research was done in order to have an orientated vision and costs and values of the business. Based in the gathered information, a tool to simulate economic viability was developed on Microsoft Excel™. This tool will allow defining different base scenario reflecting the current target industries. The simulation tool makes it possible to display results in a simple and comprehensible way through figures, graphs and tables. Simulation on a current target industry situation shows an estimated graph of its cost structures. Based on this results a return of investment table is calculated showing the different key financial concepts that will allow to know if a determined PnP investment will be viable in that current scenario or not. Finally with the performance data of the developed prototypes, it will be able the display of a new scenario which will tell the final benefits for the company after implementing these new PnP modules.
Evaluation of the societal impact of robotics systems in the food packaging sector is investigated, a tool for assessing the impact of the PicknPack line in the workers welfare in terms of occupational Risk Prevention (ORP) was designed, focusing mainly on the ergonomic aspects of the manual labour done by the workers. The objective was to define the strategy in order to check the line design, identify and reduce the ergonomics hazards present in it and minimize their effect on the workers’ health. For that issue a theoretical identification of the critical points was studied and suggestions of improvement were described. Moreover the methodology to follow was defined and explained to the partners. In the study several ergonomic risks and suggestions for minimizing them were done. Related to the influence of the PicknPack line on consumers, the impacts of the line on the final consumers of products were identified. The automation and the use of new technologies enable to have high process control and production flexibility. This allows meeting the expectations of consumers more accurately and efficiently than traditional forms of production. This impact was also confirmed with the acceptance study performed in the same work package, in which several European companies were surveyed about needs and perceptions about automation in their business. From this study the impacts on social aspects and the impacts that were related to the consumer’s product acceptance were confirmed.
The final objective of this work package was to Promote exploitation of the project results and stimulating new applications. The above described tasks helped as a tool for discussing with technology developers of the project about the corresponding potential exploitation paths. The first task by gathering form the acceptance study the information to draw up a SWOT analysis, and the economic task by providing an estimation of the impact that the implementation of the equipment in the business would have in the potential target companies.
Even though the Picknpack line as a whole system was not entirely functional, the different modules that composed the PicknPack project reached a different state of development during the course of the project. Most of them have reached a TRL between 4 and 7. Some of the modules were even quite close to market. After finishing the PicknPack project, the aim is each individual partner to keep on developing the technology, alone or in collaboration with partners, in order to reach a higher TRL closer to market. Under the life of the project a close communication between partners was kept in order to define and discuss the characteristics, benefits and exploitation plans of the achieved modules and results.
The objective of this task was to describe the results of the PicknPack project and the potential ways of their exploitation via different approaches for future scenarios in order to reach a closer level of readiness towards the market. With this aim several communications and two workshops were done under the project, in order to explain to partners the main concepts regarding exploitation and to ask them oriented inputs. This exercise was accomplished with the aim of not only them to describe the innovations, but to really translate the impact of the developments by describing the benefits of the innovations in terms that are understandable to the potential clients. One challenge of this task was to make the partners conscious of the importance of describing, to the potential clients, the added value of their research in an understandable and clear way. For that objective, the information was asked in several formats (questionnaires, tables, message maps, elevator pitch, etc.) in order to get clear answers to the problem solved by the results, a clear description of the solution, the benefits and differentiation from the competitors (unique selling point), among others. The two demonstrations that took place in Wageningen (Netherlands) and in Holbeach (United Kingdom) respectively, allowed not only to gather information about the requirements of the industry, but also the opportunity to make contacts and to build a strong network that will help in the promotion of further developments. Deliverable 12.5. Exploitation plans, is the summary of the developed or of exploitable outcomes of PnP project, and the elaboration of plans for their exploitation by partners involved in the project.
List of Websites:
Public webiste: www.picknpack.eu
Access deliverables: erik.pekkeriet@wur.nl
Contact:
Erik Pekkeriet (project coordinator)
Email: erik.pekkeriet@wur.nl
Tel: +31317483372
Address: Droevendaalsesteeg 1, 6708 PD Wageningen
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
Functionality that was achieved:
- Flexible tray sizes by a digital mould system, including a fast changeover system
- Bin picking robot module with cameras in the working range of the robot to assist the motion and failures
- Adaptive grippers to grab a wide range and fragile food products
- QAS module with (3D colour cameras, hyperspectral image analysis, 3D x-ray, microwave scanning)
- Flexible sealing and cutting with a laser
- Flexible heating system for convenient baking and cooking
- Flexible decoration by direct printing
- Integrity check camera system of the seal
- Cable robot to pick and place packages in a flexible working range
- Full track and trace system including RFID antennas and flexible software to keep track when products and line configurations are changing
- Software for flexible systems integration
- Cleaning robot and systems for cleaning the line from the inside (only food contact parts)
Functionality was performed on the focus products: vine tomatoes, grapes and ready meals/chicken breasts. Also other products are tested by specific modules.
The demonstration line operated at a cycle time of 17 packages per minute. The goal was to achieve 30 packages per minute, which is the industrial standard. The bottleneck was now in the robot handling and laser sealing. By increasing the capacity (motion planning) of the robot and increase the power of the laser, 30 packages per minute should be possible.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations. A life cycle assessment was carried out to assess the impact of the developed technology towards the industry and society. This all looks very positive for the PicknPack systems.
The line is demonstrated in a video and two live demonstration sessions at Wageningen (NL) and Holbeach (UK). Exploitation plans were made for all exploitable results.
Project Context and Objectives:
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
To meet these current and future challenges of the European food industry the PicknPack consortium will develop three types of modules that can package a wide variety of food products in packages that easily and quickly can be adapted to specific markets.
These modules work closely together:
1. a sensing module that assesses quality and other critical parameters of the individual or small batch products before and/or after packaging;
2. a vision controlled robotic handling module that picks up and separates the product from a harvest bin or conveyor belt and places it in the right position and/or correctly in a package; and
3. an adaptive packaging module that can accommodate various types of packaging with flexibility in terms of package shape, size, product environment, sealing and printing.
In the first year all specifications and concept designs of modules (functionality) were discussed and fixed. These specifications resulted in detailed designs of the line and modules. In the second year partners started to build their modules and line integration functionality. In the third year this work continued and modules were tested first in standalone operation and later in line operation. In the fourth year modules were placed in the line and line tests were done to achieve a high level, flexible and autonomous food packaging system. This line was demonstrated at the end of the project to the food equipment and food industrial sector. During this project a high level of progress was made. In parallel research was done on hygienic issues, societal impact, life cycle assessment and exploitation opportunities.
Functionality that was achieved during this project:
- Flexible tray sizes by a digital mould system, including a fast changeover system
- Bin picking robot module with cameras in the working range of the robot to assist the motion and failures
- Adaptive grippers to grab a wide range and fragile food products
- QAS module with (3D colour cameras, hyperspectral image analysis, 3D x-ray, microwave scanning)
- Flexible sealing and cutting with a laser
- Flexible heating system for convenient baking and cooking
- Flexible decoration by direct printing
- Integrity check camera system of the seal
- Cable robot to pick and place packages in a flexible working range
- Full track and trace system including RFID antennas and flexible software to keep track when products and line configurations are changing
- Software for flexible systems integration
- Cleaning robot and systems for cleaning the line from the inside (only food contact parts)
Functionality was performed on the focus products: vine tomatoes, grapes and ready meals/chicken breasts. Also other products are tested by specific modules.
The demonstration line operated at a cycle time of 17 packages per minute. The goal was to achieve 30 packages per minute, which is the industrial standard. The bottleneck was now in the robot handling and laser sealing. By increasing the capacity (motion planning) of the robot and increase the power of the laser, 30 packages per minute should be possible.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations. A life cycle assessment was carried out to assess the impact of the developed technology towards the industry and society. This all looks very positive for the PicknPack systems.
The line is demonstrated in a video and two live demonstration sessions at Wageningen (NL) and Holbeach (UK). Exploitation plans were made for all exploitable results.
Project Results:
1. Introduction
The PicknPack project developed three types of modules that can cope with the typical variability of food products and the requirements of the food sector regarding hygiene, economics and adaptability. It assessed the quality and shape of individual products, handle the products in a flexible way and adaptively pack them in an optimized packaging to add maximum value to the quality of the product and provide convenience to the consumer. Information obtained in the process is transmitted and could be used to optimize performance upstream and downstream in the chain in logistics ensuring the highest quality for the consumer, minimum waste and full traceability. To ensure utilization of the results of PicknPack, special attention was given to overcoming barriers and ensuring adoption of the system by the food industry.
The modules are connected to an adaptive multipoint framework for flexible integration into a production line that optimally makes use of the capabilities of the individual modules. The communication between modules is based on a shared, vendor-independent vocabulary. The system will be designed in such a way that a wide range of fresh and processed food products and packaging concepts can be handled. It will also be able to single out an individual product from a group (bin picking) and correctly orient it for packaging. Tools for fast change-overs and adaption to new products will be implemented to reduce the time required to program the system for new product/packaging combinations.
2. A flexible food production line
A work package to setup the line was committed to the following main objectives:
- To develop and test a fresh food production line, focused on quality assessment, separation tasks, handling and packaging of vine fruits and vegetables (case focus: vine tomatoes and grapes).
- To develop and test a processed food production line focused on quality assessment of a variety of processed food components, arranging these components in a package and closing the package by sealing (case focus ready meals)
- To develop and evaluate generic concepts and control within the production line that can perform on other products within fresh and processed food applications.
In the first half of the project, this work package focused on drafting the requirements of the first two objectives. This was defined in the first task where the definition of the lines would take shape in an iterative manner in cooperation with all the partners involved.
This task included a system analysis and bottleneck assessment. However, early on it became apparent that most requirements were hard to set in stone. Instead we focused on a range of values for each requirement and fall-back scenarios in case some could not be met. The design included requirements for hygiene standards in the food industry and should adhere to the economic viability.
To obtain the desired minimum level of performance of these requirements, multiple site visits were performed with a part of the consortium. For the fresh food lines, a tomato packaging facility was visited. To get an idea about the processed food production lines, members of the projectteam went to a ready meal factory,
Team members responsible for the software integration, adjustments to the design requirements were made for allow a flexible systems integration. The hierarchical concept of modules was introduced and templates were written for each partner to implement the communication protocols. Furthermore, overviews were made of each module’s input and output and integrated in the grant design of the lines.
It became apparent after a few iterations of requirements that modules for both lines could be integrated in a single flexible line. The design was then fixated on this single integrated food processing line for both fresh and processed food products.
Furthermore, a very important design principle was fixated. Instead of having conveyor belts for each module and attach them into a single line, an integrated backbone of sectional frames was chosen. The main benefits of this choice is that the web of trays does not have to be separated in the beginning and that at each point in time it is known where each product is located. This information is shared knowledge through the software. The modules could then be placed over the sectional frames and if necessary moved in a different order to accommodate different food processing functions. However, the downside of such a setup was later realized that the development was a major dependency for each module, slowing the overall process significantly.
After the requirements for the line were fixated as much as possible, each individual module could start fixating their module, moving from proven prototype concepts. WP7 supported in this stage the test trials of critical functions and working principles. Also here an iterative process took place; after modules became more finalized, more and more requirement parameters could be updated. Dependencies between modules crystallize. Teammembers of Work Package 7 were responsible for the integrated overview of the module and their parameters.
Part of creating this overview included quantitative simulations and mathematical models to calculate the effect of all parameters on the throughput of the product. The main performance measure of the line in both food products was set on the amount of packages produced per minute. Because the line is a serial process, the slowest modules would dictate the maximum speed.
In the second part of the project, modules started to emerge from assembly and came out of prototype status. To support integration of all the modules, a large food grade hall was found and leased at Wageningen University and Research to enable close support. The hall provided power, water and for cleanable a drainage system.
It became apparent that not all module owners could deliver the hardware in time and although some modules were delivered early, not all functionality was present upon delivery. WP7 supported the integration efforts by extending help also in finalizing the modules and connecting them to the sectional frames, which could not be performed before arrival to the food hall.
The thermoformer was the first module to arrive, after which it was connected to the sectional frames for package transport. Finding the right foil composition and thickness proved a challenge, putting a potential delay on the project as other modules required trays as input to perform integration tests. Preformed packages were made as a contingency plan.
Each partner visited the facility often to do their part for the integration efforts, usually for a week period at a time. A team of hard- and software engineers from Wageningen DLO supported their efforts, although often tasks were taken over to speed up the integration.
The final deadline was set at May 1st, 2016. At that point the line should be fully functional and working. However, this was not met for all functionalities, e.g. printer line controller software and automatic mould exchange of the thermoformer. At the deadline the functionality was frozen and all efforts were focused to create demonstrations for the demo day. Still, the main functionality was able to be shown and physical integration was completed, except for the cable pick and place robot which was demonstrated separately. The line was fully operational during demonstration time.
Improvements were continuously suggested by WP7 and implemented by the partners during the third period. A taskforce for the software was formed in February 2016 to ensure all modules were connected through software.
The cable pick and place robot at the end of the line was demonstrated separately. It functioned stand-alone and was connected to the line controller. This is a deviation from what was promised, mainly due to the partner’s struggle with hardware sub functions in this period.
The printer module was not integrated with the line underneath that provided the packages. The top foil was moved over the web of packages and rolled up thereafter, but did not connect to the packages with the planed motor system in the sectional frames. This was due to the limit in software engineering capability under supervision of DTI and could not be solved by other partners.
The laser was demonstrated separately in another room. The laser was not able to weld and cut the required PET foils in a robust and fast way. PET foils were too transparent for this laser and other foils as PP were not possible to mould in the thermeformer.
The line was integrated with the automated cleaning system. However, not all modules supported this. The second half of the line (X-ray, Printer and Cable Picking Robot) was not automatically cleanable. The first part was successfully implemented and tested on site several times without damage to the modules. This was tested after the first demonstration at Wageningen, due to the risk of possible breakdown involved. A plastic cover was implemented over the sectional frames to prevent water distribution to other places.
After all modules are integrated, the total line performance was measured. The values were inherently the minimum of all modules’ performances, because the sequential nature of the line; hence the bottleneck module will determine the values of the overall performance. The following indicators for the complete line were previously distinguished:
• Packages created per minute.
• Number of fresh and processed food types.
The number of fresh and processed food types the line could handle was 2: tomato and chicken breasts. The second indicator depends on how many trays per cycle the thermoformer creates. For tomato, this was 3 trays per 11 seconds, or 16 packages per minute. For the chicken breasts this was 6 trays per 21 seconds, or 17 packages per minute. The objective for the demonstrator was set at 30 packs per minute, which is considered state-of-the-art. The bottleneck for both lines was the pick and place robot for the food products. Other modules were able to speed up to the set goal of 30 packages per minute. For each individual module, it’s current performance is reported in deliverble 7.5 Report of prototype test under laboratory (and industrial) conditions, as well as suggestions for future improvements.
After the line was integrated, it was successfully demonstrated to the public, picking and placing tomatoes and switching to chicken breasts after a 5 minute change-over.
3. S&T-results on flexible systems integration
3.1 Pick-n-Pack data bus for improved integration and data exchange
The Pick-n-Pack data bus allowed all partners to interconnect to each other, share data in a generic decoupled way and interface to legacy software and hardware in a flexible and easy manner. The step change here with respect to traditional food packaging lines was the addition of auto-discovery and auto-configuration. This was achieved by exploiting state of the art solutions from the IT and networking community. The Pick-n-Pack data bus defined the communication backbone and improved flexibility and dynamism in the Pick-n-Pack line by allowing software entities to dynamically join and leave while all connections to other entities are configured or reconfigured automatically. Two communication primitives were implememted: whispering and shouting. The former was used in peer to peer messaging, while the latter was designed for group messaging.
3.2 JSON message models and 5P production model for improved traceability
The Pick-n-Pack messages were represented in JSON and this format was chosen, since it has a lot of traction in the world wide web community and allowed us to efficiently integrate and interpret the messages in web applications. The model includes a header and payload to first identify the type of message, its sender and receiver. Additionally, a URI of the data model and meta model can be supplied in the message to make it self-descriptive. The step change here was also the introduction of Universally Unique Identifiers or UUIDs at all levels in the Pick-n-Pack line, resulting in a much smaller footprint, in the form of meta-data, in the tracing database and keeping all other raw data locally at the Module that created this data. Additionally, the 5P generic production model described what (meta) data we need to store to effectively trace back to a particular production, which is a composition of a product handled by processors that executed a set of processes with a set of policies.
3.3 Queries and dialogs for improved robustness in data exchange
The rationale behind stateful queries is to move away from stateless data exchange which does not contain context and to introduce dialogs. The bidirectionality of a dialog makes a trade-off between scalability and reliability in a sense that you need to be aware of which peer you are sending messages to or receiving messages from but it allows handshakes to ensure important data is sent and received. In Pick-n-Pack queries were used to communicate with the ontology that was defined in the project around tomato trusses and to query the semantic database that was used to store data from the Pick-n-Pack data bus.
3.4 Life-cycle state machine for improved flexibility in deployment & reconfiguration
All Pick-n-Pack Modules and their devices have to work together to accomplish the food packaging task. The Line Controller was introduced to manage this process. To increase flexibility here with respect to identifying problems in one or more modules and knowing when the line can proceed, the Life-Cycle State Machine was designed. This Life-Cycle State Machine defines a set of states each Pick-n-Pack Module has to go through before it can safely join the food packaging task. This consists of allocating the right resources, setting up communication, configuring the correct software for the task at hand and running its particular task. It provides a homogeneous startup procedure for the wholePick-n-Pack line at different levels of abstraction. The Line Controller exploits this information to confirm if it is safe to continue operation or not. This behaviour was captured in the Stop light protocol. Modules communicate their state as ready (green), busy (orange) or not ready (red) and the Line Controller only sends a go signal to all Modules in the Pick-n-Pack line if it received a green light from all Modules. Additionally, adoption of the Life Cycle State Machine facilitates reconfiguration, both in software and hardware. This can now be done by emiting a reconfiguration signal, either by the Line Controller or directly by an operator from within the Flexible GUI. This signal is transmitted onto the Pick-n-Pack data bus and each Module change their state accordingly. For some Modules this can be done purely in software, while others require manual intervention. The same stop light protocol and Life Cycle State Machine come into play here.
3.5 Flexible GUI for improved visualisation and operator feedback
The architecture of the Flexib GUI is distributed, running partly on a Web-enabled server and partly in the client browser. The Flexible GUI receives data from the Pick-n-Pack databus through the Mediator pattern, which listens to the databus, converts this data to useable data for the web visualisation and adds this data to the semantic realtime database. The user interface itself is built in a modular way, meaning that it will visualise active Modules on the Pick-n-Pack databus and dynamically update events it receives from them through the Mediator. In addition, a Query interface was made availabe, to query the semantic database and visualise, for instance, statistics on the quality assessment or generate labels for the Printer Module. The design of the web application was further modularised by adding a hierarchy and creating a generic visualisation framework which can be reused independent of Pick-n-Pack.
3.6 Mediator design pattern for improved flexibility in software
The Mediator pattern was developed to decouple software entities which share data but not necessarily need to be aware of each others knowledge domain. The Mediator has knowledge about both the agreed data format on the Pick-n-Pack databus and data format used by the visualisation framework.Another way of mediation was done at the Module level, where internal specific knowledge of the Module developer needed to be converted to comply with the agreed Pick-n-Pack language. The same design pattern are applicable at a higher level of abstraction such as a Plant with several Lines, or a finer grained level such as individual Modules, Devices or even Algorithms. This is not only what makes these patterns flexible as required by the Pick-n-Pack objectives, but also generic such that they can be used outside the context of this project.
4. S&T-results on information and traceability
The RFID traceability system integrates the RFID technology for production process monitoring and data integration of the automated PicknPack line. The application of RFID technology and Information and communication technologies allows the comprehensive monitoring and data collection of products for data tracing or other further uses such as process optimisation and vendor managed inventory.
The design and development of RFID traceability system as a module of PicknPack line follows the new concept of digital manufacturing, which features smarter machines, machine collaboration, and data based system optimization. The results of the RFID traceability system are summarized as follows:
4.1 RFID enabled traceability system hardware integration
The system integrates the state of art RFID systems in the production process for product information tracking and tracing. The RFID reader/antennas with Internet connection can talk to the traceability software to notify the sensed RFID tags. The operation can be based on wired Internet or Wi-Fi. The system contains the following devices:
• An RFID reader with four antennas
• A computer with traceability software
• A handheld RFID reader for RFID and QR code reading
• A RFID applicator to place the RFID tags on product units
• A video camera for QR code reading
• A router to build the connection of the devices
One RFID antenna is integrated in the ‘pickrobot’ to monitor the input crate with raw materials. One is installed between ‘laser cutter’ and ‘packrobot’ to monitor the individual product units. Another antenna is installed at the end of the line to monitor the output crates. In addition, one extra antenna is used for information tracing purpose.
Since each product unit in the production process has a Universal Unique Identity (UUID), the data generated by each module in the line can be integrated with this UUID. The UUID can then linked to the current valid RFID labelled input crate and output crate. The UUID can also be linked to a RFID when an RFID is placed. Therefore, with a particular product unit, its associated input crate (source information), output crate (outgoing information), RFID number, and printed QR code are all linked to its UUID (online generated product information), and product information can be traced back from database with an RFID or QR code reader.
The integration of RFID recognition, data communication, and database technologies has successfully led to the real-time online monitoring of the PicknPack line production process.
4.2 RFID traceability system software
Traceability software is developed to deal with user interaction, hardware events, communication events, and database handling, etc. The development of this software focuses on the efficiency, reliability, and Simplicity in operation. Since the software handles the line communication, RFID reader, video camera, and database access, its efficiency and reliability are critical parameters to evaluate the system. The multi-thread technology of visual C# platform is employed to deal with the multi-task events handling.
Software GUI is designed to simplify the operations and enhance the efficiency. The user can navigate to the functions with top level navigation icons, and the selected functions are displayed in the centre workspace of the interface. The current running production job information is displayed on the right column of the interface. Then, communication messages, hardware events, and recognized RFID tag information is displayed at the bottom of the interface.
4.3 Custom-designed database for PicknPack automated food manufacturing system
A flexible database model is developed for the traceability system of PicknPack automated production line. The database model integrates source material information and outgoing information with online generated information. By using UUID as identification of online product unit and RFID as a machine readable ID for offline use, the relationship is built for data integration and information tracing.
Considering the specificity of PackNPack line, the flexibility in food categories and production strategies may result in different product process and different parameters. To deal with this problem, the data modelling integrates relational database with JSON descriptions. The SQL server tables are simplified by using text format to store data of some dynamic structures in JSON string.
With customized design, the database model therefore satisfies the requirements of PicknPack line in terms of flexibility.
4.4. Flexible inter-module communication
Inter-module communication is a critical enabling technology for the automated digital manufacturing systems like PicknPack line. The traceability system communicates to the modules in the line with a communication protocol based on Zyre. It can initiate a Peer-to-Peer (P2P) communication and one-to-multiple broadcast. Therefore, it can communicate to each module in the line for data communication and event notification.
There are two important communication tasks for the traceability system: (1) talk to ‘pickrobot’ to notify the current valid input crate, and (2) listen to ‘linecontroller/worldmodel’ to obtain the online generated data. With this flexible communication protocol, the traceability can collaborate with peer modules in the line for data integration.
4.5 Multiple product information tracing methods
The product information tracing is an essential task for the RFID traceability system. This system provides four approaches to trace information of a product: (1) Fixed RFID reader/antenna; (2) Video camera for QR coder reading; (3) Handheld device RFID reader; (4) Handheld device QR coder reader.
The system supports information tracing of products during production process and post-production. The handheld RFID/QR code reader with WIFI connection to the database can be used for information tracing anywhere, and information of the product item being traced is displayed on the GUI of the device interface.
The capability of the traceability system has been demonstrated in Wageningen and Holbeach in May and September 2016 respectively. The developed traceability system could be extended and customised for applications in practical food manufacturing environments and other industrial sectors.
5. S&T-results on Quality Assessment and Sensing
The main objective of the design and development of a Quality Assessment and Sensing (QAS) module which provides information on various quality aspects of all the individual products on the line. The collected information can then be used by the rest of the line for product sorting, robotic handling, traceability or adaptive packaging.
5.1 Product ontology
In the first stage of the project, a food (quality) ontology was created for the food products to be processed on the PicknPack food line, including vine tomatoes, table grapes and ready meals. This was done based on a listing of key quality parameters for each of the products as well as the acceptable range for these parameters.
The most important quality parameters which were identified, are:
- For fresh fruits: maturation stage, external defects, internal damages, colour, size and shape
- Ready meals: colour, topping, weight, size and composition of the meal
The most important marking and packaging parameters which were identified, are:
- For fresh fruits: shelflife, weight, packaging and traceability
- Ready meals: traceability
The product description provided by the ontology was then used as a basis for the design and development of the sensor technology and capabilities required in the QAS module. For more details the reader is referred to deliverable reports D4.1 and D4.2.
5.2 Design and functionality of the QAS module
The QAS module is designed as a system comprised of multipurpose sensors that each can assess individual products in a unique way. The sensors are divided over two submodules which were designed to fit on the sectional frame transport system which was designed in the PicknPack project. The first submodule was constructed in a collaboration between KUL, WUR and MU and contains an RGB camera, a 3D laser scanner, a microwave line scanner and a hyperspectral camera system. The second submodule was designed and constructed by KUL and InnoS and contains an X-ray unit. A more detailed overview along with further references can be found in deliverable report D4.10.
5.3 Module design and topology
Following the design guidelines and topology of the PicknPack line concept, the QAS module consists of a central module controller and a number of sensor units, called “devices”. Each device consists of a sensor, appropriate illumination and a matching data acquisition and processing unit. The module controller interacts with the devices as well as with the PicknPack line. All communication between the various actors in the QAS module is established via the PicknPack communication protocol which was developed in WP2 & WP7 based on the zyre protocol, even allowing different devices to use different programming languages.
All sensors in the QAS module are connected to a hardware controller that translates the real-time speed of the line as captured by encoders into a trigger signal that drives the data acquisition. This made it possible to collect data at sub-mm resolution at the speed of the line, during all stages of the stop and creep regime.
The overall design of the module makes it possible to add or remove sensors and functionality in the module in a flexible and efficient way, as desired. Furthermore, hygienic requirements were taken into account and the submodules were equipped with a tunnel system, making them fully cleanable.
5.4 Sensor technology
Based on the ontology and key parameters of the products, each sensor system was first developed in stand-alone units to realize the basic functionality and start the development of data processing algorithms. Hereafter, the sensors were integrated into the submodules. The following paragraphs give an overview of the different sensors.
5.4.1 HSI camera
The hyperspectral camera is sensitive in the visible and near infrared region of the electromagnetic spectrum (600 – 1000 nm) and achieves a spatial resolution of 0.4 mm/pixel. This system can accurately measure the unique spectral fingerprint of products. To optimize the illumination, ray tracing software was applied and in the later stages of the project, a cross-polarization system was developed to avoid that specular reflections would disturb the recorded images, thus increasing the quality of the data. The HSI system was used for the following applications and products:
• Truss tomatoes: sweetness (brix), ripeness and presence of skin damage on individual tomatoes in a truss
• Table grapes: sweetness (brix)
• Poultry: discrimination between cooked and uncooked chicken and turkey breasts.
5.4.2 RGB camera
The RGB camera is a line scanning unit that achieves a high spatial resolution of 0.25 mm/pixel. It was used to assess the ripeness of individual tomatoes based on their colour. Furthermore, data recorded with this sensor was merged with that of the 3D sensor to produce accurate 3D colour models of the individual products.
5.4.3 3D sensor
The 3D sensor applies laser triangulation to create 3D models with a resolution of 1 mm/pixel. These models are then used for the contactless estimation of the size, shape and weight of products. This was demonstrated for tomatoes in a truss.
5.4.4 Microwave sensor
The microwave sensor comprises of a series of 22 transmitter-receiver antenna pairs that can achieve a resolution of 2.1 mm in the direction of motion of the line. The sensor was used to determine the composition of a ready meal and to determine the degree of cooking (rawness) of chicken breasts.
5.4.5 X-ray sensor
The X-ray unit applies a novel technique to produce 2.5D images of the internal structures of products with a resolution of 0.135 mm/pixel. The setup consist of two X-ray source-detector pairs with crossing beams. This arrangement allows to determine the 3D location of internal features. The setup was used for foreign body detection and assessment of the internal structure of food products for defects.
5.5 Advanced data processing techniques
Besides the algorithms that were used to process the data of individual sensors, other flexible software tools were developed:
- Sensor fusion: Algorithms were developed that combine the information from multiple sensors to calculate new product features. This was demonstrated in two different way: (1) by mapping the data of the RGB and 3D sensor to create full colour 3D models, (2) by clustering quality features measured by the individual sensors to automatically determine the (user-specified) quality class of each product.
- Semi-supervised segmentation: An algorithm was created that helps a non-expert user to quickly train the QAS system for new types of products in an intuitive way .The algorithm was demonstrated for segmentation of hyperspectral images.
- Automated inspection of ready meal components: An algorithm was developed that can assess the composition of ready meals based on a database of pure component information.
- Deep learning: This technique applies artificial neural networks to perform complex tasks such as identifying cuts and puncture damage in tomatoes or worm damage and bruising in apples. The deep learning algorithms can be trained by non-experts.
5.6 Demonstration of QAS
The QAS module was demonstrated successfully during the two major demonstration events of the project. In May 2016, the full QAS module was demonstrated as part of the entire PicknPack line in Wageningen (NL). The module inspected tomatoes for ripeness, shape, weight, sweetness, the presence of internal damage or foreign bodies. Besides this, chicken breasts were scanned for moisture content, shape and degree of cooking (rawness).
For the demonstration in Holbeach (UK) in September 2016, a smaller stand-alone version of the QAS module was constructed that assessed tomatoes for sweetness, ripeness, weight and the presence of surface and subsurface defects. This module ran on the original software of the main QAS module, together with a simulated version of the line, demonstrating the flexibility of the concepts developed in the PicknPack project.
6. S&T results in robotic handling
Two (three) robotic modules were researched end developed in this project. The former called cable-robot (Pickable) and the latter called delta-robot. Thus this chapter will be also divided in two sections, one for each robot. The (third) robot that can clean the line form inside out is described in the S&T results for hygienic design of the line.
6.1 S&T Results for the cable-robot
As a summary of this chapter, it could be said that a complete innovative robot concept has been built and tested. It goes one step beyond the state of the art as certifies its patent (PCT/EP2014/078932) and no previous robots with all the characteristics of this one were previously developed.
This new robot is based on the same principles as industrial parallel robots to manipulate and place objects at high speed, but the rigid bars are replaced by metallic cables. The cables are connected to a mobile platform at one side and enrolled on winches that control the length and tension of the cables for the controlled movement of the platform on the other. Thus, a rectangular workspace is obtained suitable to grasp foodstuff from conveyors.
The novel robot can work in high speed pick and place applications keeping the same dynamics and cost of commercial parallel robots and making work volume more efficient, allowing re-configurability and improving the required footprint.
Parallel kinematics manipulators or delta robots, used nowadays in food industry, are not reconfigurable in workspace, have a no optimized footprint, which is a limitation in food industry to transfer the products from one conveyor to another one, and have spherical or cylindrical inefficient workspaces in which part of the robot reach is not utilized.
Specific added value:
• Reconfigurability of the robot: Only modifying the cable length and the upper plate, the cable robot can adapt its shape (x-y). This can provide to robot manufacturers a wide range of robots regarding to standard dimensions of conveyors and so on. It is possible to manufacture new robots with the specific necessary workspace, something not possible until now with commercial robots. That is an important advantage for robot manufacturers and for food industry too, because the robots will be easier to manufacture and therefore cheaper.
• Work space-footprint ratio reduced: The difference between the total dimensions (x-y) of the cable robot including the frame and its workspace is lower than commercial parallel robots. It provides a better exploitation of the available space in the plant (it needs only 200 mm more than the work space). This is an important advantage for the industry because the space is optimized. If other conventional ceiling robots are analyzed and compared with Pickable, taking into account that these conventional robot usually need a frame so as to stand their structure, the food print ratio is a valuable more competitive characteristic of this novel cable robot.
• Integrated with the PnP line. It is able to receive and send communications signals/events and data, and thanks to that, it is able to accomplish these expected functionalities:
• Direct and “clean” connection to the PnP line, without changes needed, as long as the subnet accomplishes the Zyre based network requirements: UDP and broadcast allowed, and port 5760 opened.
• Flexible line configuration changes implemented by Cable Robot in realtime thanks to world data model reception, parse and process. This allows real time changes related to batch, tray and punnet types and their dimensions.
• Cable Robot data shipment to the Line in order to be shown in general line GUI. This data includes for example information about final crates used
• More efficient workspace: The rectangular shape of the workspace takes advantage of a more efficient way of the reach of the robot. The delta robots have cylindrical working ranges.
This shape is not the best one so as to grasp stuffs from conveyor belts as is the most common situation for the food industry. However Pickable has a working range with a rectangular shape. It is the most suitable configuration for Pick and Place applications when moving food stuff from input to output conveyors. Taking into account that the working reach of the robot fits in such a suitable way to the target area, the selections of the robot is optimal. Moreover bigger than desired robots must not be selected anymore.
• Optimized conveyor tracking: Several additional elements were added to Pickable so as to reach a more feasible and robust system. Those elements were a specific conveyor and a machine vision system. These peripherals can improve the behaviour of Pickable since they allowed to get and individual operative system. The machine vision system on the conveyor helps Pickable to identify the position and orientation of the input crates. This information can be used, once Pick-cable will be a fully calibrated system, to reach any input stuff in an accurate way.
• Innovative torque control strategy for the cables: This is probably the main scientific output of Pickable. Its control modelling and design is quite challenging. For the control of Pickable, an innovative control strategy developed by Tecnalia has been developed and takes into account the necessity of always keeping in tension the cables. The use of the standard control of the drives is not possible. That is why a crucial step in this case is to totally open the controllers that can directly send torque control references. Once this parametrization of torque control achieved in the engine drives, the first step is to control the robot in position and tune the gains in order to have a good performance in terms of settling time, overshoot and static error. The real time environment used for the control of Pickable is RTX. Its main advantage is its ability to transform a hard real-time system on a standard PC Hardware, transforming Windows into a Real-Time Operating System (RTOS).
6.2 S&T Results for the delta-robot
In the third period of the project the pick-and-place robot module was fully built up and integrated in the PicknPack line. The complete system consists of the following components in the order as they appear during processing:
• Reading of the RFID tags on the harvest crates for tracking and tracing
• Automatic transportation of crates with foodstuff into the robot cell
• Detection of the crate and of the products in the crates by the 3D+colour vision system
• Planning of grasping actions to get the products out of the crates and into the trays
• Weighing of the grasped product
• Transportation of empty crates out of the robot cell
• Flexible software integration and synchronization with the rest of the line by means of p2p internet protocols as developed in WP2
• Integration with the cleaning robot for automatic cleaning of the robot cell
In the following, a short description of the achievements of the three main objectives is given, followed by a description of the performance in the line:
Development of a food product handling module that is flexible and fulfils the criteria of the food industry regarding hygiene, economy and safety
The hardware of the delta robot is largely based on a system that Marel sells to the industry and which is completely up to the standards of the food industry regarding hygiene, safety and economy. Added components meet the same standards. The grippers developed by Lacquey have been thoroughly examined on hygiene by Fraunhofer. Improvements have been made to full fill the necessary standards, which are high, as the grippers are in contact with the food. The vision system consists of off-the-shelve components and are housed in an enclosed vision box for hygienic reasons and for cleanability. All components meet standards of safety and economy.
Development of end-effectors that allow handling of a large variety of (non-uniform, delicate) food products
Two end effectors have been developed by Lacquey, one for grasping vine tomato and grapes and one for grasping chicken filet. In the third period, the fresh-produce gripper has been further developed with different fingertips that can easily be placed on and off the gripper to deal with variation in produce shape between different products and cultivars. Results of grasping are detailed below.
Development of a reprogramming method that allows fast change-overs to other products by non-specialized workers
Physically, quick change over is facilitated by a quick gripper-release system, allowing switches under 1 minute. The software is able to change to other products on the fly. To allow flexible switching to other products, the vision and control software uses parameter files which describe parameters of the products, the gripper and the grasping process. By putting the software in a (re-)configuration state, the required new information can be loaded and the system can be brought into running state again once the line is ready.
Performance in the line – Quantitative analysis
A quantitative and a qualitative analysis of the performance of the module in the line has been reported in deliverable D7.5. In this section, we give a conclusion of these analyses. For a more detailed description, we refer to D7.5.
Software communication: Internal communication between vision system and robot works reliably, as well as the communication of the module with the line. Flexibility with respect to the layout of the web of trays should be improved and a check if the right gripper is mounted should be included.
Grasping: The vine tomatoes and the chicken filets are grasped and removed from the crates successfully and reliably given sufficient space around the products. The dragging action to move products to empty space when they are touching does not work as anticipated, as too many products are damaged in the process. Grasping table grapes showed to be too difficult because of the deform-ability of the product.
Vision: The vision methods to determine grasping actions for vine tomatoes of different colours both non-touching and touching works well. Also for non-touching chicken filet and grapes, the methods are robust. For touching filets, the methods need further improvement to correctly segment all pieces. Entangled grapes are too difficult to ever be able to correctly detect with vision. Detection of the position of the crate works robustly.
Speed: Total average time for filling a tray with vine-tomato, including transport is 4 seconds. For chicken filets, this is 3.5 seconds. This is not yet up to desired 30 picks per second. Most of the time is spend on the mechanical actions of grasping (5%), transportation (70%) and releasing (5%). The robot can move much faster. However, no path optimisation has been performed to test what the effects on higher accelerations are on the product and the stability of the product in the gripper.
7. Adaptive packaging system
For the adaptive packaging system a range of functions was researched and developed. The main functions were:
• Flexible tray sizes by a digital mould system
• Flexible sealing and cutting with a laser
• Flexible heating system for convenient baking and cooking
• Flexible decoration by direct printing
• Integrity check
7.1 Digital Mould
PicknPack has developed several flexible and digital mould systems. A pin mould system with many small pins that together create a mould can change shape all the time and make each pack different. Moulds can be produced by rapid prototyping can create moulds very fast. PicknPack ended up using a brick mould system because this technology fit best to the food and packaging industry. The advantage with the brick mould system is that worker unskilled in computing can create and produce new moulds in seconds just as assembling LEGO. These moulds can automatic be digitalised and digitalised data can be used all over the flexible packaging line. Only 3 types of bricks are needed to each packaging height plus an extra for unused area. In case of four different heights all needed packaging sizes can be created using only 13 different kind of bricks. As in LEGO many special bricks can be added to the system in order to create logos, round or oval shaped packaging.
The thermoformer was also equipped with an automatic exchange system. The moulds are placed on a placed on a plate with the same foot print as the forming chamber of the thermoformer. Several of these mould plates are stored in the automatic exchange system. After designing and producing the moulds on the plates these are slided into the thermoformer. An automated system moves the mould plates into form chamber in seconds. This movement shall best be performed as the film is moved from one position to another. This can make each packaging different to any needs.
In the demonstrations both the brick mould and exchange systems documented. The other two moulding system was demonstrated off-line.
7.2 Flexible sealing and cutting
PicknPack has selected laser cutting and sealing in order to seal the top and bottom plastic films together to one packaging. The same lased was used to each individual packaging out from the long web with filled trays. This system work flexible as the laser beam is controlled by a mirror and a scanner to focus the beam over the packaging films. PicknPack was able to control the process. But PicknPack had problems to integrate the laser system to the total line. The problem was that type of laser need to be adapted to the type of polymer used in the packaging. PicknPack purchased early in the project a laser for PP plastic because this was a normal material when PicknPack started. Over the last few years a price drop in PET has out competed PP totally from the market. It was impossible to find PP to use for the demonstration.
The advantage using laser is that the laser can seal and cut any shape created in the design process. The laser can also create small perforations in the packaging film to be used for eMAP in packaging of fruits and vegetables.
7.3 Flexible heating
PicknPack also developed a microwave active printing layer to be printed on the top film of the packaging. The print was done with current ink printed in different patterns on the top film. The current layer works as shielding in the microwave own. Printed in specific patterns the energy can be directed into the food components needed to be heated and also shield other components from the energy. In this way it is possible to control the heating process of ready meals with different food components. The technology can also remove all problems about hot and cold spots in microwave heating.
PicknPack also developed a technology to print susceptors using the same print. PicknPack made many small dots of the same shielding ink. The result was a susceptor absorbing the energy and heating up the packaging. This technology can only be used for packaging materials able to operate under high temperatures as CPET or paper.
7.4 Flexible decoration
PicknPack designed a flexible decoration system based on ink-jet technology. Ink was sprayed on the packaging in 360 dpi resolution. Each colour need a line of print heads. The flexible printer receives files from the central control system and print the package just before sealing the pack. The food and packaging industry has been very interested in this flexible printing technology because shifting decorations creates many shifts on the packaging lines. Also the delivery of printed films to the industry is a practical daily challenge as the lead time will be increased with 1-4 months. PicknPack demonstrated the flexible printing technology in two colours. The system has room for full five to six colours.
7.5Integrity checking
PicknPack has developed a flexible integrity system using a multispectral imaging system that is easy to adapt. It covers the Visible (VIS) as well as a part of the Near Infra Red (NIR) part of the electromagnetic spectrum (400 – 1000 nm). The camera with sensitivity in the above-mentioned range was mounted over the area for welding and cutting the packaging out of the web. The camera lens looks through a tailor-made filter wheel that houses a total of 6 optical filters. As such, each package is imaged with great spatial resolution at 6 different wavelengths, and a multivariate image analysis (MIA) algorithm is used to find sealing issues. The system can audit all packaging to secure full integrity of each pack. If a pack has problems the integrity system can send a message to the welding laser to upgrade the welding in specific areas.
7.6 Other functions
The PicknPack system will be covered with shields in order to prevent spraying from the hygienic robot. The area of the laser and quality assessment module also need to be protected as the laser, X-ray and microwave all involves risks. If the whole system is covered it is possible to flush this covered volume with a packaging gas. This demand glove boxes on the sides in order to intervene failures. The system can create MAP (=Modified Atmosphere Packaging) and has the extra advantage that the products will be merged together in the special atmosphere which prevent “pockets” of oxygen inside the food. As the PicknPack line never was assembled a complete line this system was only illustrated.
8. S&T results hygienic food handling
During the project the focus of work package on hygienic food handling was to implement the Cleaning System into the PicknPack line and determining its performance parameters. Therefore the concept of the Mobile Cleaning Device which was developed in the first part of the project was put into operation after serval amendments and optimization steps. After this it was integrated into the line for demonstration and experimental purposes. The concept was validated so that in the end a functioning cleaning system was available which can drive automatically through the whole line in order to clean all modules consecutively with individual cleaning programs.
In addition, for the Delta Robot module a conventional CIP-system was integrated in order to compare its performance parameters with the ones of the novel Mobile Cleaning Device. To assess those performance parameters cleaning tests with regard to cleaning efficiency were performed exemplary on the Delta Robot. Therefore, parts of the robot were reproducibly coated with a fluorescent food model soil. Then those areas were cleaned with the different cleaning systems and different cleaning parameters. The cleaning progress was monitored with a camera system in combination with a UV lamp which made the fluorescent soil visible for the camera. In this way it was possible to quantify the cleaning rate of the cleaning procedures time-resolved and to compare their efficiency. The tests showed that the Mobile Cleaning Device is able to reduce water consumption and also to improve process safety with regard to hygiene e.g. by reducing spray shadow areas.
After completion of the cleaning test a kind of feasibility study was conducted in order to develop a concept of the Mobile Cleaning Device, how it could look like in the future and which additional features can make it still more adaptive and flexible. So this future concept has an even more product-like look and contains an optical camera sensor which can automatically detect soiled areas.
8.1 Mobile cleaning device (robot)
In the first part of the project the concept of the Mobile Cleaning Device was developed. It is an automated and self-driving machine which is moving through the whole PicknPack line and at the same time cleaning all modules consecutively with individual cleaning progress. Therefore, it has different nozzle types on-board and is able to spray several cleaning agents such as foam and water. This Mobile Cleaning Device was now integrated into the PicknPack line. Therefore, it uses the bars which support the product trays during production as rails on which it can drive. For the supply of cleaning agent it is connected to an automated hose drum. The hose drum is also driven by an engine and its rotation is synchronized with the speed of the Mobile Cleaning Device. In addition, the hose drum is standing on a movable table with wheels and with a docking station for the Mobile Cleaning. Hereby, the whole system can be moved easily through the whole factory so that the Mobile Cleaning Device can be used on several machineries.
Furthermore, the Software to control the Mobile Cleaning Device was also integrated into the line control system. Hereby, it is able to communicate with all the other modules of the PicknPack line. So when the position sensor of the cleaning device recognizes that it enters a module e.g. the Delta Robot it can send a message to the robot to start moving with a special pattern in order to support the cleaning progress by reducing spray shadows and moving relevant parts like the gripper closer to the cleaning nozzles.
8.2 Cleaning Efficiency Tests
After the line integration of the Mobile Cleaning Device several cleaning tests were performed to assess its cleaning efficiency in comparison the conventional CIP-systems and to determine ideal operating parameters. Those tests were performed exemplary for the PicknPack Delta Robot and the tunnels of the Sectional Frames.
For the tests in the Delta Robot the whole inner rear cover of the robot was coated with a fluorescent food model soil. In addition, a UV lamp and a camera (both IP69) were placed inside the robot to make the soil visible and to monitor the whole cleaning process. By this the cleaning process can be quantified. After the soil had dried for 20 hours the robot was cleaned with the different systems and with different operating parameters:
- Mobile Cleaning Device vs. Conventional CIP-System
- Cleaning agent (Foam) + water vs. water only
- Operating pressure: 3 bars and 4.5 bars
- Speed of the Mobile Cleaning Device: 2 mm/s (water only), 5 mm/s (foam + water), 10 mm/s (foam + water) speed values based on first lab scale tests with rotating spray heads
From the analysis of recorded images of the monitoring system the cleaning time was determined. This time was defined as the time which is needed to remove 95 % of the soil. Together with the flow rate generated by the two different cleaning systems in combination with the operating pressure also the water consumption was calculated.
The tests showed that in comparison to the conventional CIP-system with static nozzles the Mobile Cleaning Device needs more time to clean the robotic cell. This is due to the fact that the CIP-system consisted to two rotating spray heads mounted on the side covers of the robot while the cleaning device carries only one rotating spray head. But in return the Mobile Cleaning Device is able to reduce the water consumption by around 20 %. In addition, the tests showed that the Mobile Cleaning Device is able to eliminate spray shadow area due to its movement. Thereby, it is able to clean areas which cannot be reached by the static nozzles of the conventional CIP-system. So not only water consumption is reduced but also food safety is improved significantly.
Furthermore, the tests showed that using foam and water as cleaning agents instead of water only can reduce water consumption by around 60%.
8.3 Future Concept Design of the Mobile Cleaning Device:
The current version of the Mobile Cleaning Device is a prototype to show the feasibility and its advantages in comparison to conventional cleaning methods. The module will be developed further. Size will be reduced and the hygienic design will be improved to increase its suitability for the use in product contact areas. There will also be a version without wheels which is only carried by conveyors.
To make the device also more adaptive, it is planned to add an optical sensor system for automated soil detection. Since most food products contain fluorescent ingredients, it is possible to make them visible for a camera with a UV light. Both components will be integrated into the Mobile Cleaning Device. With this sensor system it will be possible to improve adaptivity during the cleaning process. It will be possible to determine which areas are really soiled and require cleaning and which areas don’t need to be cleaned. And it will also be possible to determine if all surfaces were successfully cleaned or if further cleaning is required.
Potential Impact:
1. Life cycle assessment
A life cycle assessment was carried out to investigate the impact of the PicknPack systems. In this assessment PicknPack focuses on the following objectives
- The development of a full life cycle picture of the automated flexible systems developed in the project.
- A sustainability assessment of the effects of flexible automation in packaging of fresh and processed food, including aspects like waste minimization, quality increase and energy/logistic optimisation.
- Use of well-stablished metrics to measure the sustainability performance: Life Cycle Assessment (LCA), Life Cycle Costing (LCC), qualitative social assessment and eco-efficiency.
- Demonstrate the environmental, economic and social benefits of the PicknPack automated/flexible systems in comparison with current conventional packaging equipment
Four assessment techniques used within environmental management and technology were implemented here. Life Cycle Assessment (LCA) was implemented at a first stage for the assessment of the potential environmental impacts of the PicknPack line vs. current benchmark packaging systems. The LCA was fully ILCD compliant since all the requirements were checked (see D9.1 for further information). The LCA considered a total of 24 combinations of tray size and type of machine. The amount of energy and plastic packaging material for each packaging size was obtained by extrapolation calculation, while the amount of packaging scrap was estimated in AutoCAD. Additionally, power consumption in the line was gathered through a combination of on-site of power measurements, both for conventional thermoforming/packaging equipment and the PicknPack lines. In the specific case of the PicknPack line, the power consumption was accompanied of a desktop analysis with all the partners. The consumption of water and cleaning agents in was considered as well. With all these information life cycle diagrams and specific life cycle inventories were prepared and modelled in SimaPro 8.3 software. Then, the potential life cycle impacts were calculated for all the systems in accordance with nine impact categories, based on the recommended life cycle impact assessment methods by ILCD. The LCA results were provided with and without the impacts of the food. Additionally, a sensitivity analysis was made in the LCA in order to consider the effects of the geographic location (UK, NL, ES) because of the differences on the Country energy mixes.
Based on the previous data collection and LCA work, the economic assessment through LCC technique was also made. The LCC rely on the LCA structure, so all the background data on flows of energy and materials was considered. These amounts were then multiplied by the economic cost related to them, although other cost categories beyond materials and energy consumption were included (i.e.: labour costs, capital charges, etc.). The cost of each packaging size/machine and type of food contained were subsequently obtained.
In a third step, a social analysis was carried out. Even though it was originally planned as a packer acceptance study in order to get a quantitative value for the Willingness.to Pay (WTP) this was no further possible because of the delays on the final layout of the line and the integration problems between the modules. Because of that it was decided to change the strategy to a qualitative social analysis aimed at checking the potential improvements in terms of (1) human resources, (2) productivity of the employees and (3) expectations on the of qualification of the employees.
Finally, an eco-efficiency analysis was developed in order to analyse if the PicknPack line was able to provide the lowest environmental impact and cost compared to conventional packaging equipment. Spider and scattered dispersion diagrams were created for such purposes.
2. Main dissemination effort
2.1 Workshops
A key deliverable was to organize six dissemination workshops with a wide geographical distribution of venues targeted essentially at industry to maximise the final impact of the project. This was achieved with three of the workshops being associated with significant industrial exhibitions and two being associated with the final demonstration of the project technical outputs. A total of about 300 attendees were registered mainly senior personnel from industry or commercial organisations. A list is included in deliverable 10.3. A comprehensive set of high quality publicity leaflets were designed and produced together with a professional level conference folder for workshop delegates.
2.2 Project website
A public project website was designed and set up to provide a major communication channel to disseminate our activities and outputs. All workshop presentations were posted on the website as well as regular newsletters, details of consortium members and their roles, technical presentations, journal publications and a regular stream of videos. In October 2015 the website had 3837 hits which increased in April 2016 to 19117 and finally to 55450 in September 2016.
2.3 Newsletter
A Mail Chimp tool used for disseminating newsletters has also assisted driving engagement and increased traffic to the site. Campaigns have an above average open rate and a low bounce rate indicating that the content is engaging and thus retaining visitors. Top locations have been the UK, Netherlands and the USA.
2.4 Social and professional online media
Twitter, Facebook and YouTube have been set up to disseminate the technical outputs of the project. The pages were continuously maintained. However despite repeated requests many partners are still not connected to social media so they cannot join the PicknPack channels.
You Tube channel analytics indicates that 83% visitors are male and remaining 17% female. Top watch locations are Netherlands, UK and Spain.
2.5 Industrial Advisory Board
An Industrial Advisory Board was established and formal meetings arranged. Support for travel and accommodation was provided for members to encourage attendance. Their input proved to be valuable in determining the direction of the project.
2.6 Other dissemination activities
Strong communication links were established with a number of organisations including the UK’s Food Manufacturing Engineering Group, Chilled Food Association, Food and Drink Federation and industrial companies including Kuka, ABB, Siemens, Omron, Nestle, Bakkavor and Unilever as well as the UK’s Robotics and Autonomous Systems, Knowledge Transfer Network the Northern Robotics Network a Consortium of some 50 companies from various sectors including nuclear and aerospace that cosponsored our 4th project workshop.
The IML Publishing Group which hosts the industrial / commercial A4E Trade Fair published full details of our workshop in their Process Engineering Journal, invited a series of presentations at their events and are planning an article on expected impact of the technology. New Food has published one paper and suggested a special issue on all aspects of the project. Full details of all published articles and presentations are given below.
2.7 Training
Two training events were planned in association with the final two workshops in the series. The Project Board however recognised the need to focus our attention on delivering a full working demonstrator at the 5th workshop so the training event was postponed to the September 2016 event which was held at the Holbeach campus at the University of Lincoln.
The event attracted 83 student registrants with an additional nine academic / industrial supervisors from a range of companies including Bakkavor, Nestle, Greenvale, Dalehead and the ADDO Food Group. An added feature was the demonstration of a large (100kg payload) robot cell moving bulk food material provided courtesy of O.L Systems.
Over the span of the project we developed a liaison with the E.C project Smart-E which has focused on soft robotics for manufacturing.
2.8 Publications
All partners were continually polled to generate material for the website and provide information on all publications, industrial presentations etc. so that we could generate a comprehensive archive of disseminated materials. Similar requests were made to archive all patent applications and the results are tabled in deliverable 10.4.
2.9 Demonstration
The PicknPack was demonstrated to 349 companies and students as follows:
Denmark 23 February 2015 the thermoformer under construction was presented to 40 companies from the Danish Food and packaging industry.
The Netherlands 26 and 27 May 2016 the PicknPack line was demonstrated as an almost complete functioning line. The first day with 48 and the second day with 54 companies as participants. The demonstrations in The Netherlands was done with the thermoformer, gripper, pick-n-place-robot and the quality systems all integrated. As the laser was unable to work with PET films the laser, printer, integrity and the cable robots was demonstrated as individual units. In another room a number of other PicknPack functions was demonstrated. These functions were the design process using the three different flexible mould systems. The Printer, exchange system, quality evaluation, integrity system, and the cable robot systems was explained although also demonstrated in the other room. The heating system, line integration, software systems, cleaning system and the RFID systems had own booths in this room.
Spain in June 2016 Tecnalia demonstrated the cable robot to several delegations with about 90 participants. The visitor was from: 1. 60 experts from RoboTT-net OpenLab; 2. Visitors from 4 different universities in USA and; about 30 Australian companies also participated in the demonstration of the cable robot.
England 13 September 2016 a special demonstration was organised for 36 companies. The demonstration in England included presentations of all modules in PicknPack. The demonstration was done as demonstrations of each single module.
England 14 September 2016 PicknPack together with University of Lincoln organised a student’s day with 81 students as participants. On the Students day some of the PicknPack modules was selected for a special presentation. These modules was Robot with grippers, Hygiene robot, RFID, Printer, Quality sensing, Laser and heating systems.
2.10 Project video
A professional project video was made to demonstrate the functionality of the line and its modules. Special animation was added to explain the flexible software and line configurations. Specific modules were also produces to explain in more detail the benefits of each module.
3 Exploitable results
PicknPack dedicated a work package to exploitation of the project results. The objective of this work package was to generate useful tools in order to orientate the exploitation plans of the project results. Moreover, these tools were shared among the partners to give them useful information about the market needs and requirements in order to guide their developments in a more realistic, adjusted and optimized way and to give them clues about opportunities in the sector. The work was divided in different tasks.
As a first task, evaluation of the acceptance of the robotic products in the food packaging sector was performed. The objective was to know the real profile of the market, in order to understand the real needs of the industry and the factors influencing the acceptance and implementation of automatic systems in fresh food processing plants in Europe. This was the first step to develop adjusted technology to the market and oriented exploitation plans. For reaching this aim, a wide survey at European level was done to food industries’ decision makers. In summary, the study investigated the acceptability of innovative automatic systems by the FOOD and POSTHARVEST industry. The investment intention by companies of both sectors, and the factors that push them to declare a positive or negative intention were analyzed. The study also provided a context to these factors. Thus the knowledge of the particular and common characteristics of both sectors helped to orient the approaching strategies to design the adequate exploitation plan (in task 12.4) for automation suppliers to facilitate a greater implementation in the food industry of today. The study helped the exploitation task by identifying the helpful (OPPORTUNITIES) and harmful (THREATS) factors that could affect the business in the current situation of the sector. The discussion and conclusions from this study helped to define main recommendations for the technology suppliers in order to orientate their business in function of the factors influencing the implementation decisions of the food companies. The results and conclusions of this work were useful to contextualize the exploitation plans described in Deliverable 12.5 “Technology Exploitation plan”.
Secondly, the analysis of factors impacting the Economic viability of automated food packaging systems was carried out. Moreover, a bibliographic research was done in order to have an orientated vision and costs and values of the business. Based in the gathered information, a tool to simulate economic viability was developed on Microsoft Excel™. This tool will allow defining different base scenario reflecting the current target industries. The simulation tool makes it possible to display results in a simple and comprehensible way through figures, graphs and tables. Simulation on a current target industry situation shows an estimated graph of its cost structures. Based on this results a return of investment table is calculated showing the different key financial concepts that will allow to know if a determined PnP investment will be viable in that current scenario or not. Finally with the performance data of the developed prototypes, it will be able the display of a new scenario which will tell the final benefits for the company after implementing these new PnP modules.
Evaluation of the societal impact of robotics systems in the food packaging sector is investigated, a tool for assessing the impact of the PicknPack line in the workers welfare in terms of occupational Risk Prevention (ORP) was designed, focusing mainly on the ergonomic aspects of the manual labour done by the workers. The objective was to define the strategy in order to check the line design, identify and reduce the ergonomics hazards present in it and minimize their effect on the workers’ health. For that issue a theoretical identification of the critical points was studied and suggestions of improvement were described. Moreover the methodology to follow was defined and explained to the partners. In the study several ergonomic risks and suggestions for minimizing them were done. Related to the influence of the PicknPack line on consumers, the impacts of the line on the final consumers of products were identified. The automation and the use of new technologies enable to have high process control and production flexibility. This allows meeting the expectations of consumers more accurately and efficiently than traditional forms of production. This impact was also confirmed with the acceptance study performed in the same work package, in which several European companies were surveyed about needs and perceptions about automation in their business. From this study the impacts on social aspects and the impacts that were related to the consumer’s product acceptance were confirmed.
The final objective of this work package was to Promote exploitation of the project results and stimulating new applications. The above described tasks helped as a tool for discussing with technology developers of the project about the corresponding potential exploitation paths. The first task by gathering form the acceptance study the information to draw up a SWOT analysis, and the economic task by providing an estimation of the impact that the implementation of the equipment in the business would have in the potential target companies.
Even though the Picknpack line as a whole system was not entirely functional, the different modules that composed the PicknPack project reached a different state of development during the course of the project. Most of them have reached a TRL between 4 and 7. Some of the modules were even quite close to market. After finishing the PicknPack project, the aim is each individual partner to keep on developing the technology, alone or in collaboration with partners, in order to reach a higher TRL closer to market. Under the life of the project a close communication between partners was kept in order to define and discuss the characteristics, benefits and exploitation plans of the achieved modules and results.
The objective of this task was to describe the results of the PicknPack project and the potential ways of their exploitation via different approaches for future scenarios in order to reach a closer level of readiness towards the market. With this aim several communications and two workshops were done under the project, in order to explain to partners the main concepts regarding exploitation and to ask them oriented inputs. This exercise was accomplished with the aim of not only them to describe the innovations, but to really translate the impact of the developments by describing the benefits of the innovations in terms that are understandable to the potential clients. One challenge of this task was to make the partners conscious of the importance of describing, to the potential clients, the added value of their research in an understandable and clear way. For that objective, the information was asked in several formats (questionnaires, tables, message maps, elevator pitch, etc.) in order to get clear answers to the problem solved by the results, a clear description of the solution, the benefits and differentiation from the competitors (unique selling point), among others. The two demonstrations that took place in Wageningen (Netherlands) and in Holbeach (United Kingdom) respectively, allowed not only to gather information about the requirements of the industry, but also the opportunity to make contacts and to build a strong network that will help in the promotion of further developments. Deliverable 12.5. Exploitation plans, is the summary of the developed or of exploitable outcomes of PnP project, and the elaboration of plans for their exploitation by partners involved in the project.
List of Websites:
Public webiste: www.picknpack.eu
Access deliverables: erik.pekkeriet@wur.nl
Contact:
Erik Pekkeriet (project coordinator)
Email: erik.pekkeriet@wur.nl
Tel: +31317483372
Address: Droevendaalsesteeg 1, 6708 PD Wageningen