Final Report Summary - AMLIGHT (Development of an Automatic harvesting system for green asparagus with stalk detection in Ambient Light)
Executive Summary:
Summary description of the AmLight project context and objectives
Green asparagus is considered a very healthy vegetable not only for containing vitamins and nutrients but also because the vegetable is low in calories and perfect diabetic application. Even though some harvesting aids exist, the harvesting is 100% done manually which makes it rather labour intensive. The harvesting itself is quite hard and unhealthy for the harvesting people since the permanently work in a bend down position.
On the background of the above said the AmLight consortium came together to develop an automatic harvesting system for green asparagus.
The first period of the project was occupied by specifying the technical necessities for the harvesting machine and the development of manually operated tool. These tools were tested on asparagus field in UK and Germany during the harvesting season 2013. Those tools which have shown the best potential for success were than up-scaled and designed after the harvesting season.
During the second period of the project all sub components of the overall AmLight harvesting system (carrier unit, detection system, control soft- and hardware, fast kinematic and harvesting tool) were designed, manufactured and integrated into the AmLight prototype.
The fully assembled prototype was than first functional tested and later also tested under real conditions on asparagus fields during the harvesting season 2014.
After the end of the harvesting season the remaining time of the project was used to evaluate the field test results and to optimise the prototype. For optimisation purposes and artificial 5 meter long asparagus dam has been built outside the coordinator facilities.
The main technical achievements of the AmLight project are:
• A field tested and optimised prototype
• A list of recommended technological solutions for further improvements of the machine
With respect to the dissemination of project results the following has been achieved:
• The AmLight project was presented through posters and flyers during the exhibition expoSE (Karlsruhe, Germany) in November 2013.
• A video clip about the prototype has been uploaded to the AmLight webpage (www.amlight.eu).
Project Context and Objectives:
The overall project objective is to develop the first time ever a fully automatic harvesting machine for green asparagus working with a detection system in ambient light, which should be affordable to SMEs and which has the primary features:
• Detection success higher than 95 %
• Fully automatic harvesting (only one operator for 2-3 machines)
• Automatic quality grading during harvesting
• Harvesting capacity of up to 1.000 asparagus stalks per hour
• Collateral damages caused by injuring neighbouring asparagus stalks less than 5 %
• Increase the European competiveness in asparagus and creating new, healthier working places
For reaching this ambitious task four technical objectives have been defined, each one of them with a specific aim – to develop one of the four sub components of the harvesting machine, which are:
a) The detection system
b) The harvesting mechanism (fast kinematic and harvesting tool)
c) The self-driven a carrier unit
d) The control program
During the first period of the project the main focus of the AmLight consortium was on the development of the harvesting tool and the asparagus detection system.
The harvesting tool was developed in steps of hand-tool design, hand-tool manufacturing, and hand-tool testing and evaluating. After evaluation a new hand-tool version was designed, and this was continued until the results were promising enough to be considered as a blueprint for an automatic version.
For the development of the asparagus detection system a special stereo type of camera was chosen and calibrated, an initial test program has been developed, and the camera was mounted to a special made push-trolley and taken to a field to gather first images.
While all developments of the first year were based on manual operation and tested during the first harvesting period of the project, the developments of the second period had the aim to convert the manually tested concepts into an automatic harvesting system. From summer 2013 until the end of that year the following activities have been performed by the AmLight partners:
• Development of machine CAD designs
- CAD of harvesting tool
- CAD of fast kinematic
- CAD of carrier unit
• Purchasing all items according to parts lists from CAD drawings
• Development of IPP based on images taken during harvesting season 2013
• Development of control program, control panel and remote control
With the delivery of purchased items the partners began with the assembly of subcomponents as follows:
• Assembly of harvesting toll and fast kinematic at UNIHB
• Assembly of the carrier unit at STRAUSS
• Assembly of the control panel and remote control at PYFE
Toward spring 2014 all assembled subcomponents were shipped to the facilities of STRUASS for integrating all subcomponents into the AmLight prototype. The integration was followed by functional testing of all subcomponents and the setting of working parameter such as off-sets for the camera and the harvesting tools, speed of servo motors, encoder outputs etc.
Once all functions were tested successfully the machine was first time taken out for tests on the field of partner WERNER. In total three field tests were executed during the harvesting season 2014, each one having a specific aim:
a) Verification of the detection system under real outdoor conditions
b) Testing the precision of the detection system
c) Testing the performance of the harvesting system
The evaluation of test results after the end of the harvesting season has reviled a number of weaknesses of the harvesting machine. A list of necessary improvements was established by the consortium and all partners agreed to implement some of the listed improvements already during the remaining month of the project. The main improvements achieved before the 31st of October 2014 were:
- Integration of a stalk tracking system in the detection system
- Modification of the harvesting tool toward more gentle gripping
The final meeting of the AmLight project took place on the 7th of October at STRAUSS facilities. Besides technical and management related presentations the AmLight prototype was also demonstrated to all partners. Another highlight of the final meeting was the presentation of the project movie, which illustrates in an attractive way the functions of the AmLight harvesting machine. The movie has been uploaded to the AmLight website www.amlight.eu.
Project Results:
The S&T developments of the AmLight project were distributed into 7 work packages. A short description of the work performed and the results achieved in each work package is shown below:
Work Package 1 – System specification
Main results:
- A document in which all specification for this project are embraced
Work performed:
The specification work of the work package was subdivided into three areas
a) Specification of the visual (detection) system
The aim of this area was to determine the best hardware solution (camera + sensors) to fulfil the need of detection in ambient light, and which would allow for efficient development of an IPP (Image Processing Program). Resulting of discussions was to use a stereo camera for taking images.
b) Definition of the system performance
Mainly the end-users in the project were defining performance requirements in terms of field properties and yield/harvesting expectations. Hence this area of specification had three main parts of characterisation (i) the cultivation of asparagus, (ii) the manual harvesting efficiency, and (iii) the expected machine efficiency.
c) Definition of mechanical motion characteristics
Based on the data of the system performance the partners pre-defined potential motions, which could lead to gentle cutting at high speed. One of the key conclusions was to provide a redundant system, means several fast kinematics in row in order to be able to harvest even two (or more) asparagus stalks, which grow close to each other.
Work Package 2 – Development of the detection system
Main results:
- Development of a camera assembly, to take asparagus images in ambient light
- Development of an IPP, to determine/calculate coordinates of asparagus stalks
- Development of an integrated operator interface, for machine optimisation
- A comprehensive data-base of asparagus images
Work performed:
During the first harvesting season of the project (2013) a purchased stereo camera was fitted to push trolley. This assembly was used to calibrate the camera and to take some first images of asparagus under real conditions, which were needed for the development of the IPP.
After the harvesting season 2013 two main developments for the detection system were made:
a) The development of the environmental awareness module
The best solution of a 3D detection system capable of dealing with the ambient light was the use of a CMOS sensor based stereo camera because it supports native wide dynamic range (WDR). WDR is a feature that handles a wide range of lighting conditions in a scene and in the camera image both the darkest and the brightest parts are taken into account, resulting in more details on both ends. Hence the system can be used in changing light conditions.
b) The development of the IPP
Image Processing was parted into two important parts: the pre-processing part and segmentation part. The raw output (distorted, unrectified, and color coded) from the stereo camera is termed as “raw image”. The debayering algorithm is used to obtain a full-color image. In the next step the white balance adjustment is performed. White balance is the process of removing unrealistic color casts, so that objects which appear white for human eye are rendered white in the acquired stereo image. This step is followed by the camera calibration to extract the metric information from 2D images. Since the acquired stereo images are unrectified, they cannot be directly used to compute the 3D positions.
After the pre-processing is done, the asparagus stalks could be segmented from the environment within the segmentation module. Since the asparagus stalks are green in color, they have to be separated from the other colours in the background. To achieve this, a mixed color conversion is performed.
Feature extraction and foot point detection of the asparagus stalks are performed using the robustly segmented image. Furthermore, the classification of the asparagus stalks is done to decide if they are to be harvested or not.
Work Package 3 – Development of the harvesting mechanism
Main results:
- 3D CAD design of the harvesting tool
- 3D CAD design of the fast kinematic
- Manufacturing of two complete harvesting mechanisms
Work performed:
In the very beginning of the project several ideas for harvesting tools were built to be used manually. The manual testing of these tools took place in the first harvesting season of the project in 2013. After the harvesting season the most promising tool was then up-scaled and designed for automatic use, and two tools were manufactured.
Also the fast kinematic, was designed in the second period of the project in close cooperation with WP4 (development of the carrier unit) and WP5 (development of the control program).
The function of the fast kinematic is to move the harvesting tool from a standby (waiting) position perpendicular (y-axis) to the driving over the dam of a detected asparagus stalk. When approaching the stalk the fast kinematic will perform a movement which takes the harvesting tool in less than one second a high position down to ground level (where the asparagus stalk is harvested) and back up into the high position. From there it will move the harvesting tool (and the asparagus) to a drop-off position above a collecting belt conveyor. After the drop-off it returns the harvesting tool into the home (stand-by) position.
Y-axis movement – which is a linear track (motor not shown) to be fitted to the carrier unit (see WP4). A slide on the linear track carries the part of the fast kinematic responsible for the z-axis movement.
Z-axis movement – which is a unit fitted to the slide on the linear track. It consists of a motor connected to a pulley. A belt is connecting the driven plus to additional pulleys. Fitted to the belt is an aluminium construction (bracket) which holds at the bottom ending the harvesting tool (harvesting tool not shown). The aluminium construction is sitting on both sides (left and right) a L shaped bend track. Moving along these track perform on motor activation first a short horizontal linear movement, followed by a cornering downward movement and finally a linear vertical movement down.
In order to reach a high harvesting efficiency two of these fast kinematics were manufactured by UNIHB. The idea of the redundancy is: while one fast kinematic is in operation of harvesting an asparagus stalk the second one can go into position for the next stalk and vice versa.
Work Package 4 – Development of carrier unit
Main results:
- 3D CAD design of the carrier unit
- Carrier unit manufactured
- Calculated and integrated power supply system
Work performed:
Part of the specification (WP1) was to determine the number and strength of motors needed for the prototype. Based on that the power consumption of the entire machine was calculated and batteries as well as transformer designed and provided.
In the second period of the project the entire carrier unit was 3D CAD designed and manufactured. The carrier unit consists of:
A) A robust stainless steel frame
B) Two driven front wheels
C) Two freewheeling, steering back wheels
D) Set of batteries for the power supply
E) Belt conveyer for asparagus collection
F) Brackets to fit both fast kinematics
G) Bracket for control panel
H) Bracket for camera
The carrier unit delivers sufficient space for the integration of a camera at the front part of the unit, a control panel (front right) and a remote panel (back right), as well as two fast kinematics in the centre part of the machine.
After completion of the CAD drawing all machine parts were purchased, produced and/or taken from stock. The successful fabrication of the carrier unit was completed prior to the arrival of the subcomponents.
Work Package 5 – Development of control system
Main results:
- Fully functional PLC program for the control of all machine motions
- User friendly operation through the remote control
Work performed:
The development only started in month 9 of the project. A new version of PLC programming has lately been launched by OMRON (Type: NJ301-1200), which provides faster data transmission between a PC and the PLC. This has been chosen as the best opportunity for the AmLight motion control program.
An integral part of the control program is to establish communication between the IPP (WP2) and the harvesting mechanism (WP3). The control system will process the received data from the asparagus detection system (IPP) to provide the corresponding optimized movement pattern to the mechanism (vehicle and fast kinematic). The detection system will provide the exact position of the asparagus in the dam.
The hardware interface for the AmLight system consists of three control panels: Remote Cabinet, Main Cabinet and Encoder Box.
Main Cabinet: This is the control unit of the AmLight System. It connects all different systems and the PLC inside the Main Cabinet run the software that will makes the whole system runs together. To integrate the Control Panel with the IPP System and the mechanical parts we use the external connectors on the cabinet.
Remote Cabinet: There is a display that will allow the operator to adjust different parameters of the AmLight system such as speed of actuators, off-sets of camera and fast kinematics etc. The display also informs about detected errors, status of different parts of the system such as inputs, drivers, harvesting process. One of the main functions, however, is to choose operation in either manual or automatic mode. In automatic mode the machine is running at a set speed over an asparagus dam, searching for asparagus stalks and performing automatic harvesting cycles. In manual mode the machine can be turned at the end of the dam to the next one and all motions can be activated by push buttons.
Work Package 6 – System integration & functional testing
Main results:
- All sub-components successfully integrated into the carrier unit
- Detection system successfully calibrated for automatic use
- Communication between IPP and PLC established
- All sub-components functional
- Operator manual developed
Work performed:
Work package 6 had three very clear defined and distinguished tasks as follows:
Task 6.1: Assembly of the AmLight prototype
All subcomponents as resulting from work packages 2, 3 and 5 were delivered to STRAUSS facilities for assembly of the AmLight prototype. The work done by STRAUSS, IMIX and the RTDs included:
a) The mechanical integration of all sub components into the carrier unit.
Part of the mechanical integration was (i) the fitting of two fast kinematics into foreseen positions as explained in WP4, (ii) the installation of the camera, and (iii) the mounting of the control panel as well as the remote control.
b) The electrical assembly
The main issue of the electrical work was the wiring of the entire system, means that all actuators (fast kinematic and driving of vehicle), the camera, and the remote panel were wired in the connectors of the main control panel. Also a connection between the control panel and the (battery) power supply was established. Finally all end-switches were for correct and safe movement of the fast kinematic.
Task 6.2: Functional tests of AmLight prototype
After successful assembly of the AmLight prototype the function of all sub components was tested. This included:
• Calibration of the camera. First the off-set from a 0-position of the machine, the height above ground level and the angle of the camera were determined in order to have precise correlations between coordinates of detected asparagus stalk and the positioning of the harvesting tools.
• Encoder read-out. One of the driven front wheels had an encoder installed to measure the distance the harvesting machine would need to travel between asparagus detection and harvesting. The encoder read-out was therefore checked against real distance covered.
• Communication between IPP and PLC. One of the key issues of the functional testing was the establishment of the communication between the IPP and the PC. Here the translation program needed to be modified.
Function of fast kinematic. The functional testing of the fast kinematic started with determining the correct off-set from the 0-position of the machine. After that motors were activated the check the movement of the harvesting tool in Y- and Z-direction. Finally the speed was set to the maximum possible without vibrations. First the function of the fast kinematic were tested though the push buttons on the remote control in manual mode, later simulated positions were sent from the IPP to also get the function tested in automatic mode.
After successful completion of all functional tests the AmLight prototype was ready to be tested under real conditions on an asparagus field.
Task 6.3: Development of user manual
The user manual which has been developed along the work of WP 6 is a detailed description of menus available through the remote control panel, which is used to set-up and operate the harvesting machine. Through the user manual an operator can learn how to
• Set up motors
• Set up the encoder
• Change off-sets
• Deal with errors
• Read out asparagus positions
The complete user manual is available on the AmLight webpage and has been delivered through the document D6.2.
Work Package 7 – Field tests – operator training – evaluation – optimisation
Main results:
The main results of field tests during the harvesting season 2014 and machine optimisation after the season the following results have been achieved:
Asparagus detection success rate up to 70 %
Detection precision at ± 3 cm
Harvesting speed at about 50 % of expected maximum
A comprehensive list of still required optimisations
Work performed:
During the harvesting season in 2014 the consortium had two main tasks to reach:
a) Field tests to determine the performance of the prototype under real conditions
On different fields of end-user WERNER on three full days, each day having a specific task, as follows:
04.06.2014 – Verification of the asparagus detection system under real outdoor conditions.
10.06.2014 – Determination of the precision of positions calculated by the IPP.
19.06.2014 – Performance of the fast kinematic and the harvesting tool.
After the first and the second day of tests the IPP was improved according to the information gained by the operator training.
The performance tests of the fast kinematic have revealed two technical problems: (i) too slow image processing and (ii) movement of the carrier unit not straight-lined enough. Both problems in combination caused that no real harvesting tests could be done. To overcome this problem the consortium decided to have an artificial dam built outside STRAUSS facilities under better controllable conditions for the carrier unit.
b) Operator training to optimise the IPP
The operator training did not have the aim to train an operator, but to get information of harvesting experts (the operator of the future) to train the detection system. As mentioned above the operator training was performed during the first and second test days and had the aim to improve the correct asparagus identification by the detection system. In praxis it meant that the expert was identifying all asparagus stalks which should have been detected and harvested by the prototype. Images of those stalks were saved in a data base for later optimisation of the IPP. In the other direction the expert was also identifying those asparagus stalks which have been detected incorrectly. These images were also saved in a different data base for later analyses.
The main achievement of both testing days together with a harvesting expert was a large number of images which clearly identified asparagus stalks either ready for harvesting or not. With these image data the IPP could even be optimised without going back to a filed.
c) Test data evaluation and machine optimisation
At the end of the field tests and the operator training all test data were evaluated and the results discussed by all project partners. The outcome of the test data evaluation was a table, in which for each machine component (detection, control program, vehicle, fast kinematic and harvesting tool) all results/achievements were qualified and categorised into:
Acceptable – No need for optimisation during AmLight
Moderate – Optimisation should be considered during and/or after AmLight
Bad – Optimisation indispensable. Unclear what can be still achieved during AmLight
This table was the bases for optimisation done after the end of the harvesting season and may also be used for follow-up improvements after the end of the project.
For optimisation purposes the consortium decided to have built a 5 meter long artificial asparagus dam outside STRAUSS facilities. Being outside real conditions for the detection system were given and the carrier unit could run on levelled ground, which was required at this stage of developments to get the fast kinematic and the harvesting tool tested and optimised.
In total five sessions for testing the fast kinematic and any optimisations to the entire AmLight prototype took place during the summer and late summer time of 2014, resulting in the following:
• At a speed of 0.5 m/s about 70% of all asparagus stalks can successfully be detected and only one position with a precision of ±3cm be sent to the PLC. However, this must be confirmed by long-term real field tests.
• The principle of the motion of the fast kinematic with sufficient precision has been proven to be functional. However, to use the system at high speed the firmness must be improved.
• A redeveloped version of the harvesting tool has finally shown the potential to be successful; however it has never been tested under real field conditions.
The AmLight partners concluded at the end of the project that the prototype has reached pre-competitive status. The main components (asparagus detection, fast kinematic, harvesting tool, control program) have been proven to be functional. All systems, however, need to be improved and intensive field testing and demonstration is indispensable for any future commercial success.
Potential Impact:
4.1 Potential impact
Once the AmLight harvesting will have reached commercial status it offers great potential impact to many areas as:
a) Improved working conditions for harvesting staff
Green asparagus is cultivated in long rows at ground level or low dams and the stalks are usually cut just above or under the soil. Harvesting asparagus means that the staffs have to work all days in an uncomfortable and unhealthy bend-down position. With the application of the AmLight machine to green asparagus harvesting people will get trained to operate a machine which will then do the hard work. Hence the introduction of AmLight is offering better working conditions and thus less health problems, as well as an increase of skills for people who operate the machine.
b) Availability of green asparagus at affordable costs
Asparagus is one of the oldest vegetables crops cultivated by mankind as food. It is said to have a number of health benefits as it is low in calories, contains no cholesterol and is very low in sodium. It is a good source of folic acid, potassium, dietary fibre, vitamin C, carotene and rutin.
On the back side of the coin, fresh green asparagus is also one of the most expensive vegetables. Today harvesting is done by seasonal workers. Usually one worker is needed per hectare and season putting the harvesting cost to approx. 0.7 €/kg, which presents about 1/3 of the total production cost. The increasing labour costs (especially the start of minimum wages in many European countries) will even increase the costs of green asparagus in the coming years.
In order to maintain fresh green asparagus supply to a large share of population and at affordable costs, farmers are forced to optimize their harvesting process and to introduce harvesting aids and/or fully automatic harvesting solutions. In this direction AmLight will help to ensure that European habitant will have access to one of the healthiest vegetables produced on European fields.
c) Increased competitiveness to European asparagus farmers
The European production of green asparagus has increased to making Europe the 4th largest producer of green asparagus after China, Peru and the US in the world. Some 90% of the asparagus consumed within the EU27 is consumed as fresh product, mainly during March and June, the main harvesting season. More than 95 % of this is grown in Europe, the main countries being Spain, Italy, Germany and France.
However, despite the positive trend European green asparagus farmer are also facing a growing competition form asparagus imports from the above named countries, from where ‘fresh’ (chilled in containers during shipment) asparagus can be supplied at low cost than European farmers could produce it. As mentioned above already this situation is forcing the European asparagus producer to maximise the automation level of their processes. While process steps like cleaning, cutting, sorting and partly the packaging have already reached an advanced automation level the harvesting remained a fully manually done job. The AmLight developments will help closing that gap in automation and hence supporting asparagus farmers to remain competitive.
4.1 AmLight dissemination activities
In the duration of the AmLight project to following dissemination activities have been undertaken:
a) The press release
In the first month of the project a press release has been launched through the network of partner UNIHB. The content of the press release was a general description of the planned work, the objectives of the project and the names of participating partners.
b) The project website
The AmLight website has been established in the second month of the project – www.amlight.eu.
All information produced in the project has been uploaded to the website. Confidential information is password-protected, other areas are public accessible. The dissemination material developed in the project (see below) are free downloadable from the website.
c) Dissemination Flyer / expoSE
A dissemination flyer and two posters have been produced in October 2013. The reason for the early-in-the-project production was that a chance was provided by STRAUSS to disseminate AmLight on their stand at the expoSE exhibition in Karlsruhe (Germany).
d) Dissemination video
A dissemination video has been developed towards the end of the project showing in an attractive way the machine components and its functions. Direct link to the video: http://www.amlight.eu/results.php
List of Websites:
www.amlight.eu
Strauss Verpackungsmachinen GmbH
Email: info@strauss-pack.com
Phone: +49 4161 14220
Summary description of the AmLight project context and objectives
Green asparagus is considered a very healthy vegetable not only for containing vitamins and nutrients but also because the vegetable is low in calories and perfect diabetic application. Even though some harvesting aids exist, the harvesting is 100% done manually which makes it rather labour intensive. The harvesting itself is quite hard and unhealthy for the harvesting people since the permanently work in a bend down position.
On the background of the above said the AmLight consortium came together to develop an automatic harvesting system for green asparagus.
The first period of the project was occupied by specifying the technical necessities for the harvesting machine and the development of manually operated tool. These tools were tested on asparagus field in UK and Germany during the harvesting season 2013. Those tools which have shown the best potential for success were than up-scaled and designed after the harvesting season.
During the second period of the project all sub components of the overall AmLight harvesting system (carrier unit, detection system, control soft- and hardware, fast kinematic and harvesting tool) were designed, manufactured and integrated into the AmLight prototype.
The fully assembled prototype was than first functional tested and later also tested under real conditions on asparagus fields during the harvesting season 2014.
After the end of the harvesting season the remaining time of the project was used to evaluate the field test results and to optimise the prototype. For optimisation purposes and artificial 5 meter long asparagus dam has been built outside the coordinator facilities.
The main technical achievements of the AmLight project are:
• A field tested and optimised prototype
• A list of recommended technological solutions for further improvements of the machine
With respect to the dissemination of project results the following has been achieved:
• The AmLight project was presented through posters and flyers during the exhibition expoSE (Karlsruhe, Germany) in November 2013.
• A video clip about the prototype has been uploaded to the AmLight webpage (www.amlight.eu).
Project Context and Objectives:
The overall project objective is to develop the first time ever a fully automatic harvesting machine for green asparagus working with a detection system in ambient light, which should be affordable to SMEs and which has the primary features:
• Detection success higher than 95 %
• Fully automatic harvesting (only one operator for 2-3 machines)
• Automatic quality grading during harvesting
• Harvesting capacity of up to 1.000 asparagus stalks per hour
• Collateral damages caused by injuring neighbouring asparagus stalks less than 5 %
• Increase the European competiveness in asparagus and creating new, healthier working places
For reaching this ambitious task four technical objectives have been defined, each one of them with a specific aim – to develop one of the four sub components of the harvesting machine, which are:
a) The detection system
b) The harvesting mechanism (fast kinematic and harvesting tool)
c) The self-driven a carrier unit
d) The control program
During the first period of the project the main focus of the AmLight consortium was on the development of the harvesting tool and the asparagus detection system.
The harvesting tool was developed in steps of hand-tool design, hand-tool manufacturing, and hand-tool testing and evaluating. After evaluation a new hand-tool version was designed, and this was continued until the results were promising enough to be considered as a blueprint for an automatic version.
For the development of the asparagus detection system a special stereo type of camera was chosen and calibrated, an initial test program has been developed, and the camera was mounted to a special made push-trolley and taken to a field to gather first images.
While all developments of the first year were based on manual operation and tested during the first harvesting period of the project, the developments of the second period had the aim to convert the manually tested concepts into an automatic harvesting system. From summer 2013 until the end of that year the following activities have been performed by the AmLight partners:
• Development of machine CAD designs
- CAD of harvesting tool
- CAD of fast kinematic
- CAD of carrier unit
• Purchasing all items according to parts lists from CAD drawings
• Development of IPP based on images taken during harvesting season 2013
• Development of control program, control panel and remote control
With the delivery of purchased items the partners began with the assembly of subcomponents as follows:
• Assembly of harvesting toll and fast kinematic at UNIHB
• Assembly of the carrier unit at STRAUSS
• Assembly of the control panel and remote control at PYFE
Toward spring 2014 all assembled subcomponents were shipped to the facilities of STRUASS for integrating all subcomponents into the AmLight prototype. The integration was followed by functional testing of all subcomponents and the setting of working parameter such as off-sets for the camera and the harvesting tools, speed of servo motors, encoder outputs etc.
Once all functions were tested successfully the machine was first time taken out for tests on the field of partner WERNER. In total three field tests were executed during the harvesting season 2014, each one having a specific aim:
a) Verification of the detection system under real outdoor conditions
b) Testing the precision of the detection system
c) Testing the performance of the harvesting system
The evaluation of test results after the end of the harvesting season has reviled a number of weaknesses of the harvesting machine. A list of necessary improvements was established by the consortium and all partners agreed to implement some of the listed improvements already during the remaining month of the project. The main improvements achieved before the 31st of October 2014 were:
- Integration of a stalk tracking system in the detection system
- Modification of the harvesting tool toward more gentle gripping
The final meeting of the AmLight project took place on the 7th of October at STRAUSS facilities. Besides technical and management related presentations the AmLight prototype was also demonstrated to all partners. Another highlight of the final meeting was the presentation of the project movie, which illustrates in an attractive way the functions of the AmLight harvesting machine. The movie has been uploaded to the AmLight website www.amlight.eu.
Project Results:
The S&T developments of the AmLight project were distributed into 7 work packages. A short description of the work performed and the results achieved in each work package is shown below:
Work Package 1 – System specification
Main results:
- A document in which all specification for this project are embraced
Work performed:
The specification work of the work package was subdivided into three areas
a) Specification of the visual (detection) system
The aim of this area was to determine the best hardware solution (camera + sensors) to fulfil the need of detection in ambient light, and which would allow for efficient development of an IPP (Image Processing Program). Resulting of discussions was to use a stereo camera for taking images.
b) Definition of the system performance
Mainly the end-users in the project were defining performance requirements in terms of field properties and yield/harvesting expectations. Hence this area of specification had three main parts of characterisation (i) the cultivation of asparagus, (ii) the manual harvesting efficiency, and (iii) the expected machine efficiency.
c) Definition of mechanical motion characteristics
Based on the data of the system performance the partners pre-defined potential motions, which could lead to gentle cutting at high speed. One of the key conclusions was to provide a redundant system, means several fast kinematics in row in order to be able to harvest even two (or more) asparagus stalks, which grow close to each other.
Work Package 2 – Development of the detection system
Main results:
- Development of a camera assembly, to take asparagus images in ambient light
- Development of an IPP, to determine/calculate coordinates of asparagus stalks
- Development of an integrated operator interface, for machine optimisation
- A comprehensive data-base of asparagus images
Work performed:
During the first harvesting season of the project (2013) a purchased stereo camera was fitted to push trolley. This assembly was used to calibrate the camera and to take some first images of asparagus under real conditions, which were needed for the development of the IPP.
After the harvesting season 2013 two main developments for the detection system were made:
a) The development of the environmental awareness module
The best solution of a 3D detection system capable of dealing with the ambient light was the use of a CMOS sensor based stereo camera because it supports native wide dynamic range (WDR). WDR is a feature that handles a wide range of lighting conditions in a scene and in the camera image both the darkest and the brightest parts are taken into account, resulting in more details on both ends. Hence the system can be used in changing light conditions.
b) The development of the IPP
Image Processing was parted into two important parts: the pre-processing part and segmentation part. The raw output (distorted, unrectified, and color coded) from the stereo camera is termed as “raw image”. The debayering algorithm is used to obtain a full-color image. In the next step the white balance adjustment is performed. White balance is the process of removing unrealistic color casts, so that objects which appear white for human eye are rendered white in the acquired stereo image. This step is followed by the camera calibration to extract the metric information from 2D images. Since the acquired stereo images are unrectified, they cannot be directly used to compute the 3D positions.
After the pre-processing is done, the asparagus stalks could be segmented from the environment within the segmentation module. Since the asparagus stalks are green in color, they have to be separated from the other colours in the background. To achieve this, a mixed color conversion is performed.
Feature extraction and foot point detection of the asparagus stalks are performed using the robustly segmented image. Furthermore, the classification of the asparagus stalks is done to decide if they are to be harvested or not.
Work Package 3 – Development of the harvesting mechanism
Main results:
- 3D CAD design of the harvesting tool
- 3D CAD design of the fast kinematic
- Manufacturing of two complete harvesting mechanisms
Work performed:
In the very beginning of the project several ideas for harvesting tools were built to be used manually. The manual testing of these tools took place in the first harvesting season of the project in 2013. After the harvesting season the most promising tool was then up-scaled and designed for automatic use, and two tools were manufactured.
Also the fast kinematic, was designed in the second period of the project in close cooperation with WP4 (development of the carrier unit) and WP5 (development of the control program).
The function of the fast kinematic is to move the harvesting tool from a standby (waiting) position perpendicular (y-axis) to the driving over the dam of a detected asparagus stalk. When approaching the stalk the fast kinematic will perform a movement which takes the harvesting tool in less than one second a high position down to ground level (where the asparagus stalk is harvested) and back up into the high position. From there it will move the harvesting tool (and the asparagus) to a drop-off position above a collecting belt conveyor. After the drop-off it returns the harvesting tool into the home (stand-by) position.
Y-axis movement – which is a linear track (motor not shown) to be fitted to the carrier unit (see WP4). A slide on the linear track carries the part of the fast kinematic responsible for the z-axis movement.
Z-axis movement – which is a unit fitted to the slide on the linear track. It consists of a motor connected to a pulley. A belt is connecting the driven plus to additional pulleys. Fitted to the belt is an aluminium construction (bracket) which holds at the bottom ending the harvesting tool (harvesting tool not shown). The aluminium construction is sitting on both sides (left and right) a L shaped bend track. Moving along these track perform on motor activation first a short horizontal linear movement, followed by a cornering downward movement and finally a linear vertical movement down.
In order to reach a high harvesting efficiency two of these fast kinematics were manufactured by UNIHB. The idea of the redundancy is: while one fast kinematic is in operation of harvesting an asparagus stalk the second one can go into position for the next stalk and vice versa.
Work Package 4 – Development of carrier unit
Main results:
- 3D CAD design of the carrier unit
- Carrier unit manufactured
- Calculated and integrated power supply system
Work performed:
Part of the specification (WP1) was to determine the number and strength of motors needed for the prototype. Based on that the power consumption of the entire machine was calculated and batteries as well as transformer designed and provided.
In the second period of the project the entire carrier unit was 3D CAD designed and manufactured. The carrier unit consists of:
A) A robust stainless steel frame
B) Two driven front wheels
C) Two freewheeling, steering back wheels
D) Set of batteries for the power supply
E) Belt conveyer for asparagus collection
F) Brackets to fit both fast kinematics
G) Bracket for control panel
H) Bracket for camera
The carrier unit delivers sufficient space for the integration of a camera at the front part of the unit, a control panel (front right) and a remote panel (back right), as well as two fast kinematics in the centre part of the machine.
After completion of the CAD drawing all machine parts were purchased, produced and/or taken from stock. The successful fabrication of the carrier unit was completed prior to the arrival of the subcomponents.
Work Package 5 – Development of control system
Main results:
- Fully functional PLC program for the control of all machine motions
- User friendly operation through the remote control
Work performed:
The development only started in month 9 of the project. A new version of PLC programming has lately been launched by OMRON (Type: NJ301-1200), which provides faster data transmission between a PC and the PLC. This has been chosen as the best opportunity for the AmLight motion control program.
An integral part of the control program is to establish communication between the IPP (WP2) and the harvesting mechanism (WP3). The control system will process the received data from the asparagus detection system (IPP) to provide the corresponding optimized movement pattern to the mechanism (vehicle and fast kinematic). The detection system will provide the exact position of the asparagus in the dam.
The hardware interface for the AmLight system consists of three control panels: Remote Cabinet, Main Cabinet and Encoder Box.
Main Cabinet: This is the control unit of the AmLight System. It connects all different systems and the PLC inside the Main Cabinet run the software that will makes the whole system runs together. To integrate the Control Panel with the IPP System and the mechanical parts we use the external connectors on the cabinet.
Remote Cabinet: There is a display that will allow the operator to adjust different parameters of the AmLight system such as speed of actuators, off-sets of camera and fast kinematics etc. The display also informs about detected errors, status of different parts of the system such as inputs, drivers, harvesting process. One of the main functions, however, is to choose operation in either manual or automatic mode. In automatic mode the machine is running at a set speed over an asparagus dam, searching for asparagus stalks and performing automatic harvesting cycles. In manual mode the machine can be turned at the end of the dam to the next one and all motions can be activated by push buttons.
Work Package 6 – System integration & functional testing
Main results:
- All sub-components successfully integrated into the carrier unit
- Detection system successfully calibrated for automatic use
- Communication between IPP and PLC established
- All sub-components functional
- Operator manual developed
Work performed:
Work package 6 had three very clear defined and distinguished tasks as follows:
Task 6.1: Assembly of the AmLight prototype
All subcomponents as resulting from work packages 2, 3 and 5 were delivered to STRAUSS facilities for assembly of the AmLight prototype. The work done by STRAUSS, IMIX and the RTDs included:
a) The mechanical integration of all sub components into the carrier unit.
Part of the mechanical integration was (i) the fitting of two fast kinematics into foreseen positions as explained in WP4, (ii) the installation of the camera, and (iii) the mounting of the control panel as well as the remote control.
b) The electrical assembly
The main issue of the electrical work was the wiring of the entire system, means that all actuators (fast kinematic and driving of vehicle), the camera, and the remote panel were wired in the connectors of the main control panel. Also a connection between the control panel and the (battery) power supply was established. Finally all end-switches were for correct and safe movement of the fast kinematic.
Task 6.2: Functional tests of AmLight prototype
After successful assembly of the AmLight prototype the function of all sub components was tested. This included:
• Calibration of the camera. First the off-set from a 0-position of the machine, the height above ground level and the angle of the camera were determined in order to have precise correlations between coordinates of detected asparagus stalk and the positioning of the harvesting tools.
• Encoder read-out. One of the driven front wheels had an encoder installed to measure the distance the harvesting machine would need to travel between asparagus detection and harvesting. The encoder read-out was therefore checked against real distance covered.
• Communication between IPP and PLC. One of the key issues of the functional testing was the establishment of the communication between the IPP and the PC. Here the translation program needed to be modified.
Function of fast kinematic. The functional testing of the fast kinematic started with determining the correct off-set from the 0-position of the machine. After that motors were activated the check the movement of the harvesting tool in Y- and Z-direction. Finally the speed was set to the maximum possible without vibrations. First the function of the fast kinematic were tested though the push buttons on the remote control in manual mode, later simulated positions were sent from the IPP to also get the function tested in automatic mode.
After successful completion of all functional tests the AmLight prototype was ready to be tested under real conditions on an asparagus field.
Task 6.3: Development of user manual
The user manual which has been developed along the work of WP 6 is a detailed description of menus available through the remote control panel, which is used to set-up and operate the harvesting machine. Through the user manual an operator can learn how to
• Set up motors
• Set up the encoder
• Change off-sets
• Deal with errors
• Read out asparagus positions
The complete user manual is available on the AmLight webpage and has been delivered through the document D6.2.
Work Package 7 – Field tests – operator training – evaluation – optimisation
Main results:
The main results of field tests during the harvesting season 2014 and machine optimisation after the season the following results have been achieved:
Asparagus detection success rate up to 70 %
Detection precision at ± 3 cm
Harvesting speed at about 50 % of expected maximum
A comprehensive list of still required optimisations
Work performed:
During the harvesting season in 2014 the consortium had two main tasks to reach:
a) Field tests to determine the performance of the prototype under real conditions
On different fields of end-user WERNER on three full days, each day having a specific task, as follows:
04.06.2014 – Verification of the asparagus detection system under real outdoor conditions.
10.06.2014 – Determination of the precision of positions calculated by the IPP.
19.06.2014 – Performance of the fast kinematic and the harvesting tool.
After the first and the second day of tests the IPP was improved according to the information gained by the operator training.
The performance tests of the fast kinematic have revealed two technical problems: (i) too slow image processing and (ii) movement of the carrier unit not straight-lined enough. Both problems in combination caused that no real harvesting tests could be done. To overcome this problem the consortium decided to have an artificial dam built outside STRAUSS facilities under better controllable conditions for the carrier unit.
b) Operator training to optimise the IPP
The operator training did not have the aim to train an operator, but to get information of harvesting experts (the operator of the future) to train the detection system. As mentioned above the operator training was performed during the first and second test days and had the aim to improve the correct asparagus identification by the detection system. In praxis it meant that the expert was identifying all asparagus stalks which should have been detected and harvested by the prototype. Images of those stalks were saved in a data base for later optimisation of the IPP. In the other direction the expert was also identifying those asparagus stalks which have been detected incorrectly. These images were also saved in a different data base for later analyses.
The main achievement of both testing days together with a harvesting expert was a large number of images which clearly identified asparagus stalks either ready for harvesting or not. With these image data the IPP could even be optimised without going back to a filed.
c) Test data evaluation and machine optimisation
At the end of the field tests and the operator training all test data were evaluated and the results discussed by all project partners. The outcome of the test data evaluation was a table, in which for each machine component (detection, control program, vehicle, fast kinematic and harvesting tool) all results/achievements were qualified and categorised into:
Acceptable – No need for optimisation during AmLight
Moderate – Optimisation should be considered during and/or after AmLight
Bad – Optimisation indispensable. Unclear what can be still achieved during AmLight
This table was the bases for optimisation done after the end of the harvesting season and may also be used for follow-up improvements after the end of the project.
For optimisation purposes the consortium decided to have built a 5 meter long artificial asparagus dam outside STRAUSS facilities. Being outside real conditions for the detection system were given and the carrier unit could run on levelled ground, which was required at this stage of developments to get the fast kinematic and the harvesting tool tested and optimised.
In total five sessions for testing the fast kinematic and any optimisations to the entire AmLight prototype took place during the summer and late summer time of 2014, resulting in the following:
• At a speed of 0.5 m/s about 70% of all asparagus stalks can successfully be detected and only one position with a precision of ±3cm be sent to the PLC. However, this must be confirmed by long-term real field tests.
• The principle of the motion of the fast kinematic with sufficient precision has been proven to be functional. However, to use the system at high speed the firmness must be improved.
• A redeveloped version of the harvesting tool has finally shown the potential to be successful; however it has never been tested under real field conditions.
The AmLight partners concluded at the end of the project that the prototype has reached pre-competitive status. The main components (asparagus detection, fast kinematic, harvesting tool, control program) have been proven to be functional. All systems, however, need to be improved and intensive field testing and demonstration is indispensable for any future commercial success.
Potential Impact:
4.1 Potential impact
Once the AmLight harvesting will have reached commercial status it offers great potential impact to many areas as:
a) Improved working conditions for harvesting staff
Green asparagus is cultivated in long rows at ground level or low dams and the stalks are usually cut just above or under the soil. Harvesting asparagus means that the staffs have to work all days in an uncomfortable and unhealthy bend-down position. With the application of the AmLight machine to green asparagus harvesting people will get trained to operate a machine which will then do the hard work. Hence the introduction of AmLight is offering better working conditions and thus less health problems, as well as an increase of skills for people who operate the machine.
b) Availability of green asparagus at affordable costs
Asparagus is one of the oldest vegetables crops cultivated by mankind as food. It is said to have a number of health benefits as it is low in calories, contains no cholesterol and is very low in sodium. It is a good source of folic acid, potassium, dietary fibre, vitamin C, carotene and rutin.
On the back side of the coin, fresh green asparagus is also one of the most expensive vegetables. Today harvesting is done by seasonal workers. Usually one worker is needed per hectare and season putting the harvesting cost to approx. 0.7 €/kg, which presents about 1/3 of the total production cost. The increasing labour costs (especially the start of minimum wages in many European countries) will even increase the costs of green asparagus in the coming years.
In order to maintain fresh green asparagus supply to a large share of population and at affordable costs, farmers are forced to optimize their harvesting process and to introduce harvesting aids and/or fully automatic harvesting solutions. In this direction AmLight will help to ensure that European habitant will have access to one of the healthiest vegetables produced on European fields.
c) Increased competitiveness to European asparagus farmers
The European production of green asparagus has increased to making Europe the 4th largest producer of green asparagus after China, Peru and the US in the world. Some 90% of the asparagus consumed within the EU27 is consumed as fresh product, mainly during March and June, the main harvesting season. More than 95 % of this is grown in Europe, the main countries being Spain, Italy, Germany and France.
However, despite the positive trend European green asparagus farmer are also facing a growing competition form asparagus imports from the above named countries, from where ‘fresh’ (chilled in containers during shipment) asparagus can be supplied at low cost than European farmers could produce it. As mentioned above already this situation is forcing the European asparagus producer to maximise the automation level of their processes. While process steps like cleaning, cutting, sorting and partly the packaging have already reached an advanced automation level the harvesting remained a fully manually done job. The AmLight developments will help closing that gap in automation and hence supporting asparagus farmers to remain competitive.
4.1 AmLight dissemination activities
In the duration of the AmLight project to following dissemination activities have been undertaken:
a) The press release
In the first month of the project a press release has been launched through the network of partner UNIHB. The content of the press release was a general description of the planned work, the objectives of the project and the names of participating partners.
b) The project website
The AmLight website has been established in the second month of the project – www.amlight.eu.
All information produced in the project has been uploaded to the website. Confidential information is password-protected, other areas are public accessible. The dissemination material developed in the project (see below) are free downloadable from the website.
c) Dissemination Flyer / expoSE
A dissemination flyer and two posters have been produced in October 2013. The reason for the early-in-the-project production was that a chance was provided by STRAUSS to disseminate AmLight on their stand at the expoSE exhibition in Karlsruhe (Germany).
d) Dissemination video
A dissemination video has been developed towards the end of the project showing in an attractive way the machine components and its functions. Direct link to the video: http://www.amlight.eu/results.php
List of Websites:
www.amlight.eu
Strauss Verpackungsmachinen GmbH
Email: info@strauss-pack.com
Phone: +49 4161 14220
