Periodic Reporting for period 2 - BACCHUS (MoBile Robotic PlAtforms for ACtive InspeCtion and Harvesting in AgricUltural AreaS)
Período documentado: 2021-07-01 hasta 2022-09-30
BACCHUS is a 45-month project that aims to develop an autonomous robotic ecosystem, utilizing mobile platforms and manipulators for intelligent inspection and selective harvesting of high value crops. The two BACCHUS robots will be able to navigate the vineyard inspecting the crops and collecting data through multiple sensors, performing bi-manual harvesting operations, employing additive manufacturing for adjusting the robot gripper to the geometry of the different crops, and presenting advanced cognitive capabilities and decision-making skills. The robots will eventually take the manual legwork out and reproduce selective harvesting operations.
For BACCHUS to successfully realize its vision, a few scientific and technological objectives have been set throughout the project duration. These are listed in the following points:
1) To develop autonomous mobile robots that will be capable to explore the agricultural area and assess the crops
2) To develop a bi-manual robotic platform that will be capable to harvest the crops by using traditional “hand keeping” procedures
3) To develop an integrated grasp planning framework to plan and optimize grippers for the several type of harvesting applications
4) DSS for optimising operational capacity and planning, featuring automatic rapid reconfiguration of inspection and harvesting processes
5) To enable safe autonomous navigation and interaction in heterogeneous agricultural areas
6) To implement mechanisms for automated operation planning and task scheduling
7) Continuous validation and Proof of concept demonstrations
8) BACCHUS system commercialisation and exploitation
For BACCHUS to successfully realize its vision, a few scientific and technological objectives have been set throughout the project duration. These are listed in the following points:
1) To develop autonomous mobile robots that will be capable to explore the agricultural area and assess the crops
2) To develop a bi-manual robotic platform that will be capable to harvest the crops by using traditional “hand keeping” procedures
3) To develop an integrated grasp planning framework to plan and optimize grippers for the several type of harvesting applications
4) DSS for optimising operational capacity and planning, featuring automatic rapid reconfiguration of inspection and harvesting processes
5) To enable safe autonomous navigation and interaction in heterogeneous agricultural areas
6) To implement mechanisms for automated operation planning and task scheduling
7) Continuous validation and Proof of concept demonstrations
8) BACCHUS system commercialisation and exploitation
The first two and a half years of the project the consortium has made significant progress towards the main objectives. The work performed can be summarized below:
Multiple surveys were conducted with winemakers, vineyard managers and vineyard workers in Italy and Greece, presenting the planned use-cases, in order to collect the user-requirements via questionnaires. An initial list of requirements was furnished and the system architecture was analytically defined. The planned use-cases were refined paying more focus on the inspection and harvesting of vertical trellis vineyards. Target KPI's were set.
The mobile dual-arm harvesting platform was designed and built from scratch, taking into account the different requirements from the use-cases. It consists of the main mobile platform with pseudo-omnidirectional wheels, two integrated UR10e arms and several sensors on the platform such as lidar, cameras, security system, battery manager, etc. The platform has been tested in the past two seasons (2021 and 2022) in the Gerovassileiou vineyard in Thessaloniki, performing autonomous navigation and some initial harvesting experiments.
The inspection platform was assembled based on the Thorvals platform, carrying multiple sensors to navigate the field and assess the crops using a hyperspectral camera. In 2022, the platform has done extensive field tests and data collection in Gerovassileiou vinyeard.
Multiple datasets were collected with RGBD sensors, lidars and hyperspectral cameras. New modules were created for the semantic segmentation of the vine tree, to detect the grapes and identify leaves, canes and static obstacles. Probabilistic localization algorithms were defeloped, offering reliable sensor fusion to offer the vest of different localization methods. A topological map consisting of a set of nodes and edges was also developed, defining a high-level structure of the operational environment.
Spectral libraries have been created for the non-catastrophic measurement of the sugar level of grapes, based on near-infrared sensing at different varieties.
Research for controlling the motion of the dual-arm harvesting platform was performed, towards reaching a region of interest of grapes ready to be harvested, while optimizing the view of the stalk. A visual servoing algorithm was developed to reach the stalk for cutting while keeping it centered on the camera’s view. The two arms are coordinating their motion throughout the operation, for achieving an optimal relative pose of the two end-effectors before harvesting.
To address operation planning and task scheduling for all operations in the field, the corresponding planning algorithms were developed for the operations such as, inspection, yield prediction, harvesting, etc., as well as by interconnecting the operation planner and scheduler with the system’s HMI interface for interacting with the users, i.e. the vineyard supervisor, the human workers, etc
Most of the modules have been individually verified and have been integrated in the the robot platforms. Some initial integrated test have been performed in the vineyards. Our goal is for the inspection platform to navigate the field autonomously, creating a crop map, estimating the yield and providing the winemakers with useful information. When the time for harvesting has come, the harvesting platform will utilize the crop map and harvest the grapes of the highest quality.
Multiple surveys were conducted with winemakers, vineyard managers and vineyard workers in Italy and Greece, presenting the planned use-cases, in order to collect the user-requirements via questionnaires. An initial list of requirements was furnished and the system architecture was analytically defined. The planned use-cases were refined paying more focus on the inspection and harvesting of vertical trellis vineyards. Target KPI's were set.
The mobile dual-arm harvesting platform was designed and built from scratch, taking into account the different requirements from the use-cases. It consists of the main mobile platform with pseudo-omnidirectional wheels, two integrated UR10e arms and several sensors on the platform such as lidar, cameras, security system, battery manager, etc. The platform has been tested in the past two seasons (2021 and 2022) in the Gerovassileiou vineyard in Thessaloniki, performing autonomous navigation and some initial harvesting experiments.
The inspection platform was assembled based on the Thorvals platform, carrying multiple sensors to navigate the field and assess the crops using a hyperspectral camera. In 2022, the platform has done extensive field tests and data collection in Gerovassileiou vinyeard.
Multiple datasets were collected with RGBD sensors, lidars and hyperspectral cameras. New modules were created for the semantic segmentation of the vine tree, to detect the grapes and identify leaves, canes and static obstacles. Probabilistic localization algorithms were defeloped, offering reliable sensor fusion to offer the vest of different localization methods. A topological map consisting of a set of nodes and edges was also developed, defining a high-level structure of the operational environment.
Spectral libraries have been created for the non-catastrophic measurement of the sugar level of grapes, based on near-infrared sensing at different varieties.
Research for controlling the motion of the dual-arm harvesting platform was performed, towards reaching a region of interest of grapes ready to be harvested, while optimizing the view of the stalk. A visual servoing algorithm was developed to reach the stalk for cutting while keeping it centered on the camera’s view. The two arms are coordinating their motion throughout the operation, for achieving an optimal relative pose of the two end-effectors before harvesting.
To address operation planning and task scheduling for all operations in the field, the corresponding planning algorithms were developed for the operations such as, inspection, yield prediction, harvesting, etc., as well as by interconnecting the operation planner and scheduler with the system’s HMI interface for interacting with the users, i.e. the vineyard supervisor, the human workers, etc
Most of the modules have been individually verified and have been integrated in the the robot platforms. Some initial integrated test have been performed in the vineyards. Our goal is for the inspection platform to navigate the field autonomously, creating a crop map, estimating the yield and providing the winemakers with useful information. When the time for harvesting has come, the harvesting platform will utilize the crop map and harvest the grapes of the highest quality.
From the beginning of the projects, the BACCHUS partners have produced 21 open-access scientific publications in international conferences and journals. The progress made by the project beyond the state of the art is highlighted, among others, in the control of bi-manual tasks, grasping in clutter, SLAM, topological mapping, weed detection, scene semantic segmentation, non destructive estimation of grape ripeness
The expected potential impact from the project is highlighted as follows:
- Advancement of robotic and AI capabilities such as in mechatronics, perception, cognition and control of dual-arm mobile manipulation, that will be able to demonstrate their effectiveness in the challenging vineyard environment.
- The introduction of autonomous robotic services for supporting vineyard management towards farming efficiency and quality production.
The expected potential impact from the project is highlighted as follows:
- Advancement of robotic and AI capabilities such as in mechatronics, perception, cognition and control of dual-arm mobile manipulation, that will be able to demonstrate their effectiveness in the challenging vineyard environment.
- The introduction of autonomous robotic services for supporting vineyard management towards farming efficiency and quality production.