Periodic Reporting for period 2 - BACCHUS (MoBile Robotic PlAtforms for ACtive InspeCtion and Harvesting in AgricUltural AreaS)
Periodo di rendicontazione: 2021-07-01 al 2022-09-30
For BACCHUS to successfully realize its vision, a few scientific and technological objectives have been set throughout the project duration. These are listed in the following points:
1) To develop autonomous mobile robots that will be capable to explore the agricultural area and assess the crops
2) To develop a bi-manual robotic platform that will be capable to harvest the crops by using traditional “hand keeping” procedures
3) To develop an integrated grasp planning framework to plan and optimize grippers for the several type of harvesting applications
4) DSS for optimising operational capacity and planning, featuring automatic rapid reconfiguration of inspection and harvesting processes
5) To enable safe autonomous navigation and interaction in heterogeneous agricultural areas
6) To implement mechanisms for automated operation planning and task scheduling
7) Continuous validation and Proof of concept demonstrations
8) BACCHUS system commercialisation and exploitation
Multiple surveys were conducted with winemakers, vineyard managers and vineyard workers in Italy and Greece, presenting the planned use-cases, in order to collect the user-requirements via questionnaires. An initial list of requirements was furnished and the system architecture was analytically defined. The planned use-cases were refined paying more focus on the inspection and harvesting of vertical trellis vineyards. Target KPI's were set.
The mobile dual-arm harvesting platform was designed and built from scratch, taking into account the different requirements from the use-cases. It consists of the main mobile platform with pseudo-omnidirectional wheels, two integrated UR10e arms and several sensors on the platform such as lidar, cameras, security system, battery manager, etc. The platform has been tested in the past two seasons (2021 and 2022) in the Gerovassileiou vineyard in Thessaloniki, performing autonomous navigation and some initial harvesting experiments.
The inspection platform was assembled based on the Thorvals platform, carrying multiple sensors to navigate the field and assess the crops using a hyperspectral camera. In 2022, the platform has done extensive field tests and data collection in Gerovassileiou vinyeard.
Multiple datasets were collected with RGBD sensors, lidars and hyperspectral cameras. New modules were created for the semantic segmentation of the vine tree, to detect the grapes and identify leaves, canes and static obstacles. Probabilistic localization algorithms were defeloped, offering reliable sensor fusion to offer the vest of different localization methods. A topological map consisting of a set of nodes and edges was also developed, defining a high-level structure of the operational environment.
Spectral libraries have been created for the non-catastrophic measurement of the sugar level of grapes, based on near-infrared sensing at different varieties.
Research for controlling the motion of the dual-arm harvesting platform was performed, towards reaching a region of interest of grapes ready to be harvested, while optimizing the view of the stalk. A visual servoing algorithm was developed to reach the stalk for cutting while keeping it centered on the camera’s view. The two arms are coordinating their motion throughout the operation, for achieving an optimal relative pose of the two end-effectors before harvesting.
To address operation planning and task scheduling for all operations in the field, the corresponding planning algorithms were developed for the operations such as, inspection, yield prediction, harvesting, etc., as well as by interconnecting the operation planner and scheduler with the system’s HMI interface for interacting with the users, i.e. the vineyard supervisor, the human workers, etc
Most of the modules have been individually verified and have been integrated in the the robot platforms. Some initial integrated test have been performed in the vineyards. Our goal is for the inspection platform to navigate the field autonomously, creating a crop map, estimating the yield and providing the winemakers with useful information. When the time for harvesting has come, the harvesting platform will utilize the crop map and harvest the grapes of the highest quality.
The expected potential impact from the project is highlighted as follows:
- Advancement of robotic and AI capabilities such as in mechatronics, perception, cognition and control of dual-arm mobile manipulation, that will be able to demonstrate their effectiveness in the challenging vineyard environment.
- The introduction of autonomous robotic services for supporting vineyard management towards farming efficiency and quality production.