Skip to main content
European Commission logo print header

A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems

Periodic Reporting for period 1 - CANOPIES (A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems)

Berichtszeitraum: 2021-01-01 bis 2022-06-30

The Agri-Food sector is one of the most important providers of livelihoods and is central to the fight against poverty and hunger. Overall, three major challenges can be currently identified for the agri-food sector: i) feeding a growing population; ii) providing a livelihood for farmers and farmworkers, and iii) protecting the environment. Notably, precision agriculture (PA), which is a modern whole-farm management concept based on several technologies ranging from remote sensing and proximal data gathering to automation and robotics, represents a viable solution for addressing such triplet of challenges.

In CANOPIES we propose a novel integrated system where farmworkers can safely collaborate with multirobot systems in PA settings. In our vision, we consider a PA permanent crop setting where human workers interact with two different kinds of robotic platforms, namely i) farming robots which are dedicated to the execution of agronomic tasks such as harvesting the fruits or pruning the vines by resorting to dual arm system with actuated torso along with an area with removable boxes; and ii) logistics robots, which are focused on the execution of logistics tasks, such as the transportation of boxes of harvested grapes or removed branches, being equipped with a mechanism for exchanging boxes along with a storage area for carrying boxes.

The CANOPIES’s paradigm will simplify the management of several PA settings through a collaborative execution of several agronomic operations, leading to a practical and cost-effective solution in a critical sector for sustainability of people like agri-food. Several societal benefits are expected from such new farming paradigm, ranging from a reduction of the gender gap (by lessening typical drudgery of farming operations) to an increase of the profit margin (by introducing cutting-edge technological solutions) Finally, we expect the CANOPIES collaborative paradigm to be naturally accepted by farmers and farmworkers as they will not feel “replaced” by the machines, but rather supported by them in their action.

To attain CANOPIES vision, the project focuses on the following three macro-objectives:
i) Human-Robot Interaction (HRI): Develop novel perception and collaborative techniques to promote and facilitate the interaction between the human workers and the robotic platforms, first in simulation and then on the real robot system.
ii) Human-Robot Collaboration (HRC) and Multi-Robot Collaboration (MRC) – Development of techniques and control strategies for collaborative implementation of agronomic operations.
iii) PA Integrated System – Develop and implement the elements needed for a physical demonstration of the HRI, HRC, and MRC objectives in a permanent crops piloting scenario.
From the beginning of the project to the end of the first reporting period, the Consortium has mostly focused on:
i. The definition of the requirement, the specification, and the benchmark for the design of the proposed integrated system, ranging from the agronomic specifications to the dimensioning of the real-world table-grape scenario for validating the project outcome;
ii. The definition of the specifications, the design, and the integration of the “basic common structure” for the two robotic prototypes;
iii. The definition of the specifications, the design, and the implementation of the first version of: i) the agronomic dual arms system with actuated torso (implementation with one arm only); ii) the end-effector prototype for harvesting; iii) the box-exchange-mechanism (BEM);
iv. The implementation of the basic functionalities of the mobile base of both logistic and farming robotic prototypes, ranging from the development of the low-level kinematic control and the development of the basic navigation functionalities to the development of the coordinated maneuvering strategy for the BEM;
v. The implementation of the basic functionalities of the agronomic dual-arms system of the farming robotic prototype, ranging from the development of the kinematic control strategies for single and dual-arm manipulation to the development of the control strategies for safe HRI;
vi. The implementation of agronomic-oriented perception functionalities, ranging from a pseudo-labeling system for the detection, segmentation and tracking of grape bunches to a method for grape and peduncle localization for harvesting;
vii. The implementation of basic functionalities for HRI and HRC, such as the development of a method for the trajectory prediction of human body motion;
viii. The execution of field data collection campaigns for algorithm design and field validation;
ix. The implementation of the core-functionalities of the Virtual-Reality architecture including the virtual vineyard environment, the virtual robot model, and the virtual main sensors, and its integration with the ROS framework;
x. The development of the core methodologies for distributed estimation, collaboration and learning by demonstration in a multi-robot and multi-human scenario;
xi. The dissemination and promotion of the project actives by: a) development of the corporative image and the logo for the project, b) development of website, c) development of the communication plan; d) creation and maintenance of the social network accounts.
CANOPIES is expected to advance the state-of-the-art with respect to several core technologies, contribute to the wider integration and acceptance of agri-food technologies and promote industrial and market benefits that will emerge from their development.

In particular, CANOPIES aims to contribute by: i) developing novel methodologies for planning collaborative execution of complex tasks between teams of robots and human workers in highly dynamic outdoor environments (HDOEs); ii) developing an innovative Virtual Reality Farming Environment for fast algorithmic prototyping and human-in-the-loop algorithmic design; iii) developing novel methodologies for human-robot awareness, context awareness, and human-robot communication for enhanced safety and coexistence within HDOEs; iv) developing novel methodologies for learning by demonstration for increased system adaptability and intuitive usability in HDOEs; and v) by developing an integrated perception system for human-robot interaction, safe navigation, and agronomic-oriented perception in HDOEs.

Furthermore, CANOPIES aims to contribute to the Agri-Food Sector from different standpoints: i) technological: by reducing the gap in the development of intelligent robotic solutions for permanent crops compared to other agronomic sectors such as field crops. This will be achieved by introducing a novel concept of farming robots, where we leverage an effective interaction with the human workers to potentially reduce the complexity of the technological design; ii) economic: by promoting the development of autonomous robotics solutions which may have lower technological complexity by resorting to the collaboration with human workers; iii) societal: contributing towards an improvement of the quality of life of EU farmers, by promoting a better education driven by the interaction with cutting-edge technological solutions, ensuring a better income due to the higher margin of profit expected by the deployment of such an integrated system, and making the agri-food sector appealing also for relatively young people thus acting against the lack of labor force and entrepreneurship in this sector.