Periodic Reporting for period 2 - CANOPIES (A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems)
Reporting period: 2022-07-01 to 2023-12-31
The Agri-Food sector is one of the most important providers of livelihoods and is central to the fight against poverty and hunger. Overall, three major challenges can be currently identified for the agri-food sector: i) feeding a growing population; ii) providing a livelihood for farmers and farmworkers, and iii) protecting the environment. Notably, precision agriculture (PA), which is a modern whole-farm management concept based on several technologies ranging from remote sensing and proximal data gathering to automation and robotics, represents a viable solution for addressing such triplet of challenges.
In CANOPIES we propose a novel integrated system where farmworkers can safely collaborate with multirobot systems in PA settings. In our vision, we consider a PA permanent crop setting where human workers interact with two different kinds of robotic platforms, namely i) farming robots which are dedicated to the execution of agronomic tasks such as harvesting the fruits or pruning the vines by resorting to dual arm system with actuated torso along with an area with removable boxes; and ii) logistics robots, which are focused on the execution of logistics tasks, such as the transportation of boxes of harvested grapes or removed branches, being equipped with a mechanism for exchanging boxes along with a storage area for carrying boxes.
The CANOPIES’s paradigm will simplify the management of several PA settings through a collaborative execution of several agronomic operations, leading to a practical and cost-effective solution in a critical sector for sustainability of people like agri-food. Several societal benefits are expected from such new farming paradigm, ranging from a reduction of the gender gap (by lessening typical drudgery of farming operations) to an increase of the profit margin (by introducing cutting-edge technological solutions) Finally, we expect the CANOPIES collaborative paradigm to be naturally accepted by farmers and farmworkers as they will not feel “replaced” by the machines, but rather supported by them in their action.
To attain CANOPIES vision, the project focuses on the following three macro-objectives:
i) Human-Robot Interaction (HRI): Develop novel perception and collaborative techniques to promote and facilitate the interaction between the human workers and the robotic platforms, first in simulation and then on the real robot system.
ii) Human-Robot Collaboration (HRC) and Multi-Robot Collaboration (MRC) – Development of techniques and control strategies for collaborative implementation of agronomic operations.
iii) PA Integrated System – Develop and implement the elements needed for a physical demonstration of the HRI, HRC, and MRC objectives in a permanent crops piloting scenario.
In CANOPIES we propose a novel integrated system where farmworkers can safely collaborate with multirobot systems in PA settings. In our vision, we consider a PA permanent crop setting where human workers interact with two different kinds of robotic platforms, namely i) farming robots which are dedicated to the execution of agronomic tasks such as harvesting the fruits or pruning the vines by resorting to dual arm system with actuated torso along with an area with removable boxes; and ii) logistics robots, which are focused on the execution of logistics tasks, such as the transportation of boxes of harvested grapes or removed branches, being equipped with a mechanism for exchanging boxes along with a storage area for carrying boxes.
The CANOPIES’s paradigm will simplify the management of several PA settings through a collaborative execution of several agronomic operations, leading to a practical and cost-effective solution in a critical sector for sustainability of people like agri-food. Several societal benefits are expected from such new farming paradigm, ranging from a reduction of the gender gap (by lessening typical drudgery of farming operations) to an increase of the profit margin (by introducing cutting-edge technological solutions) Finally, we expect the CANOPIES collaborative paradigm to be naturally accepted by farmers and farmworkers as they will not feel “replaced” by the machines, but rather supported by them in their action.
To attain CANOPIES vision, the project focuses on the following three macro-objectives:
i) Human-Robot Interaction (HRI): Develop novel perception and collaborative techniques to promote and facilitate the interaction between the human workers and the robotic platforms, first in simulation and then on the real robot system.
ii) Human-Robot Collaboration (HRC) and Multi-Robot Collaboration (MRC) – Development of techniques and control strategies for collaborative implementation of agronomic operations.
iii) PA Integrated System – Develop and implement the elements needed for a physical demonstration of the HRI, HRC, and MRC objectives in a permanent crops piloting scenario.
From the beginning of the project to the end of RP1, the Consortium has mostly focused on:
i. The definition of the requirement, the specification, and the benchmark for the design of the proposed integrated system;
ii. The definition of the specifications, the design, and the integration of the “basic common structure” for the two robotic prototypes;
iii. The definition of the specifications, the design, and the implementation of the first version of: i) the agronomic dual arms system with actuated torso (with one arm only); ii) the end-effector prototype for harvesting; iii) the box-exchange-mechanism (BEM);
iv. The implementation of the basic functionalities of the mobile base of both logistic and farming robotic prototypes;
v. The implementation of the basic functionalities of the agronomic dual-arms system of the farming robotic prototype;
vi. The implementation of agronomic-oriented perception functionalities;
vii. The implementation of basic functionalities for HRI and HRC;
viii. The execution of field data collection campaigns for algorithm design and field validation;
ix. The implementation of the core functionalities of the Virtual-Reality architecture;
x. The development of the core methodologies for distributed estimation, collaboration, and learning by demonstration in a multi-robot and multi-human scenario;
xi. The dissemination and promotion of the project activities.
From the end of RP1 to the end of RP2, the Consortium has mostly focused on:
i. The development of prototypes 1.1 (single-arm) and 1.2 (dual-arm) for the experimental validation of the methodological achievements concerning perception, manipulation, and human-robot interaction;
ii. The development of the core components of the BEM;
iii. The completion of the implementation of the basic functionalities of the agronomic dual-arm system of the farming robot, ranging from the kinematic control strategies for single and dual-arm manipulation to the control strategies for safe HRI;
iv. The completion of the implementation of the functionalities for HRI and HRC, such as the development of a human body prediction method;
v. The completion of the implementation of a feature-complete simulation environment, covering architecture and software realization;
vi. The completion and field validation of the BEM coordination methodologies, and the advancement of the methodologies for distributed estimation, collaboration, and learning by demonstration;
vii. The dissemination and promotion of the project activities.
i. The definition of the requirement, the specification, and the benchmark for the design of the proposed integrated system;
ii. The definition of the specifications, the design, and the integration of the “basic common structure” for the two robotic prototypes;
iii. The definition of the specifications, the design, and the implementation of the first version of: i) the agronomic dual arms system with actuated torso (with one arm only); ii) the end-effector prototype for harvesting; iii) the box-exchange-mechanism (BEM);
iv. The implementation of the basic functionalities of the mobile base of both logistic and farming robotic prototypes;
v. The implementation of the basic functionalities of the agronomic dual-arms system of the farming robotic prototype;
vi. The implementation of agronomic-oriented perception functionalities;
vii. The implementation of basic functionalities for HRI and HRC;
viii. The execution of field data collection campaigns for algorithm design and field validation;
ix. The implementation of the core functionalities of the Virtual-Reality architecture;
x. The development of the core methodologies for distributed estimation, collaboration, and learning by demonstration in a multi-robot and multi-human scenario;
xi. The dissemination and promotion of the project activities.
From the end of RP1 to the end of RP2, the Consortium has mostly focused on:
i. The development of prototypes 1.1 (single-arm) and 1.2 (dual-arm) for the experimental validation of the methodological achievements concerning perception, manipulation, and human-robot interaction;
ii. The development of the core components of the BEM;
iii. The completion of the implementation of the basic functionalities of the agronomic dual-arm system of the farming robot, ranging from the kinematic control strategies for single and dual-arm manipulation to the control strategies for safe HRI;
iv. The completion of the implementation of the functionalities for HRI and HRC, such as the development of a human body prediction method;
v. The completion of the implementation of a feature-complete simulation environment, covering architecture and software realization;
vi. The completion and field validation of the BEM coordination methodologies, and the advancement of the methodologies for distributed estimation, collaboration, and learning by demonstration;
vii. The dissemination and promotion of the project activities.
CANOPIES is expected to advance the state-of-the-art with respect to several core technologies, contribute to the wider integration and acceptance of agri-food technologies and promote industrial and market benefits that will emerge from their development.
In particular, CANOPIES aims to contribute by: i) developing novel methodologies for planning collaborative execution of complex tasks between teams of robots and human workers in highly dynamic outdoor environments (HDOEs); ii) developing an innovative Virtual Reality Farming Environment for fast algorithmic prototyping and human-in-the-loop algorithmic design; iii) developing novel methodologies for human-robot awareness, context awareness, and human-robot communication for enhanced safety and coexistence within HDOEs; iv) developing novel methodologies for learning by demonstration for increased system adaptability and intuitive usability in HDOEs; and v) by developing an integrated perception system for human-robot interaction, safe navigation, and agronomic-oriented perception in HDOEs.
Furthermore, CANOPIES aims to contribute to the Agri-Food Sector from different standpoints: i) technological: by reducing the gap in the development of intelligent robotic solutions for permanent crops compared to other agronomic sectors such as field crops. This will be achieved by introducing a novel concept of farming robots, where we leverage an effective interaction with the human workers to potentially reduce the complexity of the technological design; ii) economic: by promoting the development of autonomous robotics solutions which may have lower technological complexity by resorting to the collaboration with human workers; iii) societal: contributing towards an improvement of the quality of life of EU farmers, by promoting a better education driven by the interaction with cutting-edge technological solutions, ensuring a better income due to the higher margin of profit expected by the deployment of such an integrated system, and making the agri-food sector appealing also for relatively young people thus acting against the lack of labor force and entrepreneurship in this sector.
In particular, CANOPIES aims to contribute by: i) developing novel methodologies for planning collaborative execution of complex tasks between teams of robots and human workers in highly dynamic outdoor environments (HDOEs); ii) developing an innovative Virtual Reality Farming Environment for fast algorithmic prototyping and human-in-the-loop algorithmic design; iii) developing novel methodologies for human-robot awareness, context awareness, and human-robot communication for enhanced safety and coexistence within HDOEs; iv) developing novel methodologies for learning by demonstration for increased system adaptability and intuitive usability in HDOEs; and v) by developing an integrated perception system for human-robot interaction, safe navigation, and agronomic-oriented perception in HDOEs.
Furthermore, CANOPIES aims to contribute to the Agri-Food Sector from different standpoints: i) technological: by reducing the gap in the development of intelligent robotic solutions for permanent crops compared to other agronomic sectors such as field crops. This will be achieved by introducing a novel concept of farming robots, where we leverage an effective interaction with the human workers to potentially reduce the complexity of the technological design; ii) economic: by promoting the development of autonomous robotics solutions which may have lower technological complexity by resorting to the collaboration with human workers; iii) societal: contributing towards an improvement of the quality of life of EU farmers, by promoting a better education driven by the interaction with cutting-edge technological solutions, ensuring a better income due to the higher margin of profit expected by the deployment of such an integrated system, and making the agri-food sector appealing also for relatively young people thus acting against the lack of labor force and entrepreneurship in this sector.