Periodic Reporting for period 3 - CANOPIES (A Collaborative Paradigm for Human Workers and Multi-Robot Teams in Precision Agriculture Systems)
Reporting period: 2024-01-01 to 2024-12-31
The Agri-Food sector is one of the most important providers of livelihoods and is central to the fight against poverty and hunger. Overall, three major challenges can be currently identified for the agri-food sector: i) feeding a growing population; ii) providing a livelihood for farmers and farmworkers, and iii) protecting the environment. Notably, precision agriculture (PA), which is a modern whole-farm management concept based on several technologies ranging from remote sensing and proximal data gathering to automation and robotics, represents a viable solution for addressing such triplet of challenges.
In CANOPIES we propose a novel integrated system where farmworkers can safely collaborate with multirobot systems in PA settings. In our vision, we consider a PA permanent crop setting where human workers interact with two different kinds of robotic platforms, namely i) farming robots which are dedicated to the execution of agronomic tasks such as harvesting the fruits or pruning the vines by resorting to dual arm system with actuated torso along with an area with removable boxes; and ii) logistics robots, which are focused on the execution of logistics tasks, such as the transportation of boxes of harvested grapes or removed branches, being equipped with a mechanism for exchanging boxes along with a storage area for carrying boxes.
The CANOPIES’s paradigm will simplify the management of several PA settings through a collaborative execution of several agronomic operations, leading to a practical and cost-effective solution in a critical sector for sustainability of people like agri-food. Several societal benefits are expected from such new farming paradigm, ranging from a reduction of the gender gap (by lessening typical drudgery of farming operations) to an increase of the profit margin (by introducing cutting-edge technological solutions) Finally, we expect the CANOPIES collaborative paradigm to be naturally accepted by farmers and farmworkers as they will not feel “replaced” by the machines, but rather supported by them in their action.
To attain CANOPIES vision, the project focuses on the following three macro-objectives:
i) Human-Robot Interaction (HRI): Develop novel perception and collaborative techniques to promote and facilitate the interaction between the human workers and the robotic platforms, first in simulation and then on the real system.
ii) Human-Robot Collaboration (HRC) and Multi-Robot Collaboration (MRC) – Development of strategies for collaborative implementation of agronomic operations.
iii) PA Integrated System – Develop and implement the elements needed for a physical demonstration of the HRI, HRC, and MRC objectives in a permanent crops piloting scenario.
In CANOPIES we propose a novel integrated system where farmworkers can safely collaborate with multirobot systems in PA settings. In our vision, we consider a PA permanent crop setting where human workers interact with two different kinds of robotic platforms, namely i) farming robots which are dedicated to the execution of agronomic tasks such as harvesting the fruits or pruning the vines by resorting to dual arm system with actuated torso along with an area with removable boxes; and ii) logistics robots, which are focused on the execution of logistics tasks, such as the transportation of boxes of harvested grapes or removed branches, being equipped with a mechanism for exchanging boxes along with a storage area for carrying boxes.
The CANOPIES’s paradigm will simplify the management of several PA settings through a collaborative execution of several agronomic operations, leading to a practical and cost-effective solution in a critical sector for sustainability of people like agri-food. Several societal benefits are expected from such new farming paradigm, ranging from a reduction of the gender gap (by lessening typical drudgery of farming operations) to an increase of the profit margin (by introducing cutting-edge technological solutions) Finally, we expect the CANOPIES collaborative paradigm to be naturally accepted by farmers and farmworkers as they will not feel “replaced” by the machines, but rather supported by them in their action.
To attain CANOPIES vision, the project focuses on the following three macro-objectives:
i) Human-Robot Interaction (HRI): Develop novel perception and collaborative techniques to promote and facilitate the interaction between the human workers and the robotic platforms, first in simulation and then on the real system.
ii) Human-Robot Collaboration (HRC) and Multi-Robot Collaboration (MRC) – Development of strategies for collaborative implementation of agronomic operations.
iii) PA Integrated System – Develop and implement the elements needed for a physical demonstration of the HRI, HRC, and MRC objectives in a permanent crops piloting scenario.
Activities during RP1:
i. Definition of the requirement, the specification, and the benchmark for the design of the proposed integrated system;
ii. Definition of the specifications, the design, and the integration of the “basic common structure” for the two robotic prototypes;
iii. Definition of the specifications, the design, and the implementation of the first version of: i) the dual-arm system with actuated torso (with one arm only); ii) the end effector prototype for harvesting; iii) the box-exchange-mechanism (BEM);
iv. Implementation of the basic functionalities of the mobile bases;
v. Implementation of the basic functionalities of the dual-arm system of the farming prototype;
vi. Implementation of agronomic-oriented perception functionalities;
vii. Implementation of basic functionalities for HRI and HRC;
viii. Execution of field data collection campaigns for algorithm design and field validation;
ix. Implementation of the core functionalities of the Virtual-Reality architecture;
x. Development of the core methodologies for distributed estimation, collaboration, and learning by demonstration in a multi-robot and multi-human scenario;
xi. Dissemination and promotion of the project activities.
Activities during RP2:
i. Development of prototypes 1.1 (single-arm) and 1.2 (dual-arm) of the farming manipulator;
ii. Development of the core components of the BEM;
iii. Completion of the implementation of the basic functionalities of the dual-arm system, ranging from the kinematic control strategies for single and dual-arm manipulation to the control strategies for safe HRI;
iv. Completion of the implementation of the functionalities for HRI and HRC, such as the development of a human body prediction method;
v. Completion of the implementation of a feature-complete simulation environment, covering architecture and software realization;
vi. Completion and field validation of the BEM coordination methodologies, and advancement of the methods for distributed estimation, collaboration, and learning from demonstration (LfD);
vii. Dissemination and promotion of the project activities.
Activities during RP3:
i. Development of the final version (2.0) of the dual-arm prototype;
ii. Completion of the BEM system with storage unit on the logistic robot;
iii. Field validation of the functionalities of the dual-arm system, ranging from the kinematic to the physical control strategies;
iv. Completion and field validation of the functionalities for HRI and HRC, such as the natural language communication with the human operator;
v. Completion of the implementation of all features in the VR, including the updated BEM module;
vi. Completion and field validation of the BEM coordination methodologies and advancement of the methodologies for distributed estimation and multi-robot coordination;
vii. Completion and validation in VR of the LfD methodologies and advancement of the strategies for multi-robot task planning;
viii. Integration of all the components, ranging from the navigation and manipulation functionalities to the high-level HRI, and field validation of the integrated system.
ix. Dissemination and promotion of the project activities.
i. Definition of the requirement, the specification, and the benchmark for the design of the proposed integrated system;
ii. Definition of the specifications, the design, and the integration of the “basic common structure” for the two robotic prototypes;
iii. Definition of the specifications, the design, and the implementation of the first version of: i) the dual-arm system with actuated torso (with one arm only); ii) the end effector prototype for harvesting; iii) the box-exchange-mechanism (BEM);
iv. Implementation of the basic functionalities of the mobile bases;
v. Implementation of the basic functionalities of the dual-arm system of the farming prototype;
vi. Implementation of agronomic-oriented perception functionalities;
vii. Implementation of basic functionalities for HRI and HRC;
viii. Execution of field data collection campaigns for algorithm design and field validation;
ix. Implementation of the core functionalities of the Virtual-Reality architecture;
x. Development of the core methodologies for distributed estimation, collaboration, and learning by demonstration in a multi-robot and multi-human scenario;
xi. Dissemination and promotion of the project activities.
Activities during RP2:
i. Development of prototypes 1.1 (single-arm) and 1.2 (dual-arm) of the farming manipulator;
ii. Development of the core components of the BEM;
iii. Completion of the implementation of the basic functionalities of the dual-arm system, ranging from the kinematic control strategies for single and dual-arm manipulation to the control strategies for safe HRI;
iv. Completion of the implementation of the functionalities for HRI and HRC, such as the development of a human body prediction method;
v. Completion of the implementation of a feature-complete simulation environment, covering architecture and software realization;
vi. Completion and field validation of the BEM coordination methodologies, and advancement of the methods for distributed estimation, collaboration, and learning from demonstration (LfD);
vii. Dissemination and promotion of the project activities.
Activities during RP3:
i. Development of the final version (2.0) of the dual-arm prototype;
ii. Completion of the BEM system with storage unit on the logistic robot;
iii. Field validation of the functionalities of the dual-arm system, ranging from the kinematic to the physical control strategies;
iv. Completion and field validation of the functionalities for HRI and HRC, such as the natural language communication with the human operator;
v. Completion of the implementation of all features in the VR, including the updated BEM module;
vi. Completion and field validation of the BEM coordination methodologies and advancement of the methodologies for distributed estimation and multi-robot coordination;
vii. Completion and validation in VR of the LfD methodologies and advancement of the strategies for multi-robot task planning;
viii. Integration of all the components, ranging from the navigation and manipulation functionalities to the high-level HRI, and field validation of the integrated system.
ix. Dissemination and promotion of the project activities.
CANOPIES advanced state-of-the-art technology with respect to several core technologies, contributed to the wider integration and acceptance of agri-food technologies, and promoted industrial and market benefits emerged from their development.
CANOPIES contributed by: i) developing novel methodologies for planning collaborative execution of complex tasks between teams of robots and human workers in highly dynamic outdoor environments (HDOEs); ii) developing an innovative Virtual Reality Farming Environment for fast algorithmic prototyping and human-in-the-loop algorithmic design; iii) developing novel methodologies for human-robot awareness, context awareness, and human-robot communication for enhanced safety and coexistence within HDOEs; iv) developing novel methodologies for learning by demonstration for increased system adaptability and intuitive usability in HDOEs; and v) developing an integrated perception system for HRI, safe navigation, and agronomic-oriented perception in HDOEs.
Furthermore, CANOPIES contributed to the Agri-Food Sector from different standpoints: i) technological: by introducing a novel concept of farming robots to effective interaction with human workers; ii) economic: by promoting the development of autonomous robotics solutions with lower technological complexity by resorting to the collaboration with human workers; iii) societal: by working towards an improvement of the quality of life of EU farmers and by promoting a better education driven by the interaction with cutting-edge technological solutions.
CANOPIES contributed by: i) developing novel methodologies for planning collaborative execution of complex tasks between teams of robots and human workers in highly dynamic outdoor environments (HDOEs); ii) developing an innovative Virtual Reality Farming Environment for fast algorithmic prototyping and human-in-the-loop algorithmic design; iii) developing novel methodologies for human-robot awareness, context awareness, and human-robot communication for enhanced safety and coexistence within HDOEs; iv) developing novel methodologies for learning by demonstration for increased system adaptability and intuitive usability in HDOEs; and v) developing an integrated perception system for HRI, safe navigation, and agronomic-oriented perception in HDOEs.
Furthermore, CANOPIES contributed to the Agri-Food Sector from different standpoints: i) technological: by introducing a novel concept of farming robots to effective interaction with human workers; ii) economic: by promoting the development of autonomous robotics solutions with lower technological complexity by resorting to the collaboration with human workers; iii) societal: by working towards an improvement of the quality of life of EU farmers and by promoting a better education driven by the interaction with cutting-edge technological solutions.