Skip to main content
European Commission logo
English English
CORDIS - EU research results
CORDIS
CORDIS Web 30th anniversary CORDIS Web 30th anniversary

Metrological Evaluation and Testing of Robots in International CompetitionS

Deliverables

Updated methodological framework (including proven evaluation plans for the four PAs)

This deliverable concatenates updated evaluation plans in the four PAs.

Methodological framework for acceptability assessment in the four PAs

This deliverable provides a global approach and specific implementations for each PA for the acceptability analysis of robots participating in the various METRICS competitions including relevant evaluation criteria and a methodology to establish the link between technical metrics agricultural robot site throughput production rate of an agile industrial robot consumption of electricity and other consumables etc and costs associated with the implementation of these robots degree of supervision required staff training to be provided estimates of savings achieved cost associated with this task before and after robot deployment and the societal and ethical impact human risk impact on employment qualification environmental impact etc

Overview of test infrastructures for robotic competitions and requirements for a sustainable activity in Europe

The deliverable contains the inventory and classification of testbeds that are available in Europe for the evaluation of robots in the four PAs their modality of access geographic accessibility structure type physical or digital It concerns both the existing infrastructures and the testbeds that will be designed for the needs of the project It identifies the potential lacks

RAMI Evaluation Plan

The evaluation plan includes the definition of scenarios episode task and functional benchmarks and their associated execution and assessment procedures It takes into account scientific economic ethical legal and safety aspects as well as inputs from stakeholders

Good practice guide for the organisation of evaluation campaigns

The good practice guide for the organisation of evaluation campaigns includes the organisational process based on a dryrun followed by official evaluation campaigns instructions for the fair treatment of participants communication and awarenessraising actions to be carried out This deliverable also includes ethical guidelines for the definition of the evaluation plan and the implementation of the competition

Mapping of the robotic competitions stakeholders' network in Europe

This deliverable presents an exhaustive mapping of the robotic competition stakeholders in Europe and identifies the potential gaps between demands and offer.

ACRE Evaluation Plan

The evaluation plan includes the definition of scenarios episode task and functional benchmarks and their associated execution and assessment procedures It takes into account scientific economic ethical legal and safety aspects as well as inputs from stakeholders

HEART-MET Evaluation Plan

The evaluation plan includes the definition of scenarios episode task and functional benchmarks and their associated execution and assessment procedures It takes into account scientific economic ethical legal and safety aspects as well as inputs from stakeholders

ADAPT Evaluation Plan

The evaluation plan includes the definition of scenarios episode task and functional benchmarks and their associated execution and assessment procedures It takes into account scientific economic ethical legal and safety aspects as well as inputs from stakeholders

Preliminary methodological framework / Evaluation plan template

This deliverable contains an evaluation plan template for the METRICS competitions including a methodology for the evaluation protocol the evaluation tasks the metrics used the test facilities test databases participation rules and operating procedures capitalising on many campaigns organised by METRICS partners in the past

METRICS website

This deliverable contains the project website which includes a page for each competition that presents news videosphoto gallery of meetings and evaluation campaigns webinars press releases related to the competition

Publications

Field campaign and experimental design for robot performance evaluation (ACRE 2023)

Author(s): Sofia Matilde Luglio; Mino Sportelli; Christian Frasconi; Marco Fontanelli; Matteo Matteucci; Giulio Fontana; Enrico Piazza; Davide Facchinetti
Published in: 2023 IEEE International Workshop on Metrology for Agriculture and Forestry (MetroAgriFor), Issue 2023 IEEE International Workshop on Metrology for Agriculture and Forestry (MetroAgriFor), 2024
Publisher: 2023 IEEE International Workshop on Metrology for Agriculture and Forestry (MetroAgriFor)
DOI: 10.1109/metroagrifor58484.2023.10424292

Quantitative Benchmarking in Agricultural Robotics

Author(s): Riccardo Bertoglio-Giulio Fontana-Matteo Matteucci-Davide Facchinetti-Stefano Santoro
Published in: 2020
Publisher: Second Italian Conference on Robotics and Intelligent Machines Program - I-RIM

On the Design of the Agri-Food Competition for Robot Evaluation (ACRE)

Author(s): Riccardo Bertoglio, Giulio Fontana, Matteo Matteucci, Davide Facchinetti, Michel Berducat, Daniel Boffety
Published in: 2021 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), 2021, Page(s) 161-166, ISBN 978-1-6654-3198-9
Publisher: IEEE
DOI: 10.1109/icarsc52212.2021.9429792

AI evaluation campaigns during robotics competitions: the METRICS paradigm

Author(s): Guillaume Avrin-Virginie Barbosa-Agnes Delaborde
Published in: 2020
Publisher: 1st International Workshop on Evaluating Progress in Artificial Intelligence - EPAI 2020 In conjunction with the 24th European Conference on Artificial Intelligence - ECAI 2020

Robustness of Deep Learning Methods for Occluded Object Detection-A Study Introducing a Novel Occlusion Dataset

Author(s): Ziling Wu; Armaghan Moemeni; Simon Castle-Green; Praminda Caleb-Solly
Published in: 2023
Publisher: IEEE
DOI: 10.1109/ijcnn54540.2023.10191368

The ACRE Crop-Weed Dataset for Benchmarking Weed Detection Models on Maize and Beans Fields

Author(s): Riccardo Bertoglio, Eli Spizzichino, Anne Kalouguine, Giuliano Vitali and Matteo Matteucci
Published in: 2024
Publisher: Deep Learning for Sustainable Precision Agriculture (DLSPA)
DOI: 10.13140/rg.2.2.10507.05927

Instructing Hierarchical Tasks to Robots by Verbal Commands

Author(s): P. Telkes, A. Angleraud, R. Pieters
Published in: IEEE/SICE International Symposium on System Integration, 2024
Publisher: IEEE

Towards Multimodal Sensing and Interaction for Assistive Autonomous Robots

Author(s): Emanuele De Pellegrin, Ronnie Smith, Scott MacLeod, Mauro Dragone & Ronald P. A. Petrick
Published in: TAROS 2023: Towards Autonomous Robotic Systems, 2023, Page(s) pp 162–173, ISBN 978-3-031-43359-7
Publisher: Annual Conference Towards Autonomous Robotic Systems
DOI: 10.1007/978-3-031-43360-3_14

Agri-Food Competition for Robot Evaluation (ACRE), the design of a competition of the METRICS project

Author(s): Daniel Boffety, Manon Boulet, Michel Berducat, Matteo Matteucci, Riccardo Bertoglio, Giulio Fontana, and Davide Facchinetti
Published in: 8 th International Conference on Machine Control & Guidance, 2022, Page(s) 7 to 13
Publisher: Laboratory of Off-Road Machine and Vehicle Engineering

RAMI: an Inspection and Maintenance Competition for Underwater Robots

Author(s): Gabriele Ferri; Alessandro Faggiani; Tommaso Fabbri; Fausto Ferreira
Published in: 2023
Publisher: IEEE
DOI: 10.1109/oceanslimerick52467.2023.10244694

From ERL to RAMI: Expanding Marine Robotics Competitions - Through Virtual Events

Author(s): Gabriele Ferri; Fausto Ferreira; Alessandro Faggiani; Tommaso Fabbri
Published in: OCEANS 2021: San Diego – Porto, 2022
Publisher: IEEE
DOI: 10.23919/oceans44145.2021.9706098

Adaptive Heading for Perception-Aware Trajectory Following

Author(s): Jonatan Scharff Willners, Sean Katagiri, Shida Xu1, Tomasz Łuczy´nski, Joshua Roe and Yvan Petillot
Published in: 2023
Publisher: 2023 IEEE International Conference on Robotics and Automation (ICRA 2023)
DOI: 10.1109/icra48891.2023.10160521

Lowering the Entry Barrier to Aerial Robotics Competitions

Author(s): Francisco J. Pérez-Grau; Pablo León Barriga; Antidio Viguria
Published in: 2023
Publisher: IEEE
DOI: 10.1109/icuas57906.2023.10156284

Observability-Aware Active Extrinsic Calibration of Multiple Sensors

Author(s): Shida Xu, Jonatan Scharff Willners, Ziyang Hong, Kaicheng Zhang, Yvan R. Petillot and Sen Wang
Published in: 2023
Publisher: 2023 IEEE International Conference on Robotics and Automation (ICRA 2023)
DOI: 10.1109/icra48891.2023.10160636

Multi-label Annotation for Visual Multi-Task Learning Models

Author(s): G. Sharma, A. Angleraud, R. Pieters
Published in: IEEE International Conference on Robotic Computing, 2023
Publisher: IEEE

Automatic Dataset Generation From CAD for Vision-Based Grasping

Author(s): Saad Ahmad, Kulunu Samarawickrama, Esa Rahtu and Roel Pieters
Published in: 2021 20th International Conference on Advanced Robotics (ICAR), 2022
Publisher: IEEE
DOI: 10.1109/icar53236.2021.9659336

Validation de méthodologies d’évaluation de solutions de désherbage autonomes, dans le cadre des projets Challenge ROSE et METRICS. Validation of methodologies for the evaluation of autonomous weed in the framework of the Challenge ROSE and METRICS projects.

Author(s): Rémi Regnier ; Guillaume Avrin ; Virginie Barbosa ; Daniel Boffety ; Anne Kalouguine ; Sophie Lardy-Fontan
Published in: Revue Ouverte d’Intelligence Artificielle-ROIA Open Journal of Artificial Intelligence-ROIA, 2021
Publisher: Revue Ouverte d’Intelligence Artificielle-ROIA Open Journal of Artificial Intelligence-ROIA
DOI: 10.5802/roia.8

Benchmarking Robots by Inducing Failures in Competition Scenarios

Author(s): Santosh Thoduka, Nico Hochgeschwender
Published in: Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. AI, Product and Service - 12th International Conference, DHM 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part II, Issue 12778, 2021, Page(s) 263-276, ISBN 978-3-030-77819-4
Publisher: Springer International Publishing
DOI: 10.1007/978-3-030-77820-0_20

A COTS (UHF) RFID Floor for Device-Free Ambient Assisted Living Monitoring

Author(s): Smith, R., Ding, Y., Goussetis, G., Dragone, M
Published in: Advances in intelligent system and computing - Vol 1239 - Ambiant intelligence - Software and applications - 11th international symposium of ambient intelligence, 2020
Publisher: International Symposium on Ambient Intelligence
DOI: 10.1007/978-3-030-58356-9

Assessing artificial intelligence capabilities

Author(s): Guillaume Avrin - Laboratoire national de métrologie et d'essais
Published in: Educational Research and Innovation - AI and the Future of Skills, Volume 1 - CAPABILITIES AND ASSESSMENTS, 2021
Publisher: OECD iLibrary
DOI: 10.1787/47d04fe3-en

The VISTA datasets, a combination of inertial sensors and depth cameras data for activity recognition

Author(s): Fiorini, L., Cornacchia Loizzo, F. G., Sorrentino, A., Rovini, E., Di Nuovo, A., & Cavallo, F.
Published in: Nature Scientific Data, Issue Scientific Data, 9(1), 1-14 -Article number: 218 (2022), 2022, ISSN 2052-4463
Publisher: Scientific Data (Sci Data)
DOI: 10.1038/s41597-022-01324-3

Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors

Author(s): Caetano Mazzoni Ranieri, Scott MacLeod, Mauro Dragone, Patricia Amancio Vargas, Roseli Aparecida Francelin Romero 
Published in: Sensors, Issue 21/3, 2021, Page(s) 768, ISSN 1424-8220
Publisher: Multidisciplinary Digital Publishing Institute (MDPI)
DOI: 10.3390/s21030768

A Comparative Study of Fourier Transform and CycleGAN as Domain Adaptation Techniques for Weed Segmentation

Author(s): Riccardo Bertoglio; Alessio Mazzucchelli ; Nico Catalano ; Matteo Matteucci
Published in: Smart Agricultural Technology, 2023, ISSN 1872-6291
Publisher: Science direct - Elsevier
DOI: 10.1016/j.atech.2023.100188

6D Assembly Pose Estimation by Point Cloud Registration for Robot Manipulation

Author(s): K. Samarawickrama, G. Sharma, A. Angleraud, R. Pieters
Published in: 2023
Publisher: arXiv
DOI: 10.48550/arxiv.2312.02593

Évaluation de l’intelligence artificielle

Author(s): Guillaume AVRIN
Published in: 2023
Publisher: Techniques de l'ingenieur
DOI: 10.51257/a-v1-r727

Automatic Robot Path Planning for Visual Inspection from Object Shape

Author(s): O. Tasneem, R. Pieters
Published in: 2023, Page(s) 8 pages
Publisher: arXiv
DOI: 10.48550/arxiv.2312.02603

Searching for OpenAIRE data...

There was an error trying to search data from OpenAIRE

No results available