One of the main objective was to define the common methodological framework (LNE, E-Civis, IMT) for the four METRICS competitions, which:
- is based on the use of metrology-grade evaluation tools;
- takes into account ethics and socio-economic aspects;
- ensures proper management of data;
- includes a good practice guide for the organization of competitions.
METRICS also addressed ethics issues (data protection, management of human participation, recruitment of participants, feedback from an external ethics committee, etc.).
A four-year competition was organised for each of the four areas and included one dry-run and two official evaluation campaigns. Each campaign included a “field evaluation campaign” (tests in physical environments) and a “cascade evaluation campaign” (tests on datasets generated during the field campaigns).
The COVID-19 health crisis context severely limited, if not completely prevented, the possibility of hosting participants and spectators to the field competitions and forced the consortium to adapt the organization of the dry-run campaigns.
The partners in charge of the competitions developed their evaluation plan and tools and organized the three campaigns:
- HEART-MET (BRSU, HWU, UNIFI, UON) focuses on assistive robotics, with a particular attention paid to human-robot interaction. Its dry-run field campaign mobilized robots from internal partners and a total of 22 teams were involved during the 2022 and 2023 campaigns.
- RAMI (CATEC, CMRE, HWU) focuses on autonomous inspection of assets (aerial and underwater). Its dry-run involved robots from internal partners. 13 and 23 teams respectively participated to the first and second campaign.
- ACRE (POLIMI, UNIMI, INRAE) focuses on automatic weeding and two dry-run field evaluation campaigns involved 4 teams. The first campaign saw 10 participating teams. The 2023 campaign attracted 11 teams.
- ADAPT (TAU, CEA, OFFIS, PRX) focuses on collaborative assembly of parts. The first campaign attracted 7 teams. 8 teams participated to the second campaign.
METRICS consortium also counted more than one thousand physical spectators in 2023.
Furthermore, RTX led the implementation of communication actions to recruit participants, sponsors and members of the robotic stakeholder board.
The results of the project were presented through 24 scientific publications (6 others are pending publication) and interventions in scientific events. Additionally, METRICS shared openly datasets generated.
The primary exploitable outcomes of METRICS include the methodological common framework, and its templates designed for assessing compliance with guidelines, and the four evaluation plans. These tools are aimed at bolstering standardization and promoting best practices for the organization of future competitions.
Furthermore, METRICS has provided partners with expertise in efficiently organizing robotics campaigns (logistics, toolkits, testbeds).
The project partners have already identified or proactively engaged in coordinated initiatives and established tangible connections with stakeholders to promote the reuse of METRICS tools. For example, the umbrella of the TEF Digital Europe project is one of these opportunities.
