Periodic Reporting for period 3 - METRICS (Metrological Evaluation and Testing of Robots in International CompetitionS)
Reporting period: 2023-01-01 to 2023-12-31
Robotics has many exciting applications in healthcare, agri-food, agile manufacturing and inspection and maintenance. In these four application areas identified as priorities by the European Commission, METRICS aimed to generate substantive scientific and technical progress and brings together three key ingredients: rigorous evaluations (protocols and metrics based on metrological principles), test environments (physical test beds and databases) and the systems to be evaluated (provided by the participants).
Concerning the evaluations implemented, the main challenges of METRICS were to ensure:
- the relevance of the evaluation tasks chosen for the different economic stakeholders,
- that ethical criteria are taken into account in order to promote the acceptability of these intelligent robots by the general public,
- the repeatability and reproducibility of performance measurements to monitor the progress made by the technologies over time, for the benefit of developers and funding bodies.
To meet these challenges, METRICS implemented two unique elements in Europe:
- the largest consortium of test centres for intelligent robots (17 partners, all expert in robotics competition and metrology, with highly complementary testing facilities and networks in the four areas);
- a unique approach to organising competitions, which takes advantage of feedback from past and current european competitions and projects.
The competitions were designed to mobilise and draw attention from the entire economic sector:
- the robotics community was mobilised on field evaluation campaigns while the AI community focused on database evaluations,
- the interest of the industrial players was ensured by the implementation of a dedicated strategy to mobilise external partners and sponsors to support the organisation of the competitions and ensure their industrial relevance, with the collaboration of the DIHs.
- a specific outreach strategy was developed to promote robotics systems towards the general public and ensure the compliance of robots with ethical and socio-economic principles.
More than 100 research and industry teams actively engaged in METRICS campaigns, leading to the evaluation of more than 50 robots during field campaigns. The AI and robotics communities now possess a generic evaluation methodology along with the four shared evaluation plans, serving as a reference and turnkey tools for organizing forthcoming robotics competitions.
Concerning the evaluations implemented, the main challenges of METRICS were to ensure:
- the relevance of the evaluation tasks chosen for the different economic stakeholders,
- that ethical criteria are taken into account in order to promote the acceptability of these intelligent robots by the general public,
- the repeatability and reproducibility of performance measurements to monitor the progress made by the technologies over time, for the benefit of developers and funding bodies.
To meet these challenges, METRICS implemented two unique elements in Europe:
- the largest consortium of test centres for intelligent robots (17 partners, all expert in robotics competition and metrology, with highly complementary testing facilities and networks in the four areas);
- a unique approach to organising competitions, which takes advantage of feedback from past and current european competitions and projects.
The competitions were designed to mobilise and draw attention from the entire economic sector:
- the robotics community was mobilised on field evaluation campaigns while the AI community focused on database evaluations,
- the interest of the industrial players was ensured by the implementation of a dedicated strategy to mobilise external partners and sponsors to support the organisation of the competitions and ensure their industrial relevance, with the collaboration of the DIHs.
- a specific outreach strategy was developed to promote robotics systems towards the general public and ensure the compliance of robots with ethical and socio-economic principles.
More than 100 research and industry teams actively engaged in METRICS campaigns, leading to the evaluation of more than 50 robots during field campaigns. The AI and robotics communities now possess a generic evaluation methodology along with the four shared evaluation plans, serving as a reference and turnkey tools for organizing forthcoming robotics competitions.
One of the main objective was to define the common methodological framework (LNE, E-Civis, IMT) for the four METRICS competitions, which:
- is based on the use of metrology-grade evaluation tools;
- takes into account ethics and socio-economic aspects;
- ensures proper management of data;
- includes a good practice guide for the organization of competitions.
METRICS also addressed ethics issues (data protection, management of human participation, recruitment of participants, feedback from an external ethics committee, etc.).
A four-year competition was organised for each of the four areas and included one dry-run and two official evaluation campaigns. Each campaign included a “field evaluation campaign” (tests in physical environments) and a “cascade evaluation campaign” (tests on datasets generated during the field campaigns).
The COVID-19 health crisis context severely limited, if not completely prevented, the possibility of hosting participants and spectators to the field competitions and forced the consortium to adapt the organization of the dry-run campaigns.
The partners in charge of the competitions developed their evaluation plan and tools and organized the three campaigns:
- HEART-MET (BRSU, HWU, UNIFI, UON) focuses on assistive robotics, with a particular attention paid to human-robot interaction. Its dry-run field campaign mobilized robots from internal partners and a total of 22 teams were involved during the 2022 and 2023 campaigns.
- RAMI (CATEC, CMRE, HWU) focuses on autonomous inspection of assets (aerial and underwater). Its dry-run involved robots from internal partners. 13 and 23 teams respectively participated to the first and second campaign.
- ACRE (POLIMI, UNIMI, INRAE) focuses on automatic weeding and two dry-run field evaluation campaigns involved 4 teams. The first campaign saw 10 participating teams. The 2023 campaign attracted 11 teams.
- ADAPT (TAU, CEA, OFFIS, PRX) focuses on collaborative assembly of parts. The first campaign attracted 7 teams. 8 teams participated to the second campaign.
METRICS consortium also counted more than one thousand physical spectators in 2023.
Furthermore, RTX led the implementation of communication actions to recruit participants, sponsors and members of the robotic stakeholder board.
The results of the project were presented through 24 scientific publications (6 others are pending publication) and interventions in scientific events. Additionally, METRICS shared openly datasets generated.
The primary exploitable outcomes of METRICS include the methodological common framework, and its templates designed for assessing compliance with guidelines, and the four evaluation plans. These tools are aimed at bolstering standardization and promoting best practices for the organization of future competitions.
Furthermore, METRICS has provided partners with expertise in efficiently organizing robotics campaigns (logistics, toolkits, testbeds).
The project partners have already identified or proactively engaged in coordinated initiatives and established tangible connections with stakeholders to promote the reuse of METRICS tools. For example, the umbrella of the TEF Digital Europe project is one of these opportunities.
- is based on the use of metrology-grade evaluation tools;
- takes into account ethics and socio-economic aspects;
- ensures proper management of data;
- includes a good practice guide for the organization of competitions.
METRICS also addressed ethics issues (data protection, management of human participation, recruitment of participants, feedback from an external ethics committee, etc.).
A four-year competition was organised for each of the four areas and included one dry-run and two official evaluation campaigns. Each campaign included a “field evaluation campaign” (tests in physical environments) and a “cascade evaluation campaign” (tests on datasets generated during the field campaigns).
The COVID-19 health crisis context severely limited, if not completely prevented, the possibility of hosting participants and spectators to the field competitions and forced the consortium to adapt the organization of the dry-run campaigns.
The partners in charge of the competitions developed their evaluation plan and tools and organized the three campaigns:
- HEART-MET (BRSU, HWU, UNIFI, UON) focuses on assistive robotics, with a particular attention paid to human-robot interaction. Its dry-run field campaign mobilized robots from internal partners and a total of 22 teams were involved during the 2022 and 2023 campaigns.
- RAMI (CATEC, CMRE, HWU) focuses on autonomous inspection of assets (aerial and underwater). Its dry-run involved robots from internal partners. 13 and 23 teams respectively participated to the first and second campaign.
- ACRE (POLIMI, UNIMI, INRAE) focuses on automatic weeding and two dry-run field evaluation campaigns involved 4 teams. The first campaign saw 10 participating teams. The 2023 campaign attracted 11 teams.
- ADAPT (TAU, CEA, OFFIS, PRX) focuses on collaborative assembly of parts. The first campaign attracted 7 teams. 8 teams participated to the second campaign.
METRICS consortium also counted more than one thousand physical spectators in 2023.
Furthermore, RTX led the implementation of communication actions to recruit participants, sponsors and members of the robotic stakeholder board.
The results of the project were presented through 24 scientific publications (6 others are pending publication) and interventions in scientific events. Additionally, METRICS shared openly datasets generated.
The primary exploitable outcomes of METRICS include the methodological common framework, and its templates designed for assessing compliance with guidelines, and the four evaluation plans. These tools are aimed at bolstering standardization and promoting best practices for the organization of future competitions.
Furthermore, METRICS has provided partners with expertise in efficiently organizing robotics campaigns (logistics, toolkits, testbeds).
The project partners have already identified or proactively engaged in coordinated initiatives and established tangible connections with stakeholders to promote the reuse of METRICS tools. For example, the umbrella of the TEF Digital Europe project is one of these opportunities.
The METRICS project pushed the state of the art in three areas:
- It designed, validated and implemented a new methodology for organising robotics competitions capitalising on most existing approaches.
- It developed new tools for evaluating with metrological rigour, which will be the subject of standards.
- It allowed to move forward from the current state of the art in robotics (in particular to reduce costs and improve quality of life for older people thanks to assistive robots, increase autonomy, navigation capabilities of intelligent robotics systems for Oil&Gas and renewable energy sectors, develop autonomous weeding robots and reduce the use of phytosanitary products, reduce the costs and human capabilities to reconfigure work cells for the manufacture of new products).
The societal impact of METRICS is produced by building trust around intelligent robotics systems among the general public, industry, public authorities through accurate information regarding their performances and limits and their societal implications.
The economic and technological impact is ensured by:
- focusing on highly relevant robotics applications related to high-value fast-growing markets with numerous active European entities,
- the sharing of knowledge between participants,
- providing access to state-of-the-art evaluation tools and open access datasets and evaluation standards,
- making robotics competitions a natural tool to bring supply and demand together, encouraging innovation on the one hand, and validating/benchmarking new technologies on the other,
- developping a sustainable robotics competition business model based on co-funding from industrial players,
- and ultimately rationalising, developing and promoting the competitiveness of European industry.
- It designed, validated and implemented a new methodology for organising robotics competitions capitalising on most existing approaches.
- It developed new tools for evaluating with metrological rigour, which will be the subject of standards.
- It allowed to move forward from the current state of the art in robotics (in particular to reduce costs and improve quality of life for older people thanks to assistive robots, increase autonomy, navigation capabilities of intelligent robotics systems for Oil&Gas and renewable energy sectors, develop autonomous weeding robots and reduce the use of phytosanitary products, reduce the costs and human capabilities to reconfigure work cells for the manufacture of new products).
The societal impact of METRICS is produced by building trust around intelligent robotics systems among the general public, industry, public authorities through accurate information regarding their performances and limits and their societal implications.
The economic and technological impact is ensured by:
- focusing on highly relevant robotics applications related to high-value fast-growing markets with numerous active European entities,
- the sharing of knowledge between participants,
- providing access to state-of-the-art evaluation tools and open access datasets and evaluation standards,
- making robotics competitions a natural tool to bring supply and demand together, encouraging innovation on the one hand, and validating/benchmarking new technologies on the other,
- developping a sustainable robotics competition business model based on co-funding from industrial players,
- and ultimately rationalising, developing and promoting the competitiveness of European industry.