Periodic Reporting for period 2 - METRICS (Metrological Evaluation and Testing of Robots in International CompetitionS)
Período documentado: 2021-07-01 hasta 2022-12-31
Robotics has many exciting applications in healthcare, agri-food, agile manufacturing and inspection and maintenance. In these four application areas identified as priorities by the European Commission, METRICS aims to generate substantive scientific and technical progress and brings together three key ingredients: rigorous evaluations (protocols and metrics based on metrological principles), test environments (physical test beds and databases) and the systems to be evaluated (provided by the participants).
Concerning the evaluations implemented, the main challenges taken up by METRICS are: i) to ensure the relevance of the evaluation tasks chosen for the different economic stakeholders, ii) to ensure that ethical criteria are taken into account in order to promote the acceptability of these intelligent robots by the general public, iii) to ensure the repeatability and reproducibility of performance measurements to monitor the progress made by the technologies over time, for the benefit of developers and funding bodies.
To meet these challenges, METRICS implements two unique elements in Europe:
- the largest consortium of test centres for intelligent robots (17 experienced partners, all expert in robotics competition and metrology, with highly complementary testing facilities and networks in the four application areas);
- a unique approach to organising competitions, which takes advantage of feedback from past and current european competitions and projects.
The competitions are designed to mobilise and draw attention from the entire economic sector:
- the robotics community is mobilised on field evaluation campaigns while the AI community focuses on database evaluations,
- the interest of the industrial players is ensured by the implementation of a dedicated strategy to mobilise external partners and sponsors to support the organisation of the competitions and ensure their industrial relevance, with the collaboration of the DIHs.
- a specific outreach strategy is developed to promote robotics systems towards the general public and ensure the compliance of robots with ethical and socio-economic principles.
Concerning the evaluations implemented, the main challenges taken up by METRICS are: i) to ensure the relevance of the evaluation tasks chosen for the different economic stakeholders, ii) to ensure that ethical criteria are taken into account in order to promote the acceptability of these intelligent robots by the general public, iii) to ensure the repeatability and reproducibility of performance measurements to monitor the progress made by the technologies over time, for the benefit of developers and funding bodies.
To meet these challenges, METRICS implements two unique elements in Europe:
- the largest consortium of test centres for intelligent robots (17 experienced partners, all expert in robotics competition and metrology, with highly complementary testing facilities and networks in the four application areas);
- a unique approach to organising competitions, which takes advantage of feedback from past and current european competitions and projects.
The competitions are designed to mobilise and draw attention from the entire economic sector:
- the robotics community is mobilised on field evaluation campaigns while the AI community focuses on database evaluations,
- the interest of the industrial players is ensured by the implementation of a dedicated strategy to mobilise external partners and sponsors to support the organisation of the competitions and ensure their industrial relevance, with the collaboration of the DIHs.
- a specific outreach strategy is developed to promote robotics systems towards the general public and ensure the compliance of robots with ethical and socio-economic principles.
One of the main objective of the first months of the project was to define the common methodological framework for the four METRICS competitions, which:
- is based on the use of metrology-grade evaluation tools (work led by LNE);
- takes into account ethics and socio-economic aspects (E-Civis);
- ensures proper management of data (IMT);
- includes a good practice guide for the organization of competitions (LNE).
LNE also defined a methodology for addressing ethics issues in METRICS (data protection, management of human participation in the competitions, recruitment of research participants, feedback gathering from an external ethics committee, etc.).
A four-year competition is organised for each of the application areas and includes one dry-run and two official evaluation campaigns. Each campaign includes a “field evaluation campaign” (tests in physical environments) and a “cascade evaluation campaign” (tests on datasets generated during the field evaluation campaign).
The first period of the project has been strongly impacted by the COVID-19 health crisis and the situation severely limited, if not completely prevented, the possibility of hosting participants and spectators to the field competitions and forced the consortium to adapt the organization of the dry-run campaigns.
The partners in charge of the four competitions have developed their evaluation plan and tools. Since the beginning of the project the dry-run campaigns (2020-2021) and the first campaigns (2022) were organised :
- HEART-MET evaluation plan (BRSU, HWU, UNIFI, UON) focuses on assistive robotics, with a particular attention paid to human-robot interaction. Its dry-run field evaluation campaign mobilized robots from internal partners and 15 teams were involved during the first campaign.
- RAMI evaluation plan (CATEC, CMRE, HWU) focuses on autonomous inspection of assets. Its dry-run involved robots from internal partners (aerial and underwater) and 13 teams participated to the first campaign .
- ACRE evaluation (POLIMI, UNIMI, INRAE) focuses on automatic weeding and two dry-run field evaluation campaigns, organised at INRAE facility, involved 4 teams. The first campaign saw 10 participating teams .
- ADAPT evaluation plan (TAU, CEA, OFFIS, PRX) focuses on collaborative assembly of parts. 3 teams participated to the first field campaign and the first cascade campaign is still on-going until end of March 2023.
Furthermore, RTX led the implementation of communication actions to recruit participants, sponsors and members of the robotic stakeholder board of the project in addition to direct solicitations. The first results of the project were also presented through scientific publications (see https://metricsproject.eu) and interventions in workshops.
Based on the feedbacks from these previous campaigns, the consortium is now on track to organise the second and final evaluation campaigns for which the calls for participation have just been launched. In parallel, important actions will be carried out in the coming months in view of the exploitation and dissemination of METRICS results.
- is based on the use of metrology-grade evaluation tools (work led by LNE);
- takes into account ethics and socio-economic aspects (E-Civis);
- ensures proper management of data (IMT);
- includes a good practice guide for the organization of competitions (LNE).
LNE also defined a methodology for addressing ethics issues in METRICS (data protection, management of human participation in the competitions, recruitment of research participants, feedback gathering from an external ethics committee, etc.).
A four-year competition is organised for each of the application areas and includes one dry-run and two official evaluation campaigns. Each campaign includes a “field evaluation campaign” (tests in physical environments) and a “cascade evaluation campaign” (tests on datasets generated during the field evaluation campaign).
The first period of the project has been strongly impacted by the COVID-19 health crisis and the situation severely limited, if not completely prevented, the possibility of hosting participants and spectators to the field competitions and forced the consortium to adapt the organization of the dry-run campaigns.
The partners in charge of the four competitions have developed their evaluation plan and tools. Since the beginning of the project the dry-run campaigns (2020-2021) and the first campaigns (2022) were organised :
- HEART-MET evaluation plan (BRSU, HWU, UNIFI, UON) focuses on assistive robotics, with a particular attention paid to human-robot interaction. Its dry-run field evaluation campaign mobilized robots from internal partners and 15 teams were involved during the first campaign.
- RAMI evaluation plan (CATEC, CMRE, HWU) focuses on autonomous inspection of assets. Its dry-run involved robots from internal partners (aerial and underwater) and 13 teams participated to the first campaign .
- ACRE evaluation (POLIMI, UNIMI, INRAE) focuses on automatic weeding and two dry-run field evaluation campaigns, organised at INRAE facility, involved 4 teams. The first campaign saw 10 participating teams .
- ADAPT evaluation plan (TAU, CEA, OFFIS, PRX) focuses on collaborative assembly of parts. 3 teams participated to the first field campaign and the first cascade campaign is still on-going until end of March 2023.
Furthermore, RTX led the implementation of communication actions to recruit participants, sponsors and members of the robotic stakeholder board of the project in addition to direct solicitations. The first results of the project were also presented through scientific publications (see https://metricsproject.eu) and interventions in workshops.
Based on the feedbacks from these previous campaigns, the consortium is now on track to organise the second and final evaluation campaigns for which the calls for participation have just been launched. In parallel, important actions will be carried out in the coming months in view of the exploitation and dissemination of METRICS results.
The METRICS project is pushing the state of the art in three areas:
- It designs, validates and implements a new methodology for organising robotics competitions capitalising on most existing approaches.
- It develops new tools for evaluating with metrological rigour, which will be the subject of standards.
- It allows to move forward from the current state of the art in robotics (in particular to reduce costs and improve quality of life for older people thanks to assistive robots, increase autonomy, navigation capabilities of intelligent robotics systems for Oil&Gas and renewable energy sectors, develop autonomous weeding robots and reduce the use of phytosanitary products, reduce the costs and human capabilities to reconfigure work cells for the manufacture of new products).
The societal impact of METRICS is produced by building trust around intelligent robotics systems among the general public, industry, public authorities through accurate information regarding their performances and limits and their societal implications.
The economic and technological impact is ensured by:
- focusing on highly relevant robotics applications related to high-value fast-growing markets with numerous active European entities,
- the sharing of knowledge between participants,
- providing access to state-of-the-art evaluation tools and open access datasets and evaluation standards,
- making robotics competitions a natural tool to bring supply and demand together, encouraging innovation on the one hand, and validating/benchmarking new technologies on the other,
- developping a sustainable robotics competition business model based on co-funding from industrial players,
- and ultimately rationalising, developing and promoting the competitiveness of European industry.
- It designs, validates and implements a new methodology for organising robotics competitions capitalising on most existing approaches.
- It develops new tools for evaluating with metrological rigour, which will be the subject of standards.
- It allows to move forward from the current state of the art in robotics (in particular to reduce costs and improve quality of life for older people thanks to assistive robots, increase autonomy, navigation capabilities of intelligent robotics systems for Oil&Gas and renewable energy sectors, develop autonomous weeding robots and reduce the use of phytosanitary products, reduce the costs and human capabilities to reconfigure work cells for the manufacture of new products).
The societal impact of METRICS is produced by building trust around intelligent robotics systems among the general public, industry, public authorities through accurate information regarding their performances and limits and their societal implications.
The economic and technological impact is ensured by:
- focusing on highly relevant robotics applications related to high-value fast-growing markets with numerous active European entities,
- the sharing of knowledge between participants,
- providing access to state-of-the-art evaluation tools and open access datasets and evaluation standards,
- making robotics competitions a natural tool to bring supply and demand together, encouraging innovation on the one hand, and validating/benchmarking new technologies on the other,
- developping a sustainable robotics competition business model based on co-funding from industrial players,
- and ultimately rationalising, developing and promoting the competitiveness of European industry.