The EUROBENCH project aims to reach the following global objectives, each representing an advance on the State of the Art in the evaluation of robotic performance:
1. Achieving an international consensus in the industrial and academic community on the benchmarking requirements. The involvement of the international community is a required step to convert benchmarks into future standards. The EUROBENCH project is leading several efforts to create a worldwide network of industrial and academic entities in discussing on the specific requirements on standardization and terminology of robotic systems. This is expected to facilitate the inclusion of good practice and regulations for the smooth introduction of robotics systems in the society.
2. Identifying a scientific method to quantitatively assess the system abilities of bipedal robots and establish the causal relationships with the technologies composing them. The EUROBENCH is gathering together algorithms and data from many Third Parties involved through the cascade funding Open Calls. This massive software and data sharing has been never achieved before in the bipedal locomotion field, and is expected to advance the knowledge and habilitate to application of artificial intelligence algorithms to establish reliable predictors of robot performance, and help developers identify technical shortcomings efficiently.
3. Providing practical tools to industry and academy to efficiently test robotic platforms at any stage of development, from early prototyping to commercial products. The novel value proposition of EUROBENCH stands on three main pillars, i.e. facilities, benchmarking algorithms, and datasets, which are being designed to be easy accessible by anyone worldwide. Current Testing facilities (in Europe and US) only provide very simple benchmarks, based on completion time and do/not-do metrics. EUROBENCH, instead, will include a comprehensive set of test benches and metrics to evaluate bipedal robotic performance under a multi-faceted perspective. This will increase the adoption of benchmarking methods as a common practice, and therefore permit to achieve systems with higher quality and lower costs.
4. Achieving an effective utilisation of the benchmarking methodology in the bipedal robotics community and an efficient transferability to other application domains. The main advance in this respect will be to have flexible tools, focused on sub-tasks, which can be used across technological fields and application domains. For this reason the Benchmarking Software and Database will be modular and prone to be extended over time. This approach is not in conflict with the competition approaches, but represents a complementary tool that can help researchers to find causal relationships between the performance during competition and the performance in each of the sub-functions, taken separately.
5. Create a sustainable “benchmarking infrastructure” for the European robotics community. The EUROBENCH project contains a sustainability plan that aims to convert the EUROBENCH outcomes (facilities, software, database) into a business model, based on pay-per-use model, which will offer users different kinds of services. This model, partly based on existing Testing Facilities (e.g. NIST-RACE), includes additional services U8e.g. calculation of Perfromance Indicators, or access to a common Database) which have never been offered before by any other facility for Robot testing worldwide.
