Periodic Reporting for period 3 - OpenDR (Open Deep Learning Toolkit for Robotics)
Reporting period: 2023-01-01 to 2023-12-31
The aim of OpenDR project was to develop a modular, open, and non-proprietary toolkit for core robotic functionalities by harnessing deep learning to provide advanced perception and cognition capabilities, meeting in this way the general requirements of robotics applications in the applications areas of healthcare, agri-food and agile production. The term toolkit in OpenDR refers to a set of deep learning software functions, packages and utilities used to help roboticists to develop and test a robotic application that incorporates deep learning. OpenDR focused on the AI and Cognition core technology in order to provide tools that make robotic systems cognitive, giving them the ability to a) interact with people and environments by developing deep learning methods for human centric and environment active perception and cognition, b) learn and categorise by developing deep learning tools for training and inference in common robotics settings, and c) make decisions and derive knowledge by developing deep learning tools for cognitive robot action and decision making. By these it wished to enable a greater range of robotics applications that can be demonstrated at TRL 3 and above, thus lowering the technical barriers within the prioritised application areas. We do believe that the major outcome of the project, namely the OpenDR toolkit, has fulfilled its goal of providing easy to train and deploy, real-time, lightweight, Robot Operating System (ROS) compliant deep learning models for robotics that provide advanced perception and cognition capabilities. Apart from this, the scientific outcomes of OpenDR, documented in 100 publications, have pushed the state of the art in areas such as deep environment and human-centric active perception and cognition, deep robot action and decision making, simulation environments and data generation etc.
The first aim of the project’s work was the development of detailed requirements and specifications for the toolkit design. Afterwards, work progressed on developing DL methodologies that fulfilled the requirements of robotics applications. Significant progress was made in all directions OpenDR aimed to, i.e. deep human-centric and environment perception and cognition methods, deep robot action and decision making, as well as appropriate simulation environments and compilation of datasets to support the training of advanced DL models for robotics. More specifically, significant progress was witnessed on all tasks of human-centric perception, that include deep person/face/body part active detection/recognition and pose estimation, deep person/face/body part tracking, human activity recognition, social signal (facial expression, gesture, posture, etc.) analysis and recognition, deep speech and biosignals analysis and recognition, as well as multi-modal human centric perception and cognition. To this end, a multitude of state-of-the-art methodologies and models from targeted applications were implemented and evaluated, in addition to novel work in a wide range of domains, e.g. adaptive inference models, active perception methodologies, continual learning and inference, etc. Significant progress was also witnessed in the task of deep environment perception and cognition, where several novel state-of-the-art methods were developed. Significant results were also obtained in all tasks for deep robot action and decision making, i.e. deep planning, navigation, action and control and human-robot interaction. OpenDR partners also worked towards developing simulation tools for training efficient DL algorithms. More specifically, the Webots simulator was extended by improving its simulation capabilities, adjusting the simulation environment to make it highly compatible with the ROS framework, and consequently with the corresponding real robotics systems, as well as preparing the infrastructure to run simulations on the web to give a high visibility to the OpenDR results. The project also developed 15 open datasets and software modules to create data. OpenDR tools were extensively evaluated and the results of the evaluation, along with the computing requirements and potential limitations were carefully documented. Furthermore, the OpenDR toolkit was successfully integrated and demonstrated in the three targeted use cases, i.e. healthcare, agriculture and agile production. Finally, the consortium has released three major versions of the toolkit. The latest one has been publicly released in December 2023 and integrates more than 30 tools for various tasks (activity recognition, face detection and recognition, human pose estimation, object detection and tracking, semantic and panoptic segmentation, facial emotion recognition, hand gesture recognition etc.), updates to support more recent software versions (e.g. CUDA and ROS2), improved and efficient implementations of several tools, agile integration and testing pipelines, as well as detailed documentation and usage examples. The OpenDR team will continue supporting the toolkit in the years to come. The toolkit reception from the robotics / deep learning /computer vision community is already very encouraging: so far, the GitHub repository was awarded more than 590 stars and the toolkit (as a whole or individual tools) has been downloaded more than 17000 times. Moreover, the project results have been disseminated in 33 journal articles, 66 conference papers and 1 edited book.
OpenDR has produced numerous state-of-the-art research results, ranging from three major versions of the toolkit to fast and efficient lightweight DL models that are suitable for on-board deployment on robots and results that demonstrate the potential of active DL perception methods. The final version of the OpenDR toolkit, released in December 2023, is already having an impact on the way DL functionalities are built on robotics applications, since it provides an easy-to-use toolkit with highly efficient state-of-the-art perception algorithms along with pre-trained models, as also witnessed by the lively community built around it. Furthermore, the research that has been conducted on active perception capabilities has set new frontiers for DL-based active perception on robotics, going beyond existing static perception and enabling the development of the next generation of robots that can process environmental stimuli to achieve their goals in highly dynamic and unconstrained environments. The toolkit is expected to significantly lower the technical barriers by providing easy to use tools, which in turn can further enhance innovation capacity and will lead to the creation of new market opportunities. The developed technologies are ready to be taken up by the industry, transforming the European robotics industry into a key player. At the same time, OpenDR is expected to create new market opportunities by providing active deep learning approaches that smoothly co-integrate with simulation environments, providing a new training paradigm for active deep robotics, with one partner of the project (Cyberbotics) already having a solid exploitation plan toward this direction. As a result, we expect OpenDR to have a significant impact on small to medium robotics companies. Moreover, having available a series of tools along with the simulation environment will enable many new application ideas to reach the market or to be demonstrated at least as TRL3 systems, thus making a significant step to the market.