Hereinafter, relevant activities carried out in each WP during the first 32 months towards the achievement of the above objectives are briefly summarized:
WP1: kick-off the project, organization of regular meetings among partners, submissions and approvals of Ethics committee documentations for preliminary experiments, mitigation of the effect of COVID-19 pandemic on project activities, request of amendment to the European Commission, reshaping of the project’s timeline, addition of Newcastle University as new coordinator unit.
WP2: definition of the user needs and requirements of the CONBOTS platform, technical and functional specifications of the different platform components, as well as for their integration; analysis of the non-technical specifications arising from the ethical, legal, socio-economic, privacy and cybersecurity issues related to the use of the CONBOTS technologies.
WP3: design of haptic robotic devices for human-human physical communication, i.e. end-effector haptic device for handwriting learning and exoskeleton devices for music learning
WP4: study of the physiological mechanisms of human-human physical communication, i.e. how performance and perception resulting from interaction with a robotic partner compare with the interaction between humans, how human-human interaction changes in a 3DoF task considering gravity and asymmetric interaction, and what are the effects of delay on haptic communication.
WP5: realization of the bi-directional user interface, which encompasses AR/VR based serious games, wearable sensors, and instrumented objects. Software components of the AR environment were developed to facilitate children learning in the two application scenarios, while initial experiments with adult violin players have been performed to test how suitable is the VR for this application scenario.
WP6: release of the machine learning models to estimate the physical and mental state of the user.
WP7: integration of the different technologies of the CONBOTS platform and its customization for the two application scenarios. Definition of the general architecture of the platform, and first implementation of the developed technologies, that include wearable sensors, robotic haptic devices, a graphical user interface, AR-based software environment developed in Unity3D, and instrumented objects.
WP8: planning and executing preliminary experiments in the two application scenarios, i.e. handwriting and music learning. As regards handwriting, twelve children were recruited and evaluated using the GHEE (Grapho-motor Handwriting Exercise & Empowerment) protocol. For the music scenario, another exploratory study has been setup and carried out to test the use of AR for violin education. Moreover, 10 drum students were followed up for 1 year, every 2 months, in the laboratory environment of GU. In every lesson, a fixed set of exercises and a play-along drum sequence was taught by the teacher and performed by the students, while audio, video and MoCap data of the students were collected.
WP9: dissemination activities and the activities related to the exploitation of project results.
WP10: submission of the deliverables related to the Ethics issues highlighted in the Ethics Summary Report.
CONBOTS research activities have achieved all of the main goals of the project, from technology to experiments, including ethical and legal aspects. In particular, we have addressed privacy and cybersecurity issues, validated the machine learning model in the use cases, and validated the integrated CONBOTS platform in the relevant scenarios of violin playing and handwriting through a number of experimental studies. CONBOTS empirical results demonstrated the effectiveness of robot mediated haptic feedback in relevant musical tasks (i.e. playing violin) and handwriting. CONBOTS research output can be summarized in the following numbers: 31 deliverables submitted, 10 milestones achieved, 65+ publications in journals/conference proceedings, 50+ invited talks/workshops/other public events, 20+ new experimental protocols designed with data from 300+ subjects gathered, 5 new hardware prototypes, 3 AR/VR apps, 5 software tools, 9 PhD fellowships, and 9 post-docs promoted to lecturer/assist prof positions.
