Periodic Reporting for period 2 - CONBOTS (CONnected through roBOTS: physically coupling humans to boost handwriting and music learning)
Okres sprawozdawczy: 2021-07-01 do 2022-08-31
The project tackles the challenge of introducing in education scenarios a new class of robots, i.e. the CONBOTS, which physically couple people to facilitate learning and improve efficiency during training of new motor skills. The CONBOTS platform will be used in two different contexts:
a) Children who are learning to handwrite.
b) Beginners who are learning to play violin or drums.
The CONBOTS project will impact on society in different ways: i) spreading the adoption of robots to a novel application scenario, i.e. education; ii) fostering the growing area of Education 4.0; iii) having children as target end-users and thus fostering children capability to work at the intersection of people and technology.
CONBOTS objectives are:
A) To define the user needs and platform requirements and to deploy measures to raise awareness and take-up of CONBOTS scientific and technological achievements;
B) To design physically interacting robotic haptic devices in the form of an end-effector robotic workstation and a wearable exoskeleton;
C) To exploit recent neuroscience insights on physical interaction and modern control theory to establish a Game Theory of physical communication for improving human-human and human-robot interactions;
D) To design a bi-directional user interface with Augmented Reality serious games, wearable sensors and instrumented objects for maximising the impact of physical interaction in learning contexts;
E) To develop machine learning algorithms for describing the physical, emotional, and mental state of the users, in order to tailor serious games and optimise the training process to the specific user needs;
F) To Integrate the different CONBOTS enabling technologies and optimise them for the two application scenarios;
G) To test the efficacy of CONBOTS platforms for training sensorimotor skills of children during handwriting and music learning.
a) Children who are learning to handwrite.
b) Beginners who are learning to play violin or drums.
The CONBOTS project will impact on society in different ways: i) spreading the adoption of robots to a novel application scenario, i.e. education; ii) fostering the growing area of Education 4.0; iii) having children as target end-users and thus fostering children capability to work at the intersection of people and technology.
CONBOTS objectives are:
A) To define the user needs and platform requirements and to deploy measures to raise awareness and take-up of CONBOTS scientific and technological achievements;
B) To design physically interacting robotic haptic devices in the form of an end-effector robotic workstation and a wearable exoskeleton;
C) To exploit recent neuroscience insights on physical interaction and modern control theory to establish a Game Theory of physical communication for improving human-human and human-robot interactions;
D) To design a bi-directional user interface with Augmented Reality serious games, wearable sensors and instrumented objects for maximising the impact of physical interaction in learning contexts;
E) To develop machine learning algorithms for describing the physical, emotional, and mental state of the users, in order to tailor serious games and optimise the training process to the specific user needs;
F) To Integrate the different CONBOTS enabling technologies and optimise them for the two application scenarios;
G) To test the efficacy of CONBOTS platforms for training sensorimotor skills of children during handwriting and music learning.
Hereinafter, relevant activities carried out in each WP during the first 32 months towards the achievement of the above objectives are briefly summarized:
WP1: kick-off the project, organization of regular meetings among partners, submissions and approvals of Ethics committee documentations for preliminary experiments, mitigation of the effect of COVID-19 pandemic on project activities, request of amendment to the European Commission, reshaping of the project’s timeline, addition of Newcastle University as new coordinator unit.
WP2: definition of the user needs and requirements of the CONBOTS platform, technical and functional specifications of the different platform components, as well as for their integration; analysis of the non-technical specifications arising from the ethical, legal, socio-economic, privacy and cybersecurity issues related to the use of the CONBOTS technologies.
WP3: design of haptic robotic devices for human-human physical communication, i.e. end-effector haptic device for handwriting learning and exoskeleton devices for music learning
WP4: study of the physiological mechanisms of human-human physical communication, i.e. how performance and perception resulting from interaction with a robotic partner compare with the interaction between humans, how human-human interaction changes in a 3DoF task considering gravity and asymmetric interaction, and what are the effects of delay on haptic communication.
WP5: realization of the bi-directional user interface, which encompasses AR/VR based serious games, wearable sensors, and instrumented objects. Software components of the AR environment were developed to facilitate children learning in the two application scenarios, while initial experiments with adult violin players have been performed to test how suitable is the VR for this application scenario.
WP6: release of the machine learning models to estimate the physical and mental state of the user.
WP7: integration of the different technologies of the CONBOTS platform and its customization for the two application scenarios. Definition of the general architecture of the platform, and first implementation of the developed technologies, that include wearable sensors, robotic haptic devices, a graphical user interface, AR-based software environment developed in Unity3D, and instrumented objects.
WP8: planning and executing preliminary experiments in the two application scenarios, i.e. handwriting and music learning. As regards handwriting, twelve children were recruited and evaluated using the GHEE (Grapho-motor Handwriting Exercise & Empowerment) protocol. For the music scenario, another exploratory study has been setup and carried out to test the use of AR for violin education. Moreover, 10 drum students were followed up for 1 year, every 2 months, in the laboratory environment of GU. In every lesson, a fixed set of exercises and a play-along drum sequence was taught by the teacher and performed by the students, while audio, video and MoCap data of the students were collected.
WP9: dissemination activities and the activities related to the exploitation of project results.
WP10: submission of the deliverables related to the Ethics issues highlighted in the Ethics Summary Report.
WP1: kick-off the project, organization of regular meetings among partners, submissions and approvals of Ethics committee documentations for preliminary experiments, mitigation of the effect of COVID-19 pandemic on project activities, request of amendment to the European Commission, reshaping of the project’s timeline, addition of Newcastle University as new coordinator unit.
WP2: definition of the user needs and requirements of the CONBOTS platform, technical and functional specifications of the different platform components, as well as for their integration; analysis of the non-technical specifications arising from the ethical, legal, socio-economic, privacy and cybersecurity issues related to the use of the CONBOTS technologies.
WP3: design of haptic robotic devices for human-human physical communication, i.e. end-effector haptic device for handwriting learning and exoskeleton devices for music learning
WP4: study of the physiological mechanisms of human-human physical communication, i.e. how performance and perception resulting from interaction with a robotic partner compare with the interaction between humans, how human-human interaction changes in a 3DoF task considering gravity and asymmetric interaction, and what are the effects of delay on haptic communication.
WP5: realization of the bi-directional user interface, which encompasses AR/VR based serious games, wearable sensors, and instrumented objects. Software components of the AR environment were developed to facilitate children learning in the two application scenarios, while initial experiments with adult violin players have been performed to test how suitable is the VR for this application scenario.
WP6: release of the machine learning models to estimate the physical and mental state of the user.
WP7: integration of the different technologies of the CONBOTS platform and its customization for the two application scenarios. Definition of the general architecture of the platform, and first implementation of the developed technologies, that include wearable sensors, robotic haptic devices, a graphical user interface, AR-based software environment developed in Unity3D, and instrumented objects.
WP8: planning and executing preliminary experiments in the two application scenarios, i.e. handwriting and music learning. As regards handwriting, twelve children were recruited and evaluated using the GHEE (Grapho-motor Handwriting Exercise & Empowerment) protocol. For the music scenario, another exploratory study has been setup and carried out to test the use of AR for violin education. Moreover, 10 drum students were followed up for 1 year, every 2 months, in the laboratory environment of GU. In every lesson, a fixed set of exercises and a play-along drum sequence was taught by the teacher and performed by the students, while audio, video and MoCap data of the students were collected.
WP9: dissemination activities and the activities related to the exploitation of project results.
WP10: submission of the deliverables related to the Ethics issues highlighted in the Ethics Summary Report.
CONBOTS’ major progresses beyond the state of the art are:
1) Design and test of haptic device conceived as a system capable to feedback the sense of touch with minor exchange of forces to increase user’s sensory experience.
2) Design and test of a portable upper-limb exoskeleton for learning to play violin and drums.
3) Customization of a non-invasive, easy-to-use, instrumented wearable system, capable to measure physiological signals of arousal to map users’ stress and mental state.
The CONBOTS project is expected to impact in different ways, such as strengthening European excellence in robotics S&T; boosting the use of robotics in promising application areas; and lowering barriers in the deployment of robotics-based solutions.
1) Strengthening European excellence in Robotics S&T: Research activities of the CONBOTS fostered the collaboration between academic and industrial partners. These collaborations strengthened the expertise of the consortium in robotics and related fields, such as wearable robotics, wearable sensors and instrumented objects, human robotics, human-robot interaction, virtual and augmented reality for robotics application, AI, machine learning and data science.
2) Boosting the use of robotics in promising application areas: we have been able to carry out preliminary experiments on few children and on a larger number of adults in the two application areas of music and handwriting, thus gathering information useful to tailor the platform for our final users.
3) Lowering barriers in the deployment of robotics-based solutions: To make robotic devices more easy-to-use and adaptable, CONBOTS developed and tested different solutions combining portable, easy-to-wear exoskeleton, with the aim to provide haptic guidance instead of motor assistance. This process has been carried out by industrial partners’ (e.g. IUVO) under the careful supervision of experimental partners (e.g. UCBM, ICL and GU) with the aim of developing an interface capable of establishing a physical communication between humans by coupling their movements, so as to improve the learning of new motor tasks. This new class of robot can foster a shift of robotic technologies from industrial settings and structured scenarios towards consumer application and new market sectors in real-life contexts.
1) Design and test of haptic device conceived as a system capable to feedback the sense of touch with minor exchange of forces to increase user’s sensory experience.
2) Design and test of a portable upper-limb exoskeleton for learning to play violin and drums.
3) Customization of a non-invasive, easy-to-use, instrumented wearable system, capable to measure physiological signals of arousal to map users’ stress and mental state.
The CONBOTS project is expected to impact in different ways, such as strengthening European excellence in robotics S&T; boosting the use of robotics in promising application areas; and lowering barriers in the deployment of robotics-based solutions.
1) Strengthening European excellence in Robotics S&T: Research activities of the CONBOTS fostered the collaboration between academic and industrial partners. These collaborations strengthened the expertise of the consortium in robotics and related fields, such as wearable robotics, wearable sensors and instrumented objects, human robotics, human-robot interaction, virtual and augmented reality for robotics application, AI, machine learning and data science.
2) Boosting the use of robotics in promising application areas: we have been able to carry out preliminary experiments on few children and on a larger number of adults in the two application areas of music and handwriting, thus gathering information useful to tailor the platform for our final users.
3) Lowering barriers in the deployment of robotics-based solutions: To make robotic devices more easy-to-use and adaptable, CONBOTS developed and tested different solutions combining portable, easy-to-wear exoskeleton, with the aim to provide haptic guidance instead of motor assistance. This process has been carried out by industrial partners’ (e.g. IUVO) under the careful supervision of experimental partners (e.g. UCBM, ICL and GU) with the aim of developing an interface capable of establishing a physical communication between humans by coupling their movements, so as to improve the learning of new motor tasks. This new class of robot can foster a shift of robotic technologies from industrial settings and structured scenarios towards consumer application and new market sectors in real-life contexts.