Maestro movements: how robots can transform the way we learn to write and play music
The advance of artificial intelligence and robotics has transformed robots in interactive partners that can make a difference in various aspects of our lives. Neuroscientific studies demonstrate the benefits of physical interaction in learning complex sensorimotor tasks. They indicate that when two people perform a common task while physically interacting with each other, their performance improves when compared to doing the same task alone. Based on this concept, a consortium of eight partners started the EU-funded CONBOTS project to design a new class of robot that physically pairs people up to facilitate learning and improvement of handwriting and music skills – in particular, playing the violin. “Learning to write is something that everyone does during their life and has a huge impact on their long-life learning. In contrast, playing the violin requires acquiring very fine and specific sensorimotor skills. Taken together, these two tasks seemed to be both challenging and complementary, and capable of demonstrating the feasibility of our approach,” explains Domenico Formica, CONBOTS project coordinator.
Developing robotic tutors
The project integrated different technologies in a single platform, including robotic devices, wearable sensors as well as advanced control and modelling algorithms. The robots were thoroughly developed to physically connect two people doing the same task. The robotic devices, namely CONBOTS, are used in pairs: each person interacts with their own robot and feels what the partner is doing, as the devices are connected by a control algorithm which allows the transmission of force and movement from one device to the other. Taking the example of a teacher and a student practicing violin, two exoskeletons were developed for the upper limb, to be worn by the subjects while practising their bowing technique. Wearable sensors are then used to collect physiological data – like heart rate and skin conductance – and motion parameters such as movement smoothness. This information feeds machine learning models to estimate the level of engagement of the subjects and modulate the physical connection between them. CONBOTS used game theory as a computational framework to model the human-robot and human-human interactions during the tasks. In addition, augmented reality serious games were applied, along with the wearable sensors and instrumented objects, to design a bidirectional user interface. “The technologies developed can improve motor learning and facilitate the acquisition of the specific motor skills required to accomplish the tasks,” says Formica. “Our approach can nevertheless be extended to several other relevant contexts, from motor rehabilitation to sport sciences.”
Putting the benefits to the test
CONBOTS brought interesting insights to the field. For instance, it demonstrated that robot-mediated haptic communication outperforms visual cues when people need to synchronise their actions. Despite the challenge of taking these robotic technologies outside lab settings to real-world tasks, the project was able to achieve all its goals thanks to a multidisciplinary team composed of engineers, roboticists, educational psychologists, musicologists and teachers. The diverse expertise allowed the partners to find the best compromise between technical performance and usability. The latest versions of the two robotic platforms were tested with more than 15 child-parent couples for handwriting learning and 60 musicians for violin training. During the lifespan of the project, different technological solutions were tested on more than 300 volunteers.
Keywords
CONBOTS, robotics, robots, wearable sensors, education, motor skills, machine learning
