Periodic Reporting for period 3 - SAHR (Skill Acquisition in Humans and Robots)
Reporting period: 2020-10-01 to 2022-03-31
Dexterous manipulation of objects is robotics’ 21st century primary goal. It envisions robots capable of sorting objects and packaging them, of chopping vegetables and folding clothes, and this, at high speed. To manipulate these objects cannot be done with traditional control approaches, for lack of accurate models of objects and contact dynamics. In the past two decades, robotics has leveraged the immense progress in artificial intelligence and machine learning to encapsulate models of uncertainty and to support further advances on adaptive and robust control. It, furthermore, benefited from advances in optimization for solving high-dimensional constrained problems. These methods are powerful for planning in slow-paced tasks and when the environment is known. They fall short to present the millisecond reactivity required to meet industry logistics needs and the need to respond rapidly to new manufacturing demands. The Skills Acquisition in Humans and Robots (SAHR) addresses society and industry need for novel control mechanisms for robots. It seeks to enable robots to acquire fine manipulation skills autonomously and to manipulate seamlessly tools at very high speed.
Humans have a remarkable capacity for fine manipulation and unique ability to acquire such skills through practice as well as through observation of more skilled practioneers. The best example is craftsmanship. Crafts demand high dexterity and proficiency at a craft is attained only after years of practice. The SAHR project aims at offering more insight on how humans acquire these skills and to use this knowledge to support the development of new learning algorithms for robots. To this end, SAHR develops novel approaches to skill acquisition in robots that follow stages of skill acquisition in humans. To inform the design of these new learning algorithms, we follow a group of apprentices at watchmaking. Watch making is an ideally suited craft as it requires very precise control of both hands’ fingers in coordination. Analysis of human acquisition skills aims to achieve a better understanding of how humans overcome their sensory-motor noise and the sheer complexity of controlling simultaneously many degrees of freedom. We develop a framework to enable robots to acquire similarly complex manipulation skills, through a two-stage process. First, a trial and error process allows to identify the task constraints and to determine a motor program satisfying these constraints. Second, an optimization process is launched to refine and optimize the skill to reach an economy of effort. Production of the skill becomes automated. To automate the skill, we use Dynamical Systems (DS), closed-form control laws that offer inherent robustness to perturbation, hence ideally suited to generate the required run-time high-speed adaptation.
Humans have a remarkable capacity for fine manipulation and unique ability to acquire such skills through practice as well as through observation of more skilled practioneers. The best example is craftsmanship. Crafts demand high dexterity and proficiency at a craft is attained only after years of practice. The SAHR project aims at offering more insight on how humans acquire these skills and to use this knowledge to support the development of new learning algorithms for robots. To this end, SAHR develops novel approaches to skill acquisition in robots that follow stages of skill acquisition in humans. To inform the design of these new learning algorithms, we follow a group of apprentices at watchmaking. Watch making is an ideally suited craft as it requires very precise control of both hands’ fingers in coordination. Analysis of human acquisition skills aims to achieve a better understanding of how humans overcome their sensory-motor noise and the sheer complexity of controlling simultaneously many degrees of freedom. We develop a framework to enable robots to acquire similarly complex manipulation skills, through a two-stage process. First, a trial and error process allows to identify the task constraints and to determine a motor program satisfying these constraints. Second, an optimization process is launched to refine and optimize the skill to reach an economy of effort. Production of the skill becomes automated. To automate the skill, we use Dynamical Systems (DS), closed-form control laws that offer inherent robustness to perturbation, hence ideally suited to generate the required run-time high-speed adaptation.
The project started with the conduct of an eighteen-months long longitudinal study of apprentices in watchmaking at the Ecole Technique de la Vallee de Joux in Switzerland. Apprentices’ manual capacities were assessed in six sessions, a month apart, over the course of two semesters to determine the evolution of the skill over the first year of training. Skill retention was further assessed by administering one more recording session at the beginning of school after the summer vacation. Apprentice skills were contrasted to professionals’. We find that while novices tend to choose comfortable hand postures, experts privilege less comfortable, but error-free postures. As novices progress in their training, they adopt postures that reveal an incremental understanding of the skills and come close to professionals'. Notably, the fingers of the two hands start to move in coordination with a role distribution that follow handedness.
In parrallel, SAHR pursued fundamental advances in machine learning with application to robot control. The notion of bifurcation in dynamical systems was exploited to embed simultaneously periodic and discrete dynamics of motion in a single control law. This technique was applied to replicate swift switching from polishing to pick and place as displayed in watchmaking with a robot arm manipulator. Additionally, novel approaches to identify automatically control laws through the use of spectral clustering and manifold learning were developed.
Finally, the project made important efforts to ensure reproducibility of results. To this end, a benchmark of bimanual coordination in robot control was devised and published. The benchmark will serve to document progresses in the project but also more generally in robotics. Additionally, the longitudinal study's data have been made available on the project’s website alongside code for the analysis and modeling of the data.
In parrallel, SAHR pursued fundamental advances in machine learning with application to robot control. The notion of bifurcation in dynamical systems was exploited to embed simultaneously periodic and discrete dynamics of motion in a single control law. This technique was applied to replicate swift switching from polishing to pick and place as displayed in watchmaking with a robot arm manipulator. Additionally, novel approaches to identify automatically control laws through the use of spectral clustering and manifold learning were developed.
Finally, the project made important efforts to ensure reproducibility of results. To this end, a benchmark of bimanual coordination in robot control was devised and published. The benchmark will serve to document progresses in the project but also more generally in robotics. Additionally, the longitudinal study's data have been made available on the project’s website alongside code for the analysis and modeling of the data.
The SAHR project advances our understanding of skill acquisition in humans. Data collected during the SAHR longitudinal study provide a detailed and accurate measurement of motion and pressure exerted by the fingers, recorded in situ. These - in the field - data differ from traditional laboratory-based study of skill acquisition.
Analysis of the data have led us to propose a novel taxonomy of hand postures that differ from hand postures documented in existing taxonomies. We also offer, for the first time, a taxonomy of bimanual coordinated hand postures that reveals interesting information on the role than handedness plays in finger coordination. While handedness is well documented for arm movements, its role in fine manipulation has received much less attention.
Watchmaking is a craft known for its complexity. Through its longitudinal study of watchmaking apprenticeship, the SAHR project generates a unique and the first of its kind detailed recordings of this craft. These data will provide a historical account of the craft, should this craft come to disappear one day.
The SAHR project contributes to theoretical and experimental advances in machine learning and robotics. It pushes forward the use of dynamical systems as a mode to control robots. While learning dynamical systems was so far performed in a fully supervised manner, results from SAHR show that unsupervised learning techniques can also be used for the identification of the systems. This is advantageous, as the approach does not necessitate explicit labelling which is time consuming and relies on expert data.
The SAHR project bet that bifurcations, a well-known property of dynamical systems, could be used to good advantage to embed multiple control laws in a single system. This departs from traditional hybrid systems approaches. Results show that it is indeed possible to learn bi-stable systems with particular dynamics leading to fixed point attractors and other dynamics leading to limit cycles. The approach is, however, restricted to linear dynamics and efforts in the second part of the project will be dedicated to extend this to more complex non-linear dynamics.
The SAHR project’s ultimate goal is to devise a learning scheme to enable robots to automatically model a new skill through practice and to automate this skill through the use of closed-form construct via dynamical systems. Such a scheme will enable robots to acquire and adapt their manipulation skills. The SAHR project will aim to demonstrate the general applicability of this scheme by having robots transfer fine insertion and polishing skills to different materials and at different scales.
Analysis of the data have led us to propose a novel taxonomy of hand postures that differ from hand postures documented in existing taxonomies. We also offer, for the first time, a taxonomy of bimanual coordinated hand postures that reveals interesting information on the role than handedness plays in finger coordination. While handedness is well documented for arm movements, its role in fine manipulation has received much less attention.
Watchmaking is a craft known for its complexity. Through its longitudinal study of watchmaking apprenticeship, the SAHR project generates a unique and the first of its kind detailed recordings of this craft. These data will provide a historical account of the craft, should this craft come to disappear one day.
The SAHR project contributes to theoretical and experimental advances in machine learning and robotics. It pushes forward the use of dynamical systems as a mode to control robots. While learning dynamical systems was so far performed in a fully supervised manner, results from SAHR show that unsupervised learning techniques can also be used for the identification of the systems. This is advantageous, as the approach does not necessitate explicit labelling which is time consuming and relies on expert data.
The SAHR project bet that bifurcations, a well-known property of dynamical systems, could be used to good advantage to embed multiple control laws in a single system. This departs from traditional hybrid systems approaches. Results show that it is indeed possible to learn bi-stable systems with particular dynamics leading to fixed point attractors and other dynamics leading to limit cycles. The approach is, however, restricted to linear dynamics and efforts in the second part of the project will be dedicated to extend this to more complex non-linear dynamics.
The SAHR project’s ultimate goal is to devise a learning scheme to enable robots to automatically model a new skill through practice and to automate this skill through the use of closed-form construct via dynamical systems. Such a scheme will enable robots to acquire and adapt their manipulation skills. The SAHR project will aim to demonstrate the general applicability of this scheme by having robots transfer fine insertion and polishing skills to different materials and at different scales.