Periodic Reporting for period 4 - CLOTHILDE (CLOTH manIpulation Learning from DEmonstrations)
Berichtszeitraum: 2022-07-01 bis 2023-12-31
Textile objects pervade human environments and their versatile manipulation by robots opens up a whole range of possibilities, from increasing the autonomy of elderly and disabled people, housekeeping and hospital logistics, to novel automation in the clothing internet business. Although efficient procedures exist for the robotic handling of rigid objects and the virtual rendering of deformable objects, cloth manipulation in the real world has proven elusive, because the vast number of degrees of freedom involved in non-rigid deformations leads to unbearable uncertainties in perception and action outcomes.
The CLOTHILDE project has put forth a well-founded framework for robotized cloth manipulation, from theoretical developments combining computational topology and machine learning, to implementation in robot prototypes.
A systematic analysis of cloth manipulation actions has led to a taxonomy of grasps and manipulation primitives, which has proven very handy to define benchmarks and to design versatile grippers specific for cloth manipulation.
The state of textile objects and their transformations under given actions have been represented in a compact operational way (i.e. encoding task-relevant topological changes), which has permitted planning cloth manipulation tasks, both one handed and bimanual.
The above representation of cloth state has been used to semantically label sequences of human manipulations using virtual reality together with a cloth simulator.
A physically-accurate model of textiles, especially oriented to simulation of cloth manipulation by robots, has been developed. The accuracy and low computational cost of simulation are crucial for sim-to-real transfer and for viable model-predictive control of cloth dynamic manipulation.
An approach has been developed to teach manipulation skills to a robot from human demonstrations, subsequently refined through reinforcement learning. The skills are encoded as parameterised dynamical systems and a predictive controller based on a model of the cloth dynamics is used.
This approach has been implemented and tested in three applications: cloth folding, putting a textile cover on a mattress, and helping people to dress.
In summary, the CLOTHILDE project has laid solid foundations for the representation, simulation, learning, control and manipulation of cloth items by robots, and has demonstrated their usefulness by implementing robot prototypes accomplishing three cloth handling tasks.
The CLOTHILDE project has put forth a well-founded framework for robotized cloth manipulation, from theoretical developments combining computational topology and machine learning, to implementation in robot prototypes.
A systematic analysis of cloth manipulation actions has led to a taxonomy of grasps and manipulation primitives, which has proven very handy to define benchmarks and to design versatile grippers specific for cloth manipulation.
The state of textile objects and their transformations under given actions have been represented in a compact operational way (i.e. encoding task-relevant topological changes), which has permitted planning cloth manipulation tasks, both one handed and bimanual.
The above representation of cloth state has been used to semantically label sequences of human manipulations using virtual reality together with a cloth simulator.
A physically-accurate model of textiles, especially oriented to simulation of cloth manipulation by robots, has been developed. The accuracy and low computational cost of simulation are crucial for sim-to-real transfer and for viable model-predictive control of cloth dynamic manipulation.
An approach has been developed to teach manipulation skills to a robot from human demonstrations, subsequently refined through reinforcement learning. The skills are encoded as parameterised dynamical systems and a predictive controller based on a model of the cloth dynamics is used.
This approach has been implemented and tested in three applications: cloth folding, putting a textile cover on a mattress, and helping people to dress.
In summary, the CLOTHILDE project has laid solid foundations for the representation, simulation, learning, control and manipulation of cloth items by robots, and has demonstrated their usefulness by implementing robot prototypes accomplishing three cloth handling tasks.
The work carried out in the 5 WPs proposed in the DoA (Theory of Cloth Manipulation, Perception/Representation, Manipulation/Planning, Learning, and Integration/Prototype Development) has crystallized in top-rank publications as well as lab demonstrators that have led to international collaborations, some new projects related to cloth manipulation and benchmarking, and have aroused interest from several companies. Below we mention the main achievements attained. Please refer to the project website (https://clothilde.iri.upc.edu) for details.
A physically-accurate model of textiles has been developed, especially oriented to simulation of cloth manipulation by robots. The model is used within a model-predictive control approach to perform a target cloth dynamic manipulation, and holds great promise to bootstrap learning approaches through simulation and surmount the sim-to-real gap.
Two topological representations of cloth state, based on cell complexes and dGLI coordinates, have been developed. This is also a successful response to another challenge repeatedly raised in the field, since the high dimensionality of C-spaces of textiles makes it impossible to apply the methods developed for the 6D C-spaces of rigid objects.
A complete system to teach cloth manipulation skills to a robot has been designed and implemented. Novel contributions include reinforcement learning algorithms that exploit symmetries, Gaussian process dynamical models coupled with variable impedance control to learn force profiles, and quadratic dynamic matrix control for fast cloth manipulation.
84 works have been published within CLOTHILDE, 4 PhD theses have been completed and 3 more are expected to be defended within 2024. See https://clothilde.iri.upc.edu/node/3 for graphs and details.
Concerning technology transfer, we have established contact with two companies in the textile sector to assess their needs in the manipulation of deformable materials. After visiting the large warehouses these companies have near Barcelona, we have jointly submitted a European project proposal coordinated by us, which if granted, would apply our expertise developed in CLOTHILDE to reverse logistics challenges in textile companies’ real scenarios.
The collaboration with CLOTHILDE’s external advisor Prof. Pokorny resulted in the joint Horizon Europe project SoftEnable, which entails combining his expertise in caging theory with the methods developed within CLOTHILDE in healthcare and agri-food use-cases.
We also established a fruitful collaboration with the company PAL Robotics for the use and testing of their robots, as well as the development of new applications in cloth manipulation scenarios.
A physically-accurate model of textiles has been developed, especially oriented to simulation of cloth manipulation by robots. The model is used within a model-predictive control approach to perform a target cloth dynamic manipulation, and holds great promise to bootstrap learning approaches through simulation and surmount the sim-to-real gap.
Two topological representations of cloth state, based on cell complexes and dGLI coordinates, have been developed. This is also a successful response to another challenge repeatedly raised in the field, since the high dimensionality of C-spaces of textiles makes it impossible to apply the methods developed for the 6D C-spaces of rigid objects.
A complete system to teach cloth manipulation skills to a robot has been designed and implemented. Novel contributions include reinforcement learning algorithms that exploit symmetries, Gaussian process dynamical models coupled with variable impedance control to learn force profiles, and quadratic dynamic matrix control for fast cloth manipulation.
84 works have been published within CLOTHILDE, 4 PhD theses have been completed and 3 more are expected to be defended within 2024. See https://clothilde.iri.upc.edu/node/3 for graphs and details.
Concerning technology transfer, we have established contact with two companies in the textile sector to assess their needs in the manipulation of deformable materials. After visiting the large warehouses these companies have near Barcelona, we have jointly submitted a European project proposal coordinated by us, which if granted, would apply our expertise developed in CLOTHILDE to reverse logistics challenges in textile companies’ real scenarios.
The collaboration with CLOTHILDE’s external advisor Prof. Pokorny resulted in the joint Horizon Europe project SoftEnable, which entails combining his expertise in caging theory with the methods developed within CLOTHILDE in healthcare and agri-food use-cases.
We also established a fruitful collaboration with the company PAL Robotics for the use and testing of their robots, as well as the development of new applications in cloth manipulation scenarios.
The physically-accurate model of textiles developed entails progress beyond the state of the art, as it provides a successful solution to an open challenge mentioned by several top researchers in the field of robotic cloth manipulation. Most cloth simulators nowadays rely on meshes and finite-element methods, which are either not accurate enough or too computationally costly to be used in a robotic manipulation pipeline.
The two topological representations of cloth state developed have shown great potential, which will even increase if a way to combine them is found that robustly characterizes regions of configuration space corresponding to cloth macro-states. We envisage the combination to be based on determinants (or tetrahedra volumes), and the key difficulties are to ensure the clean partition of C-space and the continuity of the representation for adjacent macro-states.
The system to teach cloth manipulation skills to a robot is a significant advance in the state of the art already envisaged in the CLOTHILDE proposal. Its many components have been developed, from perception and data acquisition using virtual reality, to modelling, representation and simulation, model-based and data-driven control strategies, learning from demonstration and reinforcement, and manipulation planning and execution. The approach has been tested in the three application domains foreseen in the proposal: cloth folding, putting an elastic cover, and helping to dress.
Two unexpected outcomes of the project have been the strong ethics commitment of the team and the start of a research line on robot hand design specifically for cloth manipulation. The former has given rise to many dissemination and teaching activities, even leading to the incorporation of a philosopher who developed her thesis in our group supported by a competitive scholarship she obtained, and who now has remained as a postdoc in our group. The latter originated with the initial hiring of a postdoc, whose field of research was robotic grasping and robot hand design. She has designed two gripper prototypes and built them using 3D printing. This research line has found continuation in the project SoftEnable where a new gripper to open sealed bags is under development.
The two topological representations of cloth state developed have shown great potential, which will even increase if a way to combine them is found that robustly characterizes regions of configuration space corresponding to cloth macro-states. We envisage the combination to be based on determinants (or tetrahedra volumes), and the key difficulties are to ensure the clean partition of C-space and the continuity of the representation for adjacent macro-states.
The system to teach cloth manipulation skills to a robot is a significant advance in the state of the art already envisaged in the CLOTHILDE proposal. Its many components have been developed, from perception and data acquisition using virtual reality, to modelling, representation and simulation, model-based and data-driven control strategies, learning from demonstration and reinforcement, and manipulation planning and execution. The approach has been tested in the three application domains foreseen in the proposal: cloth folding, putting an elastic cover, and helping to dress.
Two unexpected outcomes of the project have been the strong ethics commitment of the team and the start of a research line on robot hand design specifically for cloth manipulation. The former has given rise to many dissemination and teaching activities, even leading to the incorporation of a philosopher who developed her thesis in our group supported by a competitive scholarship she obtained, and who now has remained as a postdoc in our group. The latter originated with the initial hiring of a postdoc, whose field of research was robotic grasping and robot hand design. She has designed two gripper prototypes and built them using 3D printing. This research line has found continuation in the project SoftEnable where a new gripper to open sealed bags is under development.