Two use cases from our end-users guided the project: on-demand sample transportation and bioassay sample flow automation. The research outcomes focused on developing the robotic capabilities required to perform these mobile manipulation tasks.
The project leveraged two distinct robot platforms. A research system from ABB used to validate the technical advancements of the project in the mobile manipulation space. And a custom-built robot by IDMind focusing on exploring and evaluating robot shape and behaviour when interacting with the people present in a hospital environment.
A system capable of autonomously performing mobile manipulation tasks in a complex and dynamic environment such as a hospital requires innovations in every part of the robotics software stack. This is reflected in the work packages, which span the entire gamut from low-level perception to high-level reasoning. The perception work focused on a novel object representation to use in a flexible object database. The resulting representation is the foundation for a state-of-the-art pose estimation pipeline that only requires images captured around objects of interest. Similarly, the mapping research focused on an object-based map representation to handle dynamics at various time scales and leverage contextual object and language queues for localization. An active exploration component further enables the object-centric approach by autonomously improving object reconstruction quality during mapping. To safely navigate in dynamic environments such as a hospital, research focused on learning systems to adapt to the encountered behaviours and safely interact with people. A secondary component focused on the efficient and preference- and constraint-aware scheduling of fleets of robots. In manipulation, novel approaches to teach robots novel skills have been developed, as well as the ability to grasp novel unseen objects reliably. Bringing the motion and manipulation capabilities of the robot together, our work on novel compliant control approaches allows for dexterous bi-manual manipulation, both prehensile and non-prehensile, all while being compliant to external disturbances. This enables robust manipulation and safe navigation in crowded spaces.
To enable a robot equipped with these novel skills and competencies to operate effectively in a hospital environment, it must fit within that environment and communicate with people. Therefore, we investigated the design language of a robot that will be accepted in such an environment and modes of non-verbal communication. This resulted in the design and construction of a novel robot and prototypes of a shape-changing robot.
Over the course of the project, 80 publications in the top robotics, vision, and machine learning conferences have been published, disseminating the research progress achieved by Harmony. On the project's GitHub page, more than 15 open-source packages are available for researchers to utilize in their own developments. A wide range of outreach events, including workshops, conferences, symposiums, exhibitions, and general events, are organized to engage with academic professionals, medical professionals, and the general public.
The end-users gained invaluable insight into the potential future capabilities to expect and plan for with robotic systems. The industrial partners gained technical expertise and created novel commercial opportunities. The robot developed by IDMind is being further developed to become a product in the near future.
