Periodic Reporting for period 3 - HARMONY (Enhancing Healthcare with Assistive Robotic Mobile Manipulation)
Berichtszeitraum: 2023-01-01 bis 2024-06-30
Harmony will develop assistive robotic mobile manipulation technologies for use in hospital environments. Our two targeted use cases, 1) the automation of on-demand delivery tasks around the hospital, and 2) the automation of bioassay sample flow, highlight existing processes where there is a need for fast, reliable and flexible automation to undertake the dull and repetitive tasks that are currently conducted by over-qualified staff. While existing systems can automate parts of these processes, these form “islands of automation” that are limited in scope, rigid to changing demands, and still rely on staff to manually distribute goods and samples across the islands. Mobile manipulation technology is a compelling solution to this problem since it offers the capability to bridge these gaps while maintaining a high degree of flexibility to adjust to varying service demands and adapt to different user requirements and preferences.
Two use cases from our end-users guided the project: on-demand sample transportation and bioassay sample flow automation. The research outcomes focused on developing the robotic capabilities required to perform these mobile manipulation tasks.
The project leveraged two distinct robot platforms. A research system from ABB used to validate the technical advancements of the project in the mobile manipulation space. And a custom-built robot by IDMind focusing on exploring and evaluating robot shape and behaviour when interacting with the people present in a hospital environment.
A system capable of autonomously performing mobile manipulation tasks in a complex and dynamic environment such as a hospital requires innovations in every part of the robotics software stack. This is reflected in the work packages, which span the entire gamut from low-level perception to high-level reasoning. The perception work focused on a novel object representation to use in a flexible object database. The resulting representation is the foundation for a state-of-the-art pose estimation pipeline that only requires images captured around objects of interest. Similarly, the mapping research focused on an object-based map representation to handle dynamics at various time scales and leverage contextual object and language queues for localization. An active exploration component further enables the object-centric approach by autonomously improving object reconstruction quality during mapping. To safely navigate in dynamic environments such as a hospital, research focused on learning systems to adapt to the encountered behaviours and safely interact with people. A secondary component focused on the efficient and preference- and constraint-aware scheduling of fleets of robots. In manipulation, novel approaches to teach robots novel skills have been developed, as well as the ability to grasp novel unseen objects reliably. Bringing the motion and manipulation capabilities of the robot together, our work on novel compliant control approaches allows for dexterous bi-manual manipulation, both prehensile and non-prehensile, all while being compliant to external disturbances. This enables robust manipulation and safe navigation in crowded spaces.
To enable a robot equipped with these novel skills and competencies to operate effectively in a hospital environment, it must fit within that environment and communicate with people. Therefore, we investigated the design language of a robot that will be accepted in such an environment and modes of non-verbal communication. This resulted in the design and construction of a novel robot and prototypes of a shape-changing robot.
Over the course of the project, 80 publications in the top robotics, vision, and machine learning conferences have been published, disseminating the research progress achieved by Harmony. On the project's GitHub page, more than 15 open-source packages are available for researchers to utilize in their own developments. A wide range of outreach events, including workshops, conferences, symposiums, exhibitions, and general events, are organized to engage with academic professionals, medical professionals, and the general public.
The end-users gained invaluable insight into the potential future capabilities to expect and plan for with robotic systems. The industrial partners gained technical expertise and created novel commercial opportunities. The robot developed by IDMind is being further developed to become a product in the near future.
The project leveraged two distinct robot platforms. A research system from ABB used to validate the technical advancements of the project in the mobile manipulation space. And a custom-built robot by IDMind focusing on exploring and evaluating robot shape and behaviour when interacting with the people present in a hospital environment.
A system capable of autonomously performing mobile manipulation tasks in a complex and dynamic environment such as a hospital requires innovations in every part of the robotics software stack. This is reflected in the work packages, which span the entire gamut from low-level perception to high-level reasoning. The perception work focused on a novel object representation to use in a flexible object database. The resulting representation is the foundation for a state-of-the-art pose estimation pipeline that only requires images captured around objects of interest. Similarly, the mapping research focused on an object-based map representation to handle dynamics at various time scales and leverage contextual object and language queues for localization. An active exploration component further enables the object-centric approach by autonomously improving object reconstruction quality during mapping. To safely navigate in dynamic environments such as a hospital, research focused on learning systems to adapt to the encountered behaviours and safely interact with people. A secondary component focused on the efficient and preference- and constraint-aware scheduling of fleets of robots. In manipulation, novel approaches to teach robots novel skills have been developed, as well as the ability to grasp novel unseen objects reliably. Bringing the motion and manipulation capabilities of the robot together, our work on novel compliant control approaches allows for dexterous bi-manual manipulation, both prehensile and non-prehensile, all while being compliant to external disturbances. This enables robust manipulation and safe navigation in crowded spaces.
To enable a robot equipped with these novel skills and competencies to operate effectively in a hospital environment, it must fit within that environment and communicate with people. Therefore, we investigated the design language of a robot that will be accepted in such an environment and modes of non-verbal communication. This resulted in the design and construction of a novel robot and prototypes of a shape-changing robot.
Over the course of the project, 80 publications in the top robotics, vision, and machine learning conferences have been published, disseminating the research progress achieved by Harmony. On the project's GitHub page, more than 15 open-source packages are available for researchers to utilize in their own developments. A wide range of outreach events, including workshops, conferences, symposiums, exhibitions, and general events, are organized to engage with academic professionals, medical professionals, and the general public.
The end-users gained invaluable insight into the potential future capabilities to expect and plan for with robotic systems. The industrial partners gained technical expertise and created novel commercial opportunities. The robot developed by IDMind is being further developed to become a product in the near future.
Harmony will develop robust, flexible and safe autonomous mobile manipulation technology for use in human-centred environments. We will bring together a broad spectrum of robotics research, blending the latest work in areas such as object-based perception and learning for manipulation (currently demonstrated at TRLs 1-2) with established frameworks such as feature-based SLAM (demonstrated at TRL 5). Furthermore, the industry and end-user expertise present in our consortium will enable us to create and demonstrate use-inspired mobile manipulation tools at TRLs 5-6 (i.e. technology validated and demonstrated in relevant environment) with a clear route to higher TRLs.
Specifically, Harmony is extending the state of the art by:
- Formalising object-based (as opposed to classical feature-based) world representations for robotic perception,
- Developing novel robot localisation and mapping algorithms that exploit object-based world representations,
- Providing new algorithms for coupling task scheduling and motion planning that provide adaptive, congestion-free plans in shared human spaces,
- Developing an immersive control interface that provides a natural and intuitive way for medical staff to deliver and verify new robotic manipulation capabilities at aided, semi-, and autonomous levels,
- Developing robust and compliant whole-body motion planning and control for interacting with unknown objects while in close proximity or direct collaboration with human co-workers,
- Delivering strict guarantees on aspects such as safety, accuracy and patient privacy during navigation, handover, carrying and co-working tasks.
Together, the capabilities that we develop throughout Harmony will lead to faster industry take-up of assistive robotic technologies in healthcare environments and beyond.
Specifically, Harmony is extending the state of the art by:
- Formalising object-based (as opposed to classical feature-based) world representations for robotic perception,
- Developing novel robot localisation and mapping algorithms that exploit object-based world representations,
- Providing new algorithms for coupling task scheduling and motion planning that provide adaptive, congestion-free plans in shared human spaces,
- Developing an immersive control interface that provides a natural and intuitive way for medical staff to deliver and verify new robotic manipulation capabilities at aided, semi-, and autonomous levels,
- Developing robust and compliant whole-body motion planning and control for interacting with unknown objects while in close proximity or direct collaboration with human co-workers,
- Delivering strict guarantees on aspects such as safety, accuracy and patient privacy during navigation, handover, carrying and co-working tasks.
Together, the capabilities that we develop throughout Harmony will lead to faster industry take-up of assistive robotic technologies in healthcare environments and beyond.