Periodic Reporting for period 2 - MuMMER (MultiModal Mall Entertainment Robot)
Periodo di rendicontazione: 2017-03-01 al 2018-08-31
Throughout the project, the robot will be deployed in Ideapark, a large public shopping mall in Finland: initially for short visits to aid in collaborative scenario development, co-design, and system evaluation, and later for a long-term field study in the 4th year of the project. Through our co-design approach, we will both study and foster acceptance of consumer robots and thus positively influence the consumer markets of service robots.
The overall objectives of the project include:
1. Developing an interactive robot for entertainment applications.
2. Involving stakeholders throughout the project in a co-design process.
3. Allowing the robot to perceive the world through its own built-in sensors.
4. Automatically learning strategies for the robot to interact with humans.
5. Moving and navigating safely and naturally in a crowded public space.
6. Developing new business models and opportunities for socially interactive robots in public spaces.
The results of MuMMER will take the following forms:
- A co-designed interactive mobile robot with entertainment features and behaviours that is able to interact naturally with humans in a public space.
- A set of concrete, detailed, tested use and business scenarios for a mobile entertainment robot in a shopping mall.
- A set of success criteria and evaluation strategies designed to evaluate the success of the robot in its designated tasks.
- A set of publicly available, reusable, state-of-the-art components for audiovisual scene processing, social signal processing, high-level action selection, and human-aware robot navigation.
All partners received the Pepper robot in June 2016. All technical partners then developed initial versions of the components which will combine to create the MuMMER system, and these were integrated into an initial interactive system that supports the target scenario identified by the co-design process. The Pepper robot hardware was evaluated in the context of the project needs, and a concrete plan developed for hardware and software updates to be made to Pepper to allow it to fully support the project research goals.
During Period 2, we identified and refined a concrete scenario relevant to the mall situation which supports the integration of state-of-the-art research from all partners. The scenario is based around guidance – i.e. helping users to find locations in the mall – but also includes aspects of interactive chat and entertainment. We carried out a human-human study to assess how the current mall guides carry out guidance tasks, and implemented a version of the MuMMER system that supports this guidance scenario, deploying it in the mall. Significant effort was made to develop a modified version of Pepper with hardware suitable for the mall environment.
We carried out regular studies in the mall measuring user acceptance of the robot, involving several hundred participants. In addition, the human-human study mentioned above and other work with stakeholders in the mall has continued. The primary focus in WP2 has been on updating and extending the perception components to support more robust and informative perception modules. This has involved head pose estimation from colour images, body landmarks from a depth sensor, audio processing, automatic speech recognition, close range perception, and, in particular, robust re-identification which allows to handle the different gestures and motion of Pepper and keeping the interaction history with the main interaction persons.
Work has continued in WP3 to use the output of the perception modules to estimate the user’s social state, and in WP5 for tasks such as semantic route planning. In WP3, we have developed and evaluated components for generating non-verbal behaviour of the robot designed to produce particular social effects on the user. In WP4, we have developed a new dialogue system called Alana, a scalable and highly customizable open-domain dialogue system comprised of several interchangeable components, combined into 4 basic modules. The system combines hand-crafted rules with machine learning models trained on carefully chosen datasets. The Alana system, a finalist in the 2017 and 2018 Amazon Alexa Challenge contests, forms the basis of the interactions supported by the MuMMER robot.
Work in WP5 has concentrated on enhancing and integrating all the building blocks involved in the navigation and localisation tasks. Delivered software includes a Shared Visual Perspective planner, a Human Aware Navigation planner, a Situation Assessment Component, a robot supervision component specifically designed for the selected guidance task, and a component to implement localisation and relocalisation using markers in the environment. WP1's stakeholder workshops have continued to contribute to this effort. In WP8, we have outlined four possible use scenarios for a robot system like that being developed in MuMMER, and have discussed possible business advantages of each scenario.
In addition, as part of the co-design process together with stakeholders, we have also begun developing a set of metrics to evaluate the success of a socially interactive public-space robot. As such robots are more widely deployed, these metrics will play a crucial role in assessing the performance of the robots in various contexts.