Skip to main content

MultiModal Mall Entertainment Robot

Periodic Reporting for period 2 - MuMMER (MultiModal Mall Entertainment Robot)

Periodo di rendicontazione: 2017-03-01 al 2018-08-31

MuMMER (MultiModal Mall Entertainment Robot) is a four-year project with the overall goal of developing a humanoid robot (based on Softbank's Pepper platform) that can interact autonomously and naturally in the dynamic environments of a public shopping mall, providing an engaging and entertaining experience to the general public. Using co-design methods, we will work together with stakeholders including customers, retailers, and business managers to develop truly engaging robot behaviours. Crucially, our robot will exhibit behaviour that is socially appropriate: combining speech-based interaction with non-verbal communication and human-aware navigation. To support this behaviour, we will develop and integrate new methods from audiovisual scene processing, social-signal processing, high-level action selection, and human-aware robot navigation.

Throughout the project, the robot will be deployed in Ideapark, a large public shopping mall in Finland: initially for short visits to aid in collaborative scenario development, co-design, and system evaluation, and later for a long-term field study in the 4th year of the project. Through our co-design approach, we will both study and foster acceptance of consumer robots and thus positively influence the consumer markets of service robots.

The overall objectives of the project include:
1. Developing an interactive robot for entertainment applications.
2. Involving stakeholders throughout the project in a co-design process.
3. Allowing the robot to perceive the world through its own built-in sensors.
4. Automatically learning strategies for the robot to interact with humans.
5. Moving and navigating safely and naturally in a crowded public space.
6. Developing new business models and opportunities for socially interactive robots in public spaces.

The results of MuMMER will take the following forms:

- A co-designed interactive mobile robot with entertainment features and behaviours that is able to interact naturally with humans in a public space.
- A set of concrete, detailed, tested use and business scenarios for a mobile entertainment robot in a shopping mall.
- A set of success criteria and evaluation strategies designed to evaluate the success of the robot in its designated tasks.
- A set of publicly available, reusable, state-of-the-art components for audiovisual scene processing, social signal processing, high-level action selection, and human-aware robot navigation.
In the first period, we made progress on many fronts. We held the first workshops and focus groups with customers, retailers, and management at Ideapark. Workshop results were used to specify an initial target scenario for the robot system, and to develop initial versions of the acceptance questionnaire to be used to track user reactions to the robot throughout the project.

All partners received the Pepper robot in June 2016. All technical partners then developed initial versions of the components which will combine to create the MuMMER system, and these were integrated into an initial interactive system that supports the target scenario identified by the co-design process. The Pepper robot hardware was evaluated in the context of the project needs, and a concrete plan developed for hardware and software updates to be made to Pepper to allow it to fully support the project research goals.

During Period 2, we identified and refined a concrete scenario relevant to the mall situation which supports the integration of state-of-the-art research from all partners. The scenario is based around guidance – i.e. helping users to find locations in the mall – but also includes aspects of interactive chat and entertainment. We carried out a human-human study to assess how the current mall guides carry out guidance tasks, and implemented a version of the MuMMER system that supports this guidance scenario, deploying it in the mall. Significant effort was made to develop a modified version of Pepper with hardware suitable for the mall environment.

We carried out regular studies in the mall measuring user acceptance of the robot, involving several hundred participants. In addition, the human-human study mentioned above and other work with stakeholders in the mall has continued. The primary focus in WP2 has been on updating and extending the perception components to support more robust and informative perception modules. This has involved head pose estimation from colour images, body landmarks from a depth sensor, audio processing, automatic speech recognition, close range perception, and, in particular, robust re-identification which allows to handle the different gestures and motion of Pepper and keeping the interaction history with the main interaction persons.

Work has continued in WP3 to use the output of the perception modules to estimate the user’s social state, and in WP5 for tasks such as semantic route planning. In WP3, we have developed and evaluated components for generating non-verbal behaviour of the robot designed to produce particular social effects on the user. In WP4, we have developed a new dialogue system called Alana, a scalable and highly customizable open-domain dialogue system comprised of several interchangeable components, combined into 4 basic modules. The system combines hand-crafted rules with machine learning models trained on carefully chosen datasets. The Alana system, a finalist in the 2017 and 2018 Amazon Alexa Challenge contests, forms the basis of the interactions supported by the MuMMER robot.

Work in WP5 has concentrated on enhancing and integrating all the building blocks involved in the navigation and localisation tasks. Delivered software includes a Shared Visual Perspective planner, a Human Aware Navigation planner, a Situation Assessment Component, a robot supervision component specifically designed for the selected guidance task, and a component to implement localisation and relocalisation using markers in the environment. WP1's stakeholder workshops have continued to contribute to this effort. In WP8, we have outlined four possible use scenarios for a robot system like that being developed in MuMMER, and have discussed possible business advantages of each scenario.
In the first year of the project, we have developed a modular architecture to allow state-of-the-art software components to work together to support socially intelligent interaction with the MuMMER robot, and have also integrated initial versions of all components into the architecture. These components address tasks such as audiovisual processing, social signal processing, interaction management and action selection, and interactive navigation and motion planning -- all of which are necessary to support the overall task of socially intelligent and engaging interaction with a robot in a public space. With these advanced software components operating together on the Pepper robot, the most widely available current social robot platform, the potential impact is large: a robot that is able to support this sort of socially intelligent interaction can be deployed in a wide variety of public contexts.

In addition, as part of the co-design process together with stakeholders, we have also begun developing a set of metrics to evaluate the success of a socially interactive public-space robot. As such robots are more widely deployed, these metrics will play a crucial role in assessing the performance of the robots in various contexts.