Donating Robots a Theory of Mind

Project Information

DoRoThy

Grant agreement ID: 657227

DOI 10.3030/657227

Closed project

Start date 7 September 2015

End date 6 September 2017

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost € 195 454,80

EU contribution € 195 454,80

Coordinated by

UNIVERSITY OF PLYMOUTH
United Kingdom

Periodic Reporting for period 1 - DoRoThy (Donating Robots a Theory of Mind)

Reporting period: 2015-09-07 to 2017-09-06

Summary of the context and overall objectives of the project

"Human social dynamics rely upon the ability to effectively attribute beliefs,
goals and percepts to other people. This lays at the core of human interactions:
normal human social interactions depend upon the recognition of other sensory
perspectives, the understanding of other mental states, and the recognition of
complex non-verbal cues of attention and emotional state.

With the rapid development of social robotics, meaning robots that interact with
humans in usual human environments like homes, transferring these cognitive
skills to robots is an important, if difficult, scientific challenge, with a significant
societal impact with regard to our future interactions with robots. This
scientific endeavour is explicitly set as one of the EU priorities within the
Horizon 2020 framework, which emphasizes the need to endow artificial systems
with new cognitive capabilities, beyond ""repetitive problem solving"".


In this context, the DoRoThy project aims first at advancing our understanding
of the complex socio-cognitive mechanisms that underpin human social
interactions, and, second, to investigate how such mechanisms could by
applied to social robots.
"

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Evaluating human-robot social interactions in a rigorous manner is notoriously
difficult: studies are either conducted in labs with constrained protocols to
allow for robust measurements and a degree of replicability, but at the cost of
ecological validity; or in the wild, which leads to superior experimental
realism, but often with limited replicability and at the expense of rigorous
interaction metrics.

In the frame of the DoRoThy project, we have conceptualised,
designed, implemented and applied a novel interaction paradigm, designed to
elicit rich and varied social interactions while having desirable scientific
properties (replicability, clear metrics). This paradigm focuses on both child-child and child-robot
interactions, and builds on what we call a sandboxed free-play environment.

The free-play sandbox is based on free play interactions: Pairs of children
(4-8 years old in our experiments) are invited to freely draw and interact with items
displayed on an interactive table, without any explicit goal set by the
experimenter. The task is designed so that
children can engage in open-ended and non-directive play, yet it is
sufficiently constrained to be suitable for recording, and allows the
reproduction of social behaviour by an artificial agent in comparable
conditions.

Our interactive table is equipped with 3D cameras recording the faces and postures of the children,
and the quantity and thoughtfulness of information logged allows
to keep a track of every interaction happening around the game.

These advantages, combined with the openness of the proposed task, make
this setup a powerful tool to observe and quantify a large range of
social behaviours expressed by children when interacting in a natural
environment.

Using this innovative platform, we have conducted a large scale data collection campaign,
to build a first-in-this-kind dataset of social interactions, called the PInSoRo dataset: 120 children, from 4 to 8 years old,
have been recorded while playing either together, or with a robot. 45 hours of 3D video and audio
have been acquired, including close to 2 millions frames of faces. Using a new coding scheme,
developed during the project, this dataset is now being annotated by hand with the hundreds of
social micro-episodes that took place between the children and the robot.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The two main contribution of the DoRoThy project are the 'free play sandbox' experimental methodology and the new PInSoRo dataset of social interactions.

The free play sandbox paradigm is expected to play an important role for the future development of social robotics, as it offers one possible solution to the critical issue of the experimental validation of social robotics research: by offering an experimental platform eliciting natural social behaviours, relatively unconstrainted, essentially non-deterministic, yet sufficiently well defined to be measurable and easily reproducible, we have created an important tool to scaffold future research in social HRI.

The PInSoRo dataset of social interaction, created during the project, is expected to have a major impact on the way we study human-robot interaction, and more broadly, human-human interactions. For the first time, we make available to the broad academic community a dataset of social interactions rigorous enough and large enough to enable machine learning and data-mining at the behavioural level.

By releasing the dataset under an open-data license, we hope to reach a large audience, in the social robotics community, and beyond, in the broader social psychology academic community. By the end of the project, the lead DoRoThy researcher has been already invited to give seminars and talks in about 10 institutions worldwide, including a keynote during the 2017 edition of the prestigious Fall Symposium of the Association for the Advance of Artificial Intelligence, where the PInSoRo dataset will be officially presented. Details and access to the dataset are available on the project website, https://freeplay-sandbox.github.io/

Free play interaction between a child and a robot

Child-child or child-robot interaction; an interactive 'playground'; large dataset of recordings

Analysing social interactions between a pair of children