A community of surface bots for creative storytelling

Project Information

COBOTNITY

Grant agreement ID: 701991

DOI

10.3030/701991

Project closed

EC signature date 22 February 2016

Start date 1 March 2016

End date 28 February 2018

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 165 598,80

EU contribution

€ 165 598,80

Coordinated by

UNIVERSITEIT TWENTE
Netherlands

Periodic Reporting for period 1 - COBOTNITY (A community of surface bots for creative storytelling)

Reporting period: 2016-03-01 to 2018-02-28

Summary of the context and overall objectives of the project

The project coBOTnity aims to provide an affordable community of hybrid artificial agents (called surfacebots) to be used in collaborative creative storytelling. The project will explore how such surfacebot community can effectively be used to foster creative thinking and social awareness in children.
With these general objectives in mind, there are both technical and research issues/problems to be addressed. On the one hand, technical aspects such as building an IT system that enables children-robot interaction to enable them to create stories in play. The system attempts to bring an affordable and expandable implementation, which uses cheap available robotic bases, 3D printed cases, cheap tablets and commodity webcams for controlling the robots. In this way, the project opens opportunities to be used not only in a research context but also in schools or even at home with a limited budget.
On the other hand, the project has carried out research with the prototypes built in order to find out more on how children use the given technology, and then improve the design of the interactive setting in future developments. The research has focused on creating stories and facilitating social awareness and emotional understanding.
The importance of the project lies on its capacity to strengthen and promote a change in the educational landscape: from occasional supporting tools at schools to actual creative learning educational models in Europe. We are exploring how innovative technology can be used in reshaping education for the 21st century skills, including creativity, collaboration, digital literacy and social-emotional development.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Surfacebot embodiment
------------------------------
We have developed the concept and implementation of surfacebot. It is intended as a movable character that is embodied in a tablet display attached to a robotic platform (refer to the attached sample pictures). It is capable of rendering facial expressions, speaking short utterances, moving, and displaying visual assets.
A variable number of surfacebots can be dynamically added in a play activity. There a number of editors to create user assets that can be included in activities. These support the inclusion of Text-to-Speech, recorded audio, pictures and drawings based on images from the internet or drawn on the tablet itself.

Smart Navigation
---------------------
The system prototype is expanded with smart navigation features to prevent children spending cognitive resources in driving bots. Children can give single destinations or paths for the bots to follow. When giving paths, users can include modifiers at any point/landmarks in the planned path.

Enhanced storytelling setting with smart toys
--------------------------------------------------------
We have explored how to mediate interaction in the storytelling setting via a tablet app. The features in the enhanced setting support children in developing storytelling skills, helping them to structure the creation of stories and reflect on the links between the different events. The app facilitates the transition from free play towards more reflection in the child storytelling process.

Social-emotional development support
------------------------------------------------
Enhancing children’s social awareness and their responses on social issues is also important. Through stories, they can safely experience social situations. The platform has been used in an experiment to explore how social-emotional development can be affected by the format of the storytelling table and the level of assistance-responsiveness of the system. Storytelling tables enhanced by technology can be useful to reinforce emotional understanding and story event recall.

Creative dancing tablets
------------------------------
Following user-centered methods, we have designed, implemented and evaluated a dancing robotic tablet prototype for co-creative human-robot interaction. Two types of autonomous robot behavior were considered as creativity support and evaluated in a user study. While imitation behavior was perceived as more intelligent; the generation behavior that attempted to challenge users and be different to the users’ input led to a greater variety of gestures. Observations support the idea that users were inspired by the robot’s input to some extent.

All these components would not have been possible without the coordinated effort of bachelor/master students at the University of Twente, who implemented some parts or carried out field studies or helped in the technology exhibitions. We thank Roberto Campisi, Luce Sandfort, Wouter Timmermans, Wouter van Veelen, Silke ter Stal, Federico Fabiano, Hannah Pelikan, Jelle Pingen, Judith Zissoldt, Iris Visser, and Jaebok Kim for their hard work. As well, we are very grateful to teachers, parents/tutors and children who participated at different stages of the project, by giving input or testing the prototypes.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The state of the art has not typically included the possibility to bring creation by users and the flexibility given by using tablet apps with robotic characters. This also means that the platform can be reused by end-users and expanded by developers with additional app features. The research studies carried out on locality and at the University of Twente have fed the design process of both the interactive setting and the apps developed in the project.
Several project prototypes have been presented and exhibited at different events: e.g. Twente Science Night, celebrated in Enschede in The Netherlands, with the title The World of Tomorrow/Scientific Exposition, Drongo Festival, HMI Open days, Lynn Packwood's Demo day, posters at CTIT, etc.
By the participation in the activities mentioned above, either exhibitions or playful user studies, the project has had a higher societal impact regarding the active participation of stakeholders and users, who had the opportunity to try the technology and play. We also expect to continue collaborating with teachers who engaged at several project stages, who can consider using specific components of the project as a tool in class.
Furthermore, ten bachelor and master students from several programmes at the University of Twente have been directly involved in the project, carrying out their student assignments in the context of the project, gaining experience and being trained for the science and knowledge based labour market. Two more bachelor students are starting their final project to extend the results obtained in the project.
In the promotion of the Marie Skłodowska-Curie Actions (MSCA), the experienced researcher gave a talk to prospective MSCA applicants, offering his experience on the application and publicizing the MSCA-IF.
Additionally, versions of the prototypes were also demonstrated at the international conferences ACM Creativity and Cognition celebrated in Singapore, ACM Tangible, Embedded, and Embodied Interaction held in Yokohama (Japan), or Interactive Digital Storytelling in Madeira (Portugal). Our work has also been disseminated in the form of scientific publications, and presented at international conferences.

storytelling table for socialemotional development

sample character on screen

welcome screen (control app)

sample editors (II)

smart navigation control app

tracked surfacebot

surfacebots at play

sample editors (I)