Periodic Reporting for period 1 - COBRA (COnversational BRAins)
Periodo di rendicontazione: 2020-02-01 al 2022-01-31
COBRA aims to train the next generation of researchers to accurately characterize and model the linguistic, cognitive and brain mechanisms deployed by human speakers in conversational interactions with human interlocutors as well as artificial dialog systems. It relies on a cross-sectoral international network of 11 world-level academic research centers and 4 non-academic partners with 3 fast-developing SMEs and 1 world-level company. The partners' unique combined expertise and high complementarity allows COBRA to offer 15 ESRs an excellent training programme as well as strong exposure to the non-academic sector in the emerging field of conversational brains. Training covers scientific and technical skills, from the joint monitoring of brain and physiological activities in two or more people talking to each other to making multi-language databases, resources and findings available in open access, as well as transferable skills. The ESRs conduct experimental and corpus studies on the alignment and prediction processes that make conversation between people both easy and fluent, across a large variety of communicational settings and in different languages, to better understand how these processes contribute to setting up brain-to-brain coupling relationships. Collaborative work with non-academic partners fosters the development of more effective and socially acceptable text-to-speech synthesizers, artificial dialogue systems, and social humanoid robots with high-level conversational skills. The project opens new career perspectives for ESRs with interdisciplinary training in language sciences, neuroscience and dialog systems on a very fast-growing digital market. COBRA’s training programme also has major societal implications as it concerns aspects of the European citizens’ everyday life, from spoken interactions with machines to conversing in a non-native language.
The work programme is organized in 9 Work Packages. WP1 aims at training ESRs in theory, methods and analyses of the cognitive and neural mechanisms underpinning conversational interactions, with special emphasis on the alignment and prediction mechanisms employed by speakers in the building up of shared meaning. WP2’s main goal is to offer an accurate characterization of the linguistic cues to alignment and prediction in conversational interactions, with a focus on different communicative situations, including interactions with conversational agents. WP3 deals with the technical and methodological skills and resources that enable ESRs to conduct cutting-edge research on conversational interactions in a two-person perspective. In WP4, text-to-speech synthesis, human-machine dialogue systems, and artificial conversational agents form a fertile ground of application for ESRs’ research. The focus is on how computational implementations of alignment and prediction mechanisms in conversation can allow dialogue systems to better adapt to the user’s individual characteristics as a speaker. WP5 is COBRA’s training programme, which includes network-wide meetings, events and workshops. WP6 contains COBRA’s dissemination, communication and exploitation roadmap. WP7 deals with COBRA’s management strategy and procedures. WP8 and WP9 relate to ethics requirements and open research data respectively.
Work in WP1 has focused on the processes underlying prediction and alignment using minimal dyadic contexts in the laboratory, and neuroscientific measures of prediction and alignment in conversational settings. In WP2, the emphasis has been on linguistic features, from fine-grained phonetic and prosodic features to discourse-level expressions and constructions. WP3 has concentrated on data collection and processing, data analysis, and data long-term storage and sharing. A shared repertoire of set-ups and best practices for experimental studies on conversational interactions has been established. In WP4, full use is made of the leading-edge computational frameworks and powerful resources developed by academic and non-academic consortium members. In WP5, two online training events were organized. Training also comprised courses on open science, statistics, and mental health. A moodle e-learning platform has been set up. As regards WP6, the consortium has equipped itself with a digital infrastructure for both internal and external communication which comprises a website, a Zenodo Community and a Twitter account. Dissemination was limited due to the COVID crisis, but included a participation in the 2021 European Researchers' Night, among other events. An Open Data Management plan was set up and is hosted on dmp.opidor.fr.
Work in WP1 has focused on the processes underlying prediction and alignment using minimal dyadic contexts in the laboratory, and neuroscientific measures of prediction and alignment in conversational settings. In WP2, the emphasis has been on linguistic features, from fine-grained phonetic and prosodic features to discourse-level expressions and constructions. WP3 has concentrated on data collection and processing, data analysis, and data long-term storage and sharing. A shared repertoire of set-ups and best practices for experimental studies on conversational interactions has been established. In WP4, full use is made of the leading-edge computational frameworks and powerful resources developed by academic and non-academic consortium members. In WP5, two online training events were organized. Training also comprised courses on open science, statistics, and mental health. A moodle e-learning platform has been set up. As regards WP6, the consortium has equipped itself with a digital infrastructure for both internal and external communication which comprises a website, a Zenodo Community and a Twitter account. Dissemination was limited due to the COVID crisis, but included a participation in the 2021 European Researchers' Night, among other events. An Open Data Management plan was set up and is hosted on dmp.opidor.fr.
Research and training activities focus on two major mechanisms employed by speakers in a conversational interaction, interactive alignment, and prediction. ESRs are trained to explore these mechanisms across a large variety of languages and communicational settings, and address two main challenges. The first challenge is to determine how alignment and prediction may both rely on and contribute to setting up brain-to-brain coupling relationships. The second challenge relates to the development of computational models of alignment and prediction for more effective and socially-acceptable text-to-speech synthesizers, human-machine dialogue systems, and social robots. This will open the way towards using neurobehavioral measures for the on-line monitoring of artificial agents, and the offline evaluation of quality of communication in human-machine interactions.
Within the consortium, nine European languages as well as Mandarin Chinese are represented that allow the ESRs to confront a wide set of linguistic phenomena spread across the European territory and Asia. Transfer-of-technology activities include innovative experimental set-ups for the joint monitoring of brain and physiological activities in two or more people engaged in a spoken-language interactions, as well as the development of spoken-dialogue systems and social robots with high-level conversational skills.
Research in this multi-disciplinary domain will help interactive technologies improve in two ways. First, an understanding of which linguistic phenomena appear in which situations provides the technologies with improvements in naturalness, the extent to which the virtual assistant or conversational robot acts like a human interlocutor. Second, an appreciation of the cognitive impact of discourse alignment and prediction phenomena makes it possible to go beyond naturalness to select those linguistic phenomena which are cognitively useful in achieving the goals of the particular human-machine interaction.
The COBRA training programme is dedicated to both providing ESRs the scientific and technical knowledge and skills required to push research on conversational brains beyond the state of the art and giving ESRs wide exposure to highly-competitive companies in web-based speech technology, conversational agents and robotics. The programme also has major societal implications as it addresses issues that relate to communication in people’s native and non-native languages. The focus on the dynamics of conversational exchanges between people and on the mechanisms employed for the establishment of a conversational common ground makes it possible to significantly further our understanding of what makes spoken language communication efficient and successful in a large variety of languages and interaction situations.
Within the consortium, nine European languages as well as Mandarin Chinese are represented that allow the ESRs to confront a wide set of linguistic phenomena spread across the European territory and Asia. Transfer-of-technology activities include innovative experimental set-ups for the joint monitoring of brain and physiological activities in two or more people engaged in a spoken-language interactions, as well as the development of spoken-dialogue systems and social robots with high-level conversational skills.
Research in this multi-disciplinary domain will help interactive technologies improve in two ways. First, an understanding of which linguistic phenomena appear in which situations provides the technologies with improvements in naturalness, the extent to which the virtual assistant or conversational robot acts like a human interlocutor. Second, an appreciation of the cognitive impact of discourse alignment and prediction phenomena makes it possible to go beyond naturalness to select those linguistic phenomena which are cognitively useful in achieving the goals of the particular human-machine interaction.
The COBRA training programme is dedicated to both providing ESRs the scientific and technical knowledge and skills required to push research on conversational brains beyond the state of the art and giving ESRs wide exposure to highly-competitive companies in web-based speech technology, conversational agents and robotics. The programme also has major societal implications as it addresses issues that relate to communication in people’s native and non-native languages. The focus on the dynamics of conversational exchanges between people and on the mechanisms employed for the establishment of a conversational common ground makes it possible to significantly further our understanding of what makes spoken language communication efficient and successful in a large variety of languages and interaction situations.