Periodic Reporting for period 2 - COBRA (COnversational BRAins)
Berichtszeitraum: 2022-02-01 bis 2024-07-31
COBRA aimed to train the next generation of researchers to accurately characterize and model the linguistic, cognitive, and brain mechanisms deployed by human speakers during conversational interactions with both human interlocutors and artificial dialogue systems. It relied on a cross-sectoral international network of 11 world-class academic research centers and 4 non-academic partners, including 3 fast-growing SMEs and 1 global company. The unique combination of expertise and strong complementarity among the partners enabled COBRA to offer 15 Early Stage Researchers (ESRs) an excellent training program, along with significant exposure to the non-academic sector in the emerging field of conversational neuroscience.The training covered scientific and technical skills, ranging from the joint monitoring of brain and physiological activities in two or more people conversing, to creating multilingual databases, resources, and tools made available in open access. It also included the development of transferable skills. The ESRs conducted experimental and corpus studies on the alignment and prediction processes that make conversations between people both easy and fluent, across a wide variety of communication settings and languages. This work contributed to a deeper understanding of how these processes help establish brain-to-brain coupling relationships.
Collaborative efforts with non-academic partners fostered the development of more effective and socially acceptable text-to-speech synthesizers, artificial dialogue systems, and social humanoid robots with high-level conversational abilities. The project opened new career opportunities for the ESRs, providing interdisciplinary training in language sciences, neuroscience, and dialogue systems within a rapidly growing digital market. COBRA's training program also had significant societal implications, addressing aspects of European citizens' everyday lives, from spoken interactions with machines to conversations in non-native languages.
Collaborative efforts with non-academic partners fostered the development of more effective and socially acceptable text-to-speech synthesizers, artificial dialogue systems, and social humanoid robots with high-level conversational abilities. The project opened new career opportunities for the ESRs, providing interdisciplinary training in language sciences, neuroscience, and dialogue systems within a rapidly growing digital market. COBRA's training program also had significant societal implications, addressing aspects of European citizens' everyday lives, from spoken interactions with machines to conversations in non-native languages.
The COBRA network aimed to train researchers to explore the linguistic, cognitive, and brain mechanisms underlying conversation in both human-human and human-machine interactions. The project focused on mechanisms like interactive alignment and prediction, which facilitate fluent conversations. This interdisciplinary project involved 11 academic research centers and 4 industry partners, offering PhD students training in neuroscience, linguistics, and computational modeling. The core of the project was centered around 9 Work Packages (WP):
WP1: Alignment and Prediction in Conversation
This work package focused on understanding how speakers align their mental states and predict their partner’s responses during conversation. Experiments showed asymmetrical adjustments in speech perception and production and revealed how neural alignment corresponds to conversational alignment. Key findings include:
- Neural alignment is enhanced by prediction mechanisms during interaction.
- Phonetic and prosodic entrainment occurs even in human-robot interactions.
WP2: Linguistic Features of Alignment and Prediction
This WP explored how specific linguistic features (e.g. prosody, discourse markers) impact alignment and prediction. Key insights include:
- Discourse markers significantly influence conversation structure and predict shifts in topic.
- Alignment at the discourse level affects conversational fluency, especially in task-oriented dialogues.
WP3: Methods, Tools, and Databases
Researchers developed tools for analyzing conversational interactions in dual-speaker settings. Several multilingual datasets were created, and tools like the DeepCobra model improved the detection of speech convergence and alignment. Notable results include:
- Data collection tools for behavioral and EEG studies in conversational interactions.
- Innovative approaches for speech synthesis and turn-taking cues.
WP4: Improving Human-Machine Interaction
This WP applied alignment and prediction mechanisms to artificial agents, aiming to improve conversational AI systems. Findings include:
- Human interlocutors align similarly with both anthropomorphic and machine-like robots.
- Backchannels (e.g. “uh-huh”) are critical for natural conversation in AI systems.
WP5: Training Program
The training events provided ESRs with interdisciplinary knowledge and hands-on experience in cutting-edge research methodologies. The events, including training events, workshops and hackathons, fostered collaborative learning between academia and industry.
WP6: Dissemination and Exploitation
COBRA effectively disseminated its findings through scientific publications, workshops, participation in international conferences, and divulgate dissemination through blogs and science events for the general public. Furthermore, a digital infrastructure for both internal and external communication which comprises the Cobra website was set up, in addition to a Zenodo Community and a Twitter account.
Notable activities include:
- Many key publications in international, peer-reviewed journals (and many more work currently in preparation for publication)
- An international and interdisciplinary workshop on conversational (mis)alignment held in Aix-en-Provence including world leaders on the topics of alignment and conversation.
- Participation in multiple international conferences where PhD students presented their research.
Finally, WP7 dealt with COBRA's management strategy and procedures, and WP8 and WP9 related to ethics requirements and open research data (for which an open data plan was set up and is hosted at: amp.opidor.tr) respectively.
WP1: Alignment and Prediction in Conversation
This work package focused on understanding how speakers align their mental states and predict their partner’s responses during conversation. Experiments showed asymmetrical adjustments in speech perception and production and revealed how neural alignment corresponds to conversational alignment. Key findings include:
- Neural alignment is enhanced by prediction mechanisms during interaction.
- Phonetic and prosodic entrainment occurs even in human-robot interactions.
WP2: Linguistic Features of Alignment and Prediction
This WP explored how specific linguistic features (e.g. prosody, discourse markers) impact alignment and prediction. Key insights include:
- Discourse markers significantly influence conversation structure and predict shifts in topic.
- Alignment at the discourse level affects conversational fluency, especially in task-oriented dialogues.
WP3: Methods, Tools, and Databases
Researchers developed tools for analyzing conversational interactions in dual-speaker settings. Several multilingual datasets were created, and tools like the DeepCobra model improved the detection of speech convergence and alignment. Notable results include:
- Data collection tools for behavioral and EEG studies in conversational interactions.
- Innovative approaches for speech synthesis and turn-taking cues.
WP4: Improving Human-Machine Interaction
This WP applied alignment and prediction mechanisms to artificial agents, aiming to improve conversational AI systems. Findings include:
- Human interlocutors align similarly with both anthropomorphic and machine-like robots.
- Backchannels (e.g. “uh-huh”) are critical for natural conversation in AI systems.
WP5: Training Program
The training events provided ESRs with interdisciplinary knowledge and hands-on experience in cutting-edge research methodologies. The events, including training events, workshops and hackathons, fostered collaborative learning between academia and industry.
WP6: Dissemination and Exploitation
COBRA effectively disseminated its findings through scientific publications, workshops, participation in international conferences, and divulgate dissemination through blogs and science events for the general public. Furthermore, a digital infrastructure for both internal and external communication which comprises the Cobra website was set up, in addition to a Zenodo Community and a Twitter account.
Notable activities include:
- Many key publications in international, peer-reviewed journals (and many more work currently in preparation for publication)
- An international and interdisciplinary workshop on conversational (mis)alignment held in Aix-en-Provence including world leaders on the topics of alignment and conversation.
- Participation in multiple international conferences where PhD students presented their research.
Finally, WP7 dealt with COBRA's management strategy and procedures, and WP8 and WP9 related to ethics requirements and open research data (for which an open data plan was set up and is hosted at: amp.opidor.tr) respectively.
The COBRA project has advanced the state-of-the-art in understanding conversational dynamics by integrating cutting-edge interdisciplinary research from linguistics, cognitive science, and neuroscience. Traditional approaches to studying language have focused on isolated speakers, but COBRA has pioneered research on interactive alignment and prediction mechanisms in real-time conversations between multiple speakers. This shift from a single-brain to a multi-brain framework allows the project to examine how the brains of two individuals synchronize during interactions. Key innovations include:
1. Neural Alignment and Prediction: COBRA introduced novel methodologies to assess brain-to-brain coupling in conversational settings, revealing that prediction enhances neural and linguistic alignment.
2. Human-Machine Interactions: By developing computational models that incorporate alignment and prediction, the project has laid the groundwork for next-generation text-to-speech systems and socially adaptive conversational agents.
3. Multi-Language Research: COBRA explored nine European languages and Mandarin, allowing the ESRs (early-stage researchers) to confront a variety of linguistic phenomena, enriching research on cross-linguistic conversational dynamics.
1. Neural Alignment and Prediction: COBRA introduced novel methodologies to assess brain-to-brain coupling in conversational settings, revealing that prediction enhances neural and linguistic alignment.
2. Human-Machine Interactions: By developing computational models that incorporate alignment and prediction, the project has laid the groundwork for next-generation text-to-speech systems and socially adaptive conversational agents.
3. Multi-Language Research: COBRA explored nine European languages and Mandarin, allowing the ESRs (early-stage researchers) to confront a variety of linguistic phenomena, enriching research on cross-linguistic conversational dynamics.