Multilingual and Cross-cultural interactions for context-aware, and bias-controlled dialogue systems for safety-critical applications

Projektinformationen

ELOQUENCE

ID Finanzhilfevereinbarung: 101135916

DOI

10.3030/101135916

EK-Unterschriftsdatum 29 November 2023

Startdatum 1 Januar 2024

Enddatum 31 Dezember 2026

Finanziert unter

• Digital, Industry and Space

Gesamtkosten

€ 5 072 543,75

EU-Beitrag

€ 5 072 543,75

Koordiniert durch

TELEFONICA INNOVACION DIGITAL SL
Spain

CORDIS bietet Links zu öffentlichen Ergebnissen und Veröffentlichungen von HORIZONT-Projekten.

Links zu Ergebnissen und Veröffentlichungen von RP7-Projekten sowie Links zu einigen Typen spezifischer Ergebnisse wie Datensätzen und Software werden dynamisch von OpenAIRE abgerufen.

Leistungen

Documents, reports (12)

Emerging ELOQUENCE technology- approved by the ELOQUENCE Community

Finalized at TLR=3 stage, this report assesses emerging ELOQUENCE outputs as being respectful of EU values, with particular emphasis on gender, cultural or racial biases.

Dissemination, Communication and Exploitation Plan

Overall DEC plan, with KPIs and benchmarks, and campaign planning. Also includes the website.

Ethics Compliance Management Report

The Ethics requirements and compliance methodology for responsible research.

Algorithms definition, baselines, open issues and use cases

Methodologies applied for both knowledge-based approaches and semi-supervised learning, and applicability to use cases.

Retrieval model for conversational style queries

The methodologies of integrating conversational LLM with FIR subsystem, discussion of open research questions in the context of ELOQUENCE. Codebase implementing the FIR, evaluation of the FIR precision given conversational queries.

Dissemination, Communication and Exploitation Plan II

Overall DEC plan, with KPIs and benchmarks, and campaign planning. Also includes the website.

Conversational LLM

Methodologies for LLM finetuning and simulation/ augmentation of conversational training data, and applicability to use cases. Comparison with SOTA models. Codebase implementing the fine-tuning on selected datasets.

Report on linguistic expression respectful of EU values

Report on requirements for machine-generated verbal communication respectful of European values as enshrined in Article 2 of the EU Treaty.

Pilot Requirements & Usability Evaluation

Uses cases, requirements and outcomes for the different pilots. Summary of the set of KPIs, criteria and methodology to evaluate the ELOQUENCE usability.

Open source datasets suitable for semi-structured, unstructured and multi-turn conversations

Requirements for semi-structured, multi-turn and unstructured dialogues, available/relevant literature review on evaluation of dialogues and description of methodology on how the fused datasets are generated

Project Management Handbook

Management procedure, consortium communication tools and the quality and risk management plans.

Methodology for jointly training FIR and response generator

The methodologies on interconnecting FIR and response generator, analysis of scenarios relevant to ELOQUENCE. Implementation of the FIR and response generator joint training.

Data Management Plan (2)

ELOQUENCE DMP

The legal aspects and the data management plan for the collection and processing of data throughout the project activities.

ELOQUENCE DMP II

The legal aspects and the data management plan for the collection and processing of data throughout the project activities.

Demonstrators, pilots, prototypes (1)

ELOQUENCE Interactive Playground

Initial version of the Interactive Playground for the LM validation.

Veröffentlichungen

Conference proceedings (13)

Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems

Autoren: Christos Vlachos, Themos Stafylakis, Ion Androutsopoulos
Veröffentlicht in: Findings of the Association for Computational Linguistics ACL 2024, 2024
Herausgeber: Association for Computational Linguistics
DOI: 10.18653/V1/2024.FINDINGS-ACL.431

BESST Dataset: A Multimodal Resource for Speech-based Stress Detection and Analysis

Autoren: Jan Pešán, Vojtěch Juřík, Martin Karafiát, Jan Černocký
Veröffentlicht in: Interspeech 2024, 2024
Herausgeber: ISCA
DOI: 10.21437/INTERSPEECH.2024-42

BUT systems and analyses for the ASVspoof 5 Challenge

Autoren: Johan Rohdin, Lin Zhang, Plchot Oldřich, Vojtěch Staněk, David Mihola, Junyi Peng, Themos Stafylakis, Dmitriy Beveraki, Anna Silnova, Jan Brukner, Lukáš Burget
Veröffentlicht in: The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024), 2024
Herausgeber: ISCA
DOI: 10.21437/ASVSPOOF.2024-4

TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR

Autoren: Shashi Kumar, Srikanth Madikeri, Juan Pablo Zuluaga Gomez, Iuliia Thorbecke, Esaú Villatoro-tello, Sergio Burdisso, Petr Motlicek, Karthik Pandia D S, Aravind Ganapathiraju
Veröffentlicht in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024
Herausgeber: Association for Computational Linguistics
DOI: 10.18653/V1/2024.EMNLP-MAIN.1167

Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units

Autoren: Bolaji Yusuf, Jan Honza Cernocky, Murat Saraçlar
Veröffentlicht in: Interspeech 2024, 2024
Herausgeber: ISCA
DOI: 10.21437/INTERSPEECH.2024-1713

Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers

Autoren: Umberto Cappellazzo, Daniele Falavigna, Alessio Brutti, Mirco Ravanelli
Veröffentlicht in: 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), 2024
Herausgeber: IEEE
DOI: 10.1109/MLSP58920.2024.10734776

Multitask Speech Recognition and Speaker Change Detection for Unknown Number of Speakers

Autoren: Shashi Kumar, Srikanth Madikeri, Iuliia Nigmatulina, Esaú Villatoro-Tello, Petr Motlicek, Karthik Pandia, S. Pavankumar Dubagunta, Aravind Ganapathiraju
Veröffentlicht in: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024
Herausgeber: IEEE
DOI: 10.1109/ICASSP48485.2024.10446130

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Autoren: Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, Maja Pantic
Veröffentlicht in: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025
Herausgeber: IEEE
DOI: 10.1109/ICASSP49660.2025.10889251

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

Autoren: Sergio Burdisso, Srikanth Madikeri, Petr Motlicek
Veröffentlicht in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024
Herausgeber: Association for Computational Linguistics
DOI: 10.18653/V1/2024.EMNLP-MAIN.310

Multimodal Emotion Recognition Using Compressed Graph Neural Networks

Autoren: Tijana Đurkić, Nikola Simić, Siniša Suzić, Dragana Bajović, Zoran Perić, Vlado Delić
Veröffentlicht in: Lecture Notes in Computer Science, Speech and Computer, 2025
Herausgeber: Springer Nature Switzerland
DOI: 10.1007/978-3-031-78014-1_9

BUT/JHU System Description for CHiME-8 NOTSOFAR-1 Challenge

Autoren: Alexander Polok, Dominik Klement, Jiangyu Han, Šimon Sedláček, Bolaji Yusuf, Matthew Maciejewski, Matthew S Wiesner, Lukáš Burget
Veröffentlicht in: 8th International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024
Herausgeber: ISCA
DOI: 10.21437/CHIME.2024-4

Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters

Autoren: Umberto Cappellazzo, Daniele Falavigna, Alessio Brutti
Veröffentlicht in: Interspeech 2024, 2024
Herausgeber: ISCA
DOI: 10.21437/INTERSPEECH.2024-38

MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation

Autoren: Javier García Gilabert, Carlos Escolano, Audrey Mash, Xixian Liao, Maite Melero
Veröffentlicht in: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), 2025
Herausgeber: Association for Computational Linguistics
DOI: 10.18653/V1/2025.NAACL-DEMO.6