Periodic Reporting for period 1 - ELOQUENCE (Multilingual and Cross-cultural interactions for context-aware, and bias-controlled dialogue systems for safety-critical applications)
Okres sprawozdawczy: 2024-01-01 do 2025-06-30
ELOQUENCE aims to advance conversational AI by creating explainable, trustworthy, and bias-controlled language technologies for multilingual and multimodal environments. These systems will support diverse scenarios, from unstructured dialogues to safety-critical applications like emergency services. The project emphasises sustainability, adaptability from limited data, and alignment with European values, without training new models but improving existing open-source LLMs. Overall, ELOQUENCE aims to boost European technological sovereignty by delivering value-aligned AI systems for all EU citizens by
concentrating on the following primary objectives:
- Advanced Spoken Language Understanding: Support all EU languages using multimodality and efficient adaptation.
- Hybrid LLMs: Combine contextual knowledge with deep learning for explainable, compositional reasoning in dialogues.
- Trustworthiness & Bias Control: Mitigate gender, cultural, and racial biases, ensuring compliance with EU ethical and legal standards.
- Assessment Framework: Evaluate models on performance, explainability, trust, multilinguality, and bias, including human-in-the-loop validation.
- Real-World Validation: Test through four pilots—smart homes, social contexts, customer service, and emergency response.
- Sustainability & Open Science: Reduce data and energy use by building upon developed open-source LLMs and release data, models, and publications in open access.
concentrating on the following primary objectives:
- Advanced Spoken Language Understanding: Support all EU languages using multimodality and efficient adaptation.
- Hybrid LLMs: Combine contextual knowledge with deep learning for explainable, compositional reasoning in dialogues.
- Trustworthiness & Bias Control: Mitigate gender, cultural, and racial biases, ensuring compliance with EU ethical and legal standards.
- Assessment Framework: Evaluate models on performance, explainability, trust, multilinguality, and bias, including human-in-the-loop validation.
- Real-World Validation: Test through four pilots—smart homes, social contexts, customer service, and emergency response.
- Sustainability & Open Science: Reduce data and energy use by building upon developed open-source LLMs and release data, models, and publications in open access.
During the first reporting period, all WPs were active, with WP3 and WP4 starting at Month 12 (M12). Key highlights include:
WP1 focused on creating a fused dataset for unstructured and structured multi-turn dialogues by analyzing and preprocessing open-source datasets. It also initiated the definition of metrics to assess understanding in semi-structured and unstructured dialogues, as well as explainability and trustworthiness in safety-critical contexts.
WP2 worked on extending current LLMs with conversational capabilities. It developed and released code to connect speech encoders with LLMs via a lightweight projector for monolingual cases. The WP advanced factual information retrieval (FIR), explored joint speech-text training for FIR-grounded responses, and investigated federated LLMs. It also created SDialog, a Python library for synthetic dialogue generation, and a synthetic persona generator. Additionally, WP2 developed the first version of the dialogue manager for high-risk use cases.
WP3, starting at M12, focused on building reliable LLM-based dialogue systems by improving interpretability through context and domain knowledge. Early results include text summarization in a federated learning setting for multilinguality, semantic analysis of text embeddings, and interpretable fine-grained information retrieval. For robust dialogues in high-risk scenarios, WP3 developed CopyLM for text reuse and Dialog2Flow for extracting workflows in task-oriented dialogues.
WP4 also began at M12, targeting multilinguality and low-resource languages. It developed ASR methods using end-to-end and weakly supervised approaches, leveraging pretrained LLMs. The WP initiated spoken language understanding in multi-task and continual learning settings, including multi-talker scenarios with diarization. It participated in the MLC-CLS challenge and achieved initial results in bias mitigation, particularly robustness against recording devices. Additional work included context-aware speech recognition using contrastive learning and user emotion representation.
WP5 defined requirements, KPIs, and evaluation criteria for each pilot, establishing a methodology to assess feasibility, usability, and impact. It developed and deployed the Interactive Playground (IP) as the main experimentation and demonstration environment and initiated pilot integration.
WP6 created an iterative methodology for human rights assessment and alignment with EU values in Generative AI. This multidisciplinary approach ensures LLM behavior in pilot contexts meets societal expectations, producing transferable guidelines for ELOQUENCE technologies.
WP7 promoted project innovations, ensured active social media presence, and published over 20 peer-reviewed papers. Most KPIs are on track, demonstrating strong impact and partner commitment to dissemination and exploitation.
WP1 focused on creating a fused dataset for unstructured and structured multi-turn dialogues by analyzing and preprocessing open-source datasets. It also initiated the definition of metrics to assess understanding in semi-structured and unstructured dialogues, as well as explainability and trustworthiness in safety-critical contexts.
WP2 worked on extending current LLMs with conversational capabilities. It developed and released code to connect speech encoders with LLMs via a lightweight projector for monolingual cases. The WP advanced factual information retrieval (FIR), explored joint speech-text training for FIR-grounded responses, and investigated federated LLMs. It also created SDialog, a Python library for synthetic dialogue generation, and a synthetic persona generator. Additionally, WP2 developed the first version of the dialogue manager for high-risk use cases.
WP3, starting at M12, focused on building reliable LLM-based dialogue systems by improving interpretability through context and domain knowledge. Early results include text summarization in a federated learning setting for multilinguality, semantic analysis of text embeddings, and interpretable fine-grained information retrieval. For robust dialogues in high-risk scenarios, WP3 developed CopyLM for text reuse and Dialog2Flow for extracting workflows in task-oriented dialogues.
WP4 also began at M12, targeting multilinguality and low-resource languages. It developed ASR methods using end-to-end and weakly supervised approaches, leveraging pretrained LLMs. The WP initiated spoken language understanding in multi-task and continual learning settings, including multi-talker scenarios with diarization. It participated in the MLC-CLS challenge and achieved initial results in bias mitigation, particularly robustness against recording devices. Additional work included context-aware speech recognition using contrastive learning and user emotion representation.
WP5 defined requirements, KPIs, and evaluation criteria for each pilot, establishing a methodology to assess feasibility, usability, and impact. It developed and deployed the Interactive Playground (IP) as the main experimentation and demonstration environment and initiated pilot integration.
WP6 created an iterative methodology for human rights assessment and alignment with EU values in Generative AI. This multidisciplinary approach ensures LLM behavior in pilot contexts meets societal expectations, producing transferable guidelines for ELOQUENCE technologies.
WP7 promoted project innovations, ensured active social media presence, and published over 20 peer-reviewed papers. Most KPIs are on track, demonstrating strong impact and partner commitment to dissemination and exploitation.
WP1 released fused datasets and novel metrics to assess performance, trustworthiness, and explainability in structured and unstructured dialogues, addressing a key scientific need.
WP2 enhanced conversational capabilities by integrating speech input, creating tools for synthetic dialogue and persona generation, and improving trustworthiness through factual information retrieval (FIR) and privacy-preserving federated training.
WP3 explored methods to improve interpretability and developed initial tools for efficient, robust dialogue design.
WP4 advanced multilingual systems, focusing on low-resource languages, and began releasing related models and recipes. It also improved contextual integration in speech-LLM systems.
WP6 established a methodology for ethical assessment of AI systems, addressing critical needs under the EU AI Act.
WP2 enhanced conversational capabilities by integrating speech input, creating tools for synthetic dialogue and persona generation, and improving trustworthiness through factual information retrieval (FIR) and privacy-preserving federated training.
WP3 explored methods to improve interpretability and developed initial tools for efficient, robust dialogue design.
WP4 advanced multilingual systems, focusing on low-resource languages, and began releasing related models and recipes. It also improved contextual integration in speech-LLM systems.
WP6 established a methodology for ethical assessment of AI systems, addressing critical needs under the EU AI Act.