During the first reporting period, all WPs were active, with WP3 and WP4 starting at Month 12 (M12). Key highlights include:
WP1 focused on creating a fused dataset for unstructured and structured multi-turn dialogues by analyzing and preprocessing open-source datasets. It also initiated the definition of metrics to assess understanding in semi-structured and unstructured dialogues, as well as explainability and trustworthiness in safety-critical contexts.
WP2 worked on extending current LLMs with conversational capabilities. It developed and released code to connect speech encoders with LLMs via a lightweight projector for monolingual cases. The WP advanced factual information retrieval (FIR), explored joint speech-text training for FIR-grounded responses, and investigated federated LLMs. It also created SDialog, a Python library for synthetic dialogue generation, and a synthetic persona generator. Additionally, WP2 developed the first version of the dialogue manager for high-risk use cases.
WP3, starting at M12, focused on building reliable LLM-based dialogue systems by improving interpretability through context and domain knowledge. Early results include text summarization in a federated learning setting for multilinguality, semantic analysis of text embeddings, and interpretable fine-grained information retrieval. For robust dialogues in high-risk scenarios, WP3 developed CopyLM for text reuse and Dialog2Flow for extracting workflows in task-oriented dialogues.
WP4 also began at M12, targeting multilinguality and low-resource languages. It developed ASR methods using end-to-end and weakly supervised approaches, leveraging pretrained LLMs. The WP initiated spoken language understanding in multi-task and continual learning settings, including multi-talker scenarios with diarization. It participated in the MLC-CLS challenge and achieved initial results in bias mitigation, particularly robustness against recording devices. Additional work included context-aware speech recognition using contrastive learning and user emotion representation.
WP5 defined requirements, KPIs, and evaluation criteria for each pilot, establishing a methodology to assess feasibility, usability, and impact. It developed and deployed the Interactive Playground (IP) as the main experimentation and demonstration environment and initiated pilot integration.
WP6 created an iterative methodology for human rights assessment and alignment with EU values in Generative AI. This multidisciplinary approach ensures LLM behavior in pilot contexts meets societal expectations, producing transferable guidelines for ELOQUENCE technologies.
WP7 promoted project innovations, ensured active social media presence, and published over 20 peer-reviewed papers. Most KPIs are on track, demonstrating strong impact and partner commitment to dissemination and exploitation.
