My Personal AI Mediator for Virtual MEETtings BetWEEN People

Informacje na temat projektu

Meetween

Identyfikator umowy o grant: 101135798

DOI

10.3030/101135798

Data podpisania przez KE 3 Listopada 2023

Data rozpoczęcia 1 Stycznia 2024

Data zakończenia 31 Grudnia 2027

Finansowanie w ramach

• Digital, Industry and Space

Koszt całkowity

€ 7 239 725,00

Wkład UE

€ 7 239 725,00

Koordynowany przez

TRANSLATED SRL
Italy

CORDIS oferuje możliwość skorzystania z odnośników do publicznie dostępnych publikacji i rezultatów projektów realizowanych w ramach programów ramowych HORYZONT.

Odnośniki do rezultatów i publikacji związanych z poszczególnymi projektami 7PR, a także odnośniki do niektórych konkretnych kategorii wyników, takich jak zbiory danych i oprogramowanie, są dynamicznie pobierane z systemu OpenAIRE .

Rezultaty

Other (5)

Downstream task trained models - V1

This deliverable reports the efforts to adapt the best models to satisfy production constraints, such as low latency, low flickering, and low computational costs, and develop APIs to serve different applications and needs.

Speech LMM open release - V1

SpeechLMM is released three times during the project and each release is documented in a version of this deliverable.

Model cards - V1

All partners will author a deliverable containing model cards.

Speech LMM training journal - V1

This deliverable documents the “making of” SpeechLMM and is used for dissemination to the scientific and technology community in the form of the SpeechLMM training journal.

Human factors and security scenarios - V1

This deliverable provides human factors and a security report describing vulnerability and criticality scenarios.

Data sets, microdata, etc (1)

Mumospee open release - V1

This deliverable documents the creation of one of the largest open multimodal datasets, Mumospee, released over three generations, in coordination with the corresponding SpeechLMM generation activities.

Publikacje

Conference proceedings (45)

Factorized-VITS: Decoupling Prosody and Text in End-to-End Speech Synthesis without External or Secondary Aligner

Autorzy: Yining Liu, Alexander Waibel
Opublikowane w: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025
Wydawca: IEEE
DOI: 10.1109/ICASSP49660.2025.10890003

SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

Autorzy: Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.IWSLT-1.11

Weight Factorization and Centralization for Continual Learning in Speech Recognition

Autorzy: Enes Yavuz Ugan*, Ngoc-Quan Pham*, Alexander Waibel
Wydawca: INTERSPEECH 2025
DOI: 10.48550/ARXIV.2506.16574

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

Autorzy: Marco Gaido*, Sara Papi*, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri.
Wydawca: IWSLT2025
DOI: 10.48550/ARXIV.2505.23420

NUTSHELL: A Dataset for Abstract Generation from Scientific Talks

Autorzy: Maike Züfle, Sara Papi, Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Jan Niehues
Opublikowane w: Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), 2025
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2025.IWSLT-1.2

MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models

Autorzy: Thai-Binh Nguyen, Alexander Waibel
Wydawca: ICASSP 2025
DOI: 10.48550/ARXIV.2411.18152

A Bayesian Optimization Approach to Machine Translation Reranking

Autorzy: Julius Cheng, Maike Züfle, Vilém Zouhar, Andreas Vlachos
Wydawca: NAACL 2025 Main
DOI: 10.48550/ARXIV.2411.09694

FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE

Autorzy: Beatrice Savoldi, Marco Gaido, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.IWSLT-1.10

MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages

Autorzy: Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri
Opublikowane w: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.EMNLP-MAIN.771

StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection

Autorzy: Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.ACL-LONG.202

Quality Estimation with $k$-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation

Autorzy: Dinh, Tu Anh; Palzer, Tobias; Niehues, Jan
Opublikowane w: 2024
Wydawca: EAMT 2024
DOI: 10.5445/IR/1000174743

GFG - Gender-Fair Generation: A CALAMITA Challenge

Autorzy: Simona Frenda, Andrea Piergentili, Beatrice Savoldi, Marco Madeddu, Martina Rosola, Silvia Casola, Chiara Ferrando, Viviana Patti, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), 2024
Wydawca: CEUR Workshop Proceedings

FINDINGS OF THE IWSLT 2024 EVALUATION CAMPAIGN

Autorzy: Ibrahim Said Ahmad, Antonios Anastasopoulos, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, William Chen, Qianqian Dong, Marcello Federico, Barry Haddow, Dávid Javorský, Mateusz Krubiński, Tsz Kim Lam, Xutai Ma, Prashant Mathur, Evgeny Matusov, Chandresh Maurya, John McCrae, Kenton Murray, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, Atul Kr. Ojha, John Ortega, Sara Papi, Peter Polák, Adam Pospíšil, Pavel Pecina, Elizabeth Salesky, Nivedita Sethiya, Balaram Sarkar, Jiatong Shi, Claytone Sikasote, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Brian Thompson, Alex Waibel, Shinji Watanabe, Patrick Wilken, Petr Zemánek, Rodolfo Zevallos
Opublikowane w: Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.IWSLT-1.1

Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?

Autorzy: Marco Gaido, Sara Papi, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.ACL-LONG.789

Contrastive Learning for Task-Independent SpeechLLM-Pretraining

Autorzy: Maike Züfle, Jan Niehues
Wydawca: ACL 2025 Findings
DOI: 10.48550/ARXIV.2412.15712

Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactions

Autorzy: Beatrice Savoldi, Alan Ramponi, Matteo Negri, Luisa Bentivogli.
Wydawca: arXiv
DOI: 10.48550/ARXIV.2502.13780

Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages

Autorzy: Mullov, Carlos; Pham, Quan; Waibel, Alexander
Opublikowane w: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024
Wydawca: ACL
DOI: 10.5445/IR/1000174872

KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025

Autorzy: Sai Koneru*, Maike Züfle*, Thai-Binh Nguyen, Seymanur Akti, Jan Niehues, Alexander Waibel
Wydawca: IWSLT 2025
DOI: 10.48550/ARXIV.2505.13036

Findings of the IWSLT 2025 Evaluation Campaign

Autorzy: Idris Abdulmumin, Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Fethi Bougares, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, William Chen, Raj Dabre, Yannick Estève, Marcello Federico, Mark Fishe
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2025.IWSLT-1.44

Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach

Autorzy: Li, Siqi; Liu, Danni; Niehues, Jan
Opublikowane w: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024
Wydawca: EMNLP
DOI: 10.48550/ARXIV.2409.09009

Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversion Voice Conversion

Autorzy: Seymanur Akti, Tuan Nam Nguyen, Alexander Waibel
Wydawca: INTERSPEECH 2025
DOI: 10.48550/ARXIV.2506.04013

Cocktail-Party Audio-Visual Speech Recognition

Autorzy: Thai-Binh Nguyen, Ngoc-Quan Pham, Alexander Waibel
Wydawca: INTERSPEECH 2025
DOI: 10.48550/ARXIV.2506.02178

MAGNET - MAchines GeNErating Translations: A CALAMITA Challenge

Autorzy: Mauro Cettolo, Andrea Piergentili, Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), 2024
Wydawca: CEUR Workshop Proceedings

An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation

Autorzy: Andrea Piergentili, Beatrice Savoldi, Matteo Negri, Luisa Bentivogli
Opublikowane w: Proceedings of the 3rd Workshop on Gender-Inclusive Translation Technologies (GITT 2025), 2025
Wydawca: European Association for Machine Translation

From Speech to Summary: A Comprehensive Survey of Speech Summarization

Autorzy: Retkowski, Fabian; Züfle, Maike; Sudmann, Andreas; Pfau, Dinah; Niehues, Jan; Waibel, Alexander
Opublikowane w: 2025
Wydawca: EMNLP 2025 Main
DOI: 10.5445/IR/1000180972

Recent Highlights in Multilingual and Multimodal Speech Translation

Autorzy: Danni Liu and Jan Niehues
Wydawca: IWSLT 2024
DOI: 10.48550/ARXIV.2506.11160

Impact of Face Alignment on Face Image Quality

Autorzy: Eren Onaran, Erdi Sarıtaş, Hazım Kemal Ekenel
Opublikowane w: EAI ROSENET 2024
Wydawca: EAI ROSENET 2024
DOI: 10.48550/ARXIV.2507.20953

Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collection

Autorzy: Beomseok Lee, Marco Gaido, Ioan Calapodescu, Laurent Besacier, Matteo Negri
Opublikowane w: Proceedings of the 31st International Conference on Computational Linguistics, 2025
Wydawca: Association for Computational Linguistics

PIER: A Novel Metric for Evaluating What Matters in Code-Switching

Autorzy: Enes Yavuz Ugan, Ngoc-Quan Pham, Leonard Bärmann, and Alex Waibel
Wydawca: ICASSP 2025
DOI: 10.48550/ARXIV.2501.09512

FAMA: The First Large-Scale Open-Science Speech Foundation Model for Italian and English

Autorzy: Sara Papi*, Marco Gaido*, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri.
Wydawca: CLIC-it 2025
DOI: 10.48550/ARXIV.2505.22759

Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

Autorzy: Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, Barry Haddow
Opublikowane w: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2025.NAACL-LONG.153

Continuously Learning New Words in Automatic Speech Recognition

Autorzy: Christian Huber, Alexander Waibel
Wydawca: ICASSP 2025
DOI: 10.48550/ARXIV.2401.04482

Cross-lingual Evaluation of Multilingual Text Generation

Autorzy: Shamil Chollampatt, Minh Quang Pham, Sathish Reddy Indurthi, Marco Turchi
Opublikowane w: Proceedings of the 31st International Conference on Computational Linguistics, 2025
Wydawca: Association for Computational Linguistics

Audio-driven Talking Face Generation with Stabilized Synchronization Loss

Autorzy: Dogucan Yaman, Fevziye Irem Eyiokur, et al.
Wydawca: EMNLP 2025 Main
DOI: 10.48550/ARXIV.2307.09368

Analyzing the Feature Extractor Networks for Face Image Synthesis

Autorzy: Erdi Sarıtaş and Hazım Kemal Ekenel
Wydawca: IEEE FG 2024
DOI: 10.48550/ARXIV.2406.02153

What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

Autorzy: Beatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof-Arenas, Luisa Bentivogli
Opublikowane w: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.EMNLP-MAIN.1002

Explainability for Speech Models: On the Challenges of Acoustic Feature Selection

Autorzy: Dennis Fucci, Beatrice Savoldi, Marco Gaido, Matteo Negri, Mauro Cettolo, Luisa Bentivogli
Opublikowane w: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), 2024
Wydawca: CEUR Workshop Proceedings

Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024

Autorzy: Sai Koneru, Thai-Binh Nguyen, Ngoc-Quan Pham, Danni Liu, Zhaolin Li, Alexander Waibel, Jan Niehues
Wydawca: IWSLT 2024
DOI: 10.48550/ARXIV.2406.16777

Assessing the Use of Face Swapping Methods as Face Anonymizers in Videos

Autorzy: Mustafa İzzet Muştu, Hazım Kemal Ekenel
Wydawca: DSP 2025
DOI: 10.48550/ARXIV.2505.20985

Different Speech Translation Models Encode and Translate Speaker Gender Differently

Autorzy: Dennis Fucci, Marco Gaido, Matteo Negri, Luisa Bentivogli, André Martins, Giuseppe Attanasio
Wydawca: ACL 2025 Main
DOI: 10.48550/ARXIV.2506.02172

How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations

Autorzy: Hyunji Lee, Danni Liu, Supriti Sinhamahapatra, Jan Niehues
Wydawca: NAACL 2025
DOI: 10.48550/ARXIV.2411.17666

Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvement

Autorzy: Tuan-Nam Nguyen, Ngoc-Quan Pham, Seymanur Akti, Alexander Waibel
Wydawca: INTERSPEECH 2025
DOI: 10.48550/ARXIV.2506.16580

How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

Autorzy: Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, et al.
Wydawca: INTERSPEECH 2025
DOI: 10.48550/ARXIV.2409.17044

Findings of the Quality Estimation Shared Task at WMT 2024: Are LLMs Closing the Gap in QE?

Autorzy: Chrysoula Zerva, Frederic Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, André Martins
Opublikowane w: Proceedings of the Ninth Conference on Machine Translation, 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.WMT-1.3

Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

Autorzy: Dogucan Yaman, Fevziye Irem Eyiokur, et al.
Wydawca: CVPRW 2024
DOI: 10.48550/ARXIV.2405.04327

Peer reviewed articles (2)

How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System?

Autorzy: Sara Papi, Peter Polák, Dominik Macháček, Ondřej Bojar
Opublikowane w: Transactions of the Association for Computational Linguistics, Numer 13, 2025, ISSN 2307-387X
Wydawca: MIT Press
DOI: 10.1162/TACL_A_00740

A decade of gender bias in machine translation

Autorzy: Beatrice Savoldi, Jasmijn Bastings, Luisa Bentivogli, Eva Vanmassenhove
Opublikowane w: Patterns, Numer 6, 2025, ISSN 2666-3899
Wydawca: Elsevier BV
DOI: 10.1016/J.PATTER.2025.101257

Other (2)

Facial Attribute Based Text Guided Face Anonymization

Autorzy: Muştu, Mustafa İzzet; Ekenel, Hazım Kemal
Opublikowane w: 2025
Wydawca: arXiv
DOI: 10.48550/ARXIV.2505.21002

Early-Exit and Instant Confidence Translation Quality Estimation

Autorzy: Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, Jan Niehues
Wydawca: arXiv
DOI: 10.48550/ARXIV.2502.14429