Exchanges for SPEech ReseArch aNd TechnOlogies

Project Information

ESPERANTO

Grant agreement ID: 101007666

DOI 10.3030/101007666

EC signature date 27 November 2020

Start date 1 January 2021

End date 31 December 2025

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost € 1 191 400,00

EU contribution € 1 191 400,00

Coordinated by

UNIVERSITE DU MANS
France

Deliverables

Documents, reports (3)

Communication and Dissemination Plan

Organise communication activities, workshops, courses and other knowledge share actions. Ensure that the dissemination is compliant with open access requirements.

Description of explainability for speech

This deliverable will provide a list and description of criteria for explainability in the context of speech processing

Scientific dissemination guidelines

Identify partners specificities and needs Writing guidelines to the attention of partners to ensure an active and coherent communication activity among countries

Websites, patent fillings, videos etc. (1)

Project website and visual identity

Setting up a visual identity and a website presenting to the general public the objectives of the project, its partners and the main tools employed. The website will also be a resource for the partners for all that concerns good practices.

Demonstrators, pilots, prototypes (2)

SDK for human assisted learning speech processing

Software library developed in open-source licence to tackle to issue of human assisted learning for various tasks of speech processing such as speaker diarization, speaker verification, speech translation of speech recognition.

Models that learn from small data

To deal with under-resource task, this deliverable will implement different approaches dealing with low or zero resource learning such as transfer learning and modeling based on the same information shared between languages or tasks, approaches that do not need annotated data, use of expert knowledge in empirical systems, systems using multimodal data for semantic supervision.

Publications

Conference proceedings (35)

Cross-Corpus Speech Emotion Recognition with HuBERT Self-Supervised Representation

Author(s): Miguel Pastor; Dayana Ribas; Alfonso Ortega; Antonio Miguel; Eduardo Lleida
Published in: Proc. IberSPEECH, Issue 12, 2022
Publisher: ISCA
DOI: 10.21437/iberspeech.2022-16

Multi-Channel Speaker Verification with Conv-Tasnet Based Beamformer

Author(s): Mošner Ladislav, Plchot Oldřich, Burget Lukáš, Černocký Jan
Published in: Proceedings of ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, Page(s) 7982-7986, ISBN 978-1-6654-0540-9
Publisher: IEEE Signal Processing Society
DOI: 10.1109/icassp43922.2022.9747771

A transfer learning based approach for pronunciation scoring

Author(s): Sancinetti, Marcelo; Vidal, Jazmin; Bonomi, Cyntia; Ferrer, Luciana
Published in: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Issue 4, 2022, Page(s) 6812-6816, ISBN 978-1-6654-0540-9
Publisher: IEEE
DOI: 10.48550/arxiv.2111.00976

Microphone Array Channel Combination Algorithms for Overlapped Speech Detection

Author(s): Theo Mariotte; Anthony Larcher; Silvio Montrésor; Jean-Hugh Thomas
Published in: Interspeech 2022 Human and Humanizing Speech Technology, Issue 12, 2022
Publisher: ISCA
DOI: 10.21437/interspeech.2022-10758

Log-Likelihood-Ratio Cost Function as Objective Loss for Speaker Verification Systems

Author(s): Victoria Mingote, Antonio Miguel, Alfonso Ortega, Eduardo Lleida
Published in: Interspeech 2021, 2021, Page(s) 2361-2365
Publisher: ISCA
DOI: 10.21437/interspeech.2021-1085

Extracting Speaker and Emotion Information from Self-Supervised Speech Models via Channel-Wise Correlations

Author(s): Themos Stafylakis; Ladislav Mosner; Sofoklis Kakouros; Oldrich Plchot; Lukas Burget; Jan Cernocky
Published in: 2022 IEEE Spoken Language Technology Workshop (SLT), Issue 22, 2023, ISBN 979-8-3503-9690-4
Publisher: IEEE
DOI: 10.1109/slt54892.2023.10023345

Learnable Sparse Filterbank for Speaker Verification

Author(s): PENG Junyi, GU Rongzhi, MOŠNER Ladislav, PLCHOT Oldřich, BURGET Lukáš and ČERNOCKÝ Jan.
Published in: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022, Page(s) 5110-5114
Publisher: International Speech Communication Association (ISCA)
DOI: 10.21437/interspeech.2022-11309

On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding

Author(s): Gaelle Laperriere; Valentin Pelloin; Mickael Rouvier; Themos Stafylakis; Yannick Esteve
Published in: 2022 IEEE Spoken Language Technology Workshop (SLT), Issue 24, 2023
Publisher: IEEE
DOI: 10.1109/slt54892.2023.10023013

Multisv: Dataset for Far-Field Multi-Channel Speaker Verification

Author(s): Mošner Ladislav, Plchot Oldřich, Burget Lukáš, Černocký Jan
Published in: Proceedings of ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, Page(s) 7977-7981, ISBN 978-1-6654-0540-9
Publisher: IEEE Signal Processing Society
DOI: 10.1109/icassp43922.2022.9746833

Probabilistic Spherical Discriminant Analysis: An Alternative to PLDA for length-normalized embeddings.

Author(s): BRUMMER Johan Nikolaas Langenhoven, SWART Albert du Preez, MOŠNER Ladislav, SILNOVA Anna, PLCHOT Oldřich, STAFYLAKIS Themos and BURGET Lukáš.
Published in: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH., 2022, Page(s) 1446-1450
Publisher: International Speech Communication Association (ISCA)
DOI: 10.21437/interspeech.2022-731

Speaker Embeddings for Diarization of Broadcast Data In The Allies Challenge

Author(s): Anthony Larcher; Ambuj Mehrish; Marie Tahon; Sylvain Meignier; Jean Carrive; David Doukhan; Olivier Galibert; Nicholas Evans
Published in: ICASSP, Issue 1, 2021
Publisher: IEEE
DOI: 10.1109/icassp39728.2021.9414215

Semantic Enrichment Towards Efficient Speech Representations

Author(s): Gaëlle Laperrière; Ha Nguyen; Sahar Ghannay; Bassam Jabaian; Yannick Estève
Published in: Proc. INTERSPEECH 2023, 2023, Page(s) 705-709, ISSN 1990-9772
Publisher: ISCA
DOI: 10.21437/interspeech.2023-2234

Speech-Based Emotion Recognition with Self-Supervised Models Using Attentive Channel-Wise Correlations and Label Smoothing

Author(s): Sofoklis Kakouros; Themos Stafylakis; Ladislav Mošner; Lukáš Burget
Published in: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Issue 19, 2023, Page(s) pp. 1-5, ISBN 978-1-7281-6327-7
Publisher: IEEE
DOI: 10.1109/icassp49357.2023.10094673

Multi-Channel Speech Separation with Cross-Attention and Beamforming

Author(s): "Ladislav Mosner, Oldřich Plchot, Junyi Peng, Lukáš Burget, Jan ""Honza"" Černocký"
Published in: Proc. INTERSPEECH 2023, 2023, Page(s) 693-1697, ISSN 1990-9772
Publisher: International Speech Communication Association
DOI: 10.21437/interspeech.2023-2537

Speaker Embeddings by Modeling Channel-Wise Correlations

Author(s): Themos Stafylakis, Johan Rohdin, Lukáš Burget
Published in: Interspeech 2021, 2021, Page(s) 501-505
Publisher: ISCA
DOI: 10.21437/interspeech.2021-1442

A Study on the Use of wav2vec Representations for Multiclass Audio Segmentation

Author(s): Pablo Gimeno; Alfonso Ortega; Antonio Miguel; Eduardo Lleida
Published in: Proc. IberSPEECH2022, Issue 8, 2022, Page(s) 56-60
Publisher: ISCA
DOI: 10.21437/iberspeech.2022-12

End-to-End Speech Translation of Arabic to English Broadcast News

Author(s): Fethi Bougares; Salim Jouili
Published in: WANLP@ACL 2022: Abu Dhabi, United Arab Emirates, Issue 5, 2022, Page(s) 312–319, ISBN 978-1-959429-27-2
Publisher: Association for Computational Linguistics
DOI: 10.18653/v1/2022.wanlp-1.29

Improving Speaker Verification with Self-Pretrained Transformer Models

Author(s): "Junyi Peng, Oldřich Plchot, Themos Stafylakis, Ladislav Mosner, Lukáš Burget, Jan ""Honza"" Černocký"
Published in: Proc. INTERSPEECH 2023, 2023, Page(s) 5361-5365, ISSN 1990-9772
Publisher: International Speech Communication Association
DOI: 10.21437/interspeech.2023-453

Training Speaker Embedding Extractors Using Multi-Speaker Audio with Unknown Speaker Boundaries.

Author(s): STAFYLAKIS Themos, MOŠNER Ladislav, PLCHOT Oldřich, ROHDIN Johan A., SILNOVA Anna, BURGET Lukáš and ČERNOCKÝ Jan.
Published in: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022, Page(s) 605-609
Publisher: International Speech Communication Association (ISCA)
DOI: 10.21437/interspeech.2022-10165

Improving Speaker Diarization for Low-Resourced Sarawak Malay Language Conversational Speech Corpus

Author(s): Mohd Zulhafiz Rahim; Sarah Samson Juan; Fitri Suraya Mohamad
Published in: 2023 International Conference on Asian Language Processing (IALP), 2023, Page(s) 228-233
Publisher: IEEE
DOI: 10.1109/ialp61005.2023.10337314

Multi-Speaker and Wide-Band Simulated Conversations as Training Data for End-to-End Neural Diarization

Author(s): Federico Landini; Mireia Diez; Alicia Lozano-Diez; Lukáš Burget
Published in: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Issue 6, 2023, Page(s) 1-5, ISBN 978-1-7281-6327-7
Publisher: IEEE
DOI: 10.1109/icassp49357.2023.10097049

Description and Analysis of ABC Submission to NIST LRE 2022

Author(s): Pavel Matejka, Anna Silnova, Josef Slavíček, Ladislav Mosner, Oldřich Plchot, Michal Klčo, Junyi Peng, Themos Stafylakis, Lukáš Burget
Published in: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2023, Page(s) 511-515, ISSN 1990-9772
Publisher: International Speech Communication Association
DOI: 10.21437/interspeech.2023-1529

An Attention-Based Backend Allowing Efficient Fine-Tuning of Transformer Models for Speaker Verification

Author(s): Junyi Peng; Oldrich Plchot; Themos Stafylakis; Ladislav Mosner; Lukas Burget; Jan Cernocky
Published in: 2022 IEEE Spoken Language Technology Workshop (SLT), Issue 5, 2023, Page(s) 555-562, ISBN 979-8-3503-9690-4
Publisher: IEEE
DOI: 10.1109/slt54892.2023.10022775

Active Correction for Incremental Speaker Diarization of a Collection with Human in the Loop

Author(s): Larcher, Yevhenii Prokopalo; Meysam Shamsi; Loïc Barrault; Sylvain Meignier; Anthony
Published in: Applied Sciences; Volume 12; Issue 4;, Issue 20, 2022, Page(s) Pages: 1782, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app12041782

Unsupervised Representation Learning for Speech Activity Detection in the Fearless Steps Challenge 2021

Author(s): Pablo Gimeno, Alfonso Ortega, Antonio Miguel, Eduardo Lleida
Published in: Interspeech 2021, 2021, Page(s) 4359-4363
Publisher: ISCA
DOI: 10.21437/interspeech.2021-309

An Explainable Proxy Model for Multilabel Audio Segmentation

Author(s): Mariotte, Théo; Almudévar, Antonio; Tahon, Marie; Ortega, Alfonso
Published in: International Conference on Acoustics Speech and Signal Processing, IEEE, Apr 2024, Seoul (Korea), Issue 5, 2024
Publisher: IEEE Signal Processing Society
DOI: 10.1109/icassp48485.2024.10446648

Analyzing speaker verification embedding extractors and back-ends under language and channel mismatch

Author(s): SILNOVA Anna, STAFYLAKIS Themos, MOŠNER Ladislav, PLCHOT Oldřich, ROHDIN Johan A., MATĚJKA Pavel, BURGET Lukáš, GLEMBEK Ondřej a BRUMMER Johan Nikolaas Langenhoven.
Published in: Proceedings of The Speaker and Language Recognition Workshop (Odyssey 2022), 2022, Page(s) 9-16
Publisher: International Speech Communication Association (ISCA)
DOI: 10.21437/odyssey.2022-2

Toroidal Probabilistic Spherical Discriminant Analysis

Author(s): Anna Silnova; Niko Brümmer; Albert Swart; Lukáš Burget
Published in: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Issue 2, 2023, Page(s) 1-5, ISBN 978-1-7281-6327-7
Publisher: IEEE
DOI: 10.1109/icassp49357.2023.10095580

Sur la vérification du locuteur à partir de traces d’exécution de modèles acoustiques personnalisés

Author(s): Tomashenko, Natalia; Mdhaffar, Salima; Tommasi, Marc; Estève, Yannick; Bonastre, Jean-François
Published in: Journées d'Études sur la Parole - JEP2022, Jun 2022, Île de Noirmoutier, France, Issue 13, 2022
Publisher: ISCA

ON-TRAC Consortium Systems for the IWSLT 2023 Dialectal and Low-resource Speech Translation Tasks

Author(s): Antoine Laurent, Souhir Gahbiche, Ha Nguyen, Haroun Elleuch, Fethi Bougares, Antoine Thiol, Hugo Riguidel, Salima Mdhaffar, Gaëlle Laperrière, Lucas Maison, Sameer Khurana, Yannick Estève
Published in: Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), 2023, Page(s) 219–226, ISBN 2-9517408-4-0
Publisher: Association for Computational Linguistics
DOI: 10.18653/v1/2023.iwslt-1.18

Strategies for Improving Low Resource Speech to Text Translation Relying on Pre-trained ASR Models

Author(s): Santosh Kesiraju; Marek Sarvaš; Tomáš Pavlíček; Cécile Macaire; Alejandro Ciuba
Published in: Proc. INTERSPEECH 2023, Issue 14, 2023, Page(s) 2148--2152, ISSN 1990-9772
Publisher: ISCA
DOI: 10.21437/interspeech.2023-2506

Discriminative Training of VBx Diarization

Author(s): Klement, Dominik; Diez, Mireia; Landini, Federico; Burget, Lukáš; Silnova, Anna; Delcroix, Marc; Tawara, Naohiro
Published in: Crossref, Issue 5, 2024
Publisher: IEEE
DOI: 10.1109/icassp48485.2024.10446119

BUT CHiME-7 system description

Author(s): Karafiát, Martin; Veselý, Karel; Szöke, Igor; Mošner, Ladislav; Beneš, Karel; Witkowski, Marcin; Barchi, Germán; Pepino, Leonardo
Published in: CHiME-7 proceedings, Issue 8, 2023
Publisher: arxiv
DOI: 10.48550/arxiv.2310.11921

Development of ABC systems for the 2021 edition of NIST Speaker Recognition Evaluation

Author(s): ALAM Jahangir, BURGET Lukáš, GLEMBEK Ondřej, MATĚJKA Pavel, MOŠNER Ladislav, PLCHOT Oldřich, ROHDIN Johan A., SILNOVA Anna and STAFYLAKIS Themos et al.
Published in: Proceedings of The Speaker and Language Recognition Workshop (Odyssey 2022), 2022, Page(s) 346-353
Publisher: International Speech Communication Association (ISCA)
DOI: 10.21437/odyssey.2022-48

Parameter-Efficient Transfer Learning of Pre-Trained Transformer Models for Speaker Verification Using Adapters

Author(s): Junyi Peng; Themos Stafylakis; Rongzhi Gu; Oldřich Plchot; Ladislav Mošner; Lukáš Burget; Jan Černocký
Published in: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Issue 6, 2023, Page(s) 1-5, ISBN 978-1-7281-6327-7
Publisher: IEEE
DOI: 10.1109/icassp49357.2023.10094795

Peer reviewed articles (13)

Generalizing AUC Optimization to Multiclass Classification for Audio Segmentation With Limited Training Data

Author(s): Pablo Gimeno; Victoria Mingote; Alfonso Ortega; Antonio Miguel; Eduardo Lleida
Published in: IEEE Signal Processing Letters, Issue 4, 2021, Page(s) 1135 - 1139, ISSN 1070-9908
Publisher: Institute of Electrical and Electronics Engineers
DOI: 10.1109/lsp.2021.3084501

YembaTones: a syllable-tone annotated dataset for speech recognition and prosodic analysis of the Yemba language.

Author(s): Kenfack Jeuguim Marc Sturm; Paulin Melatagia Yonta; Sandembouo Etienne
Published in: Data in Brief, Issue 11, 2023, ISSN 2352-3409
Publisher: Elsevier BV
DOI: 10.1016/j.dib.2023.109860

Towards Lifelong Human Assisted Speaker Diarization

Author(s): Meysam Shamsi; Anthony Larcher; Loic Barrault; Sylvain Meignier; Yevheni Prokopalo; Marie Tahon; Ambuj Mehrish; Simon Petitrenaud; Olivier Galibert; Samuel Gaist; André Anjos; Sebastien Marcel; Marta R. Costa-jussà
Published in: Computer Speech & Language, Issue 4, 2023, ISSN 0885-2308
Publisher: Academic Press
DOI: 10.1016/j.csl.2022.101437

aDCF Loss Function for Deep Metric Learning in End-to-End Text-Dependent Speaker Verification Systems

Author(s): Victoria Mingote; Antonio Miguel; Dayana Ribas; Alfonso Ortega; Eduardo Lleida
Published in: IEEE/ACM Transactions on Audio, Speech, and Language Processing, Issue 25 January 2022, 2022, Page(s) 772-784, ISSN 2329-9290
Publisher: IEEE Advancing Technology for Humanity
DOI: 10.1109/taslp.2022.3145307

Direct Text to Speech Translation System Using Acoustic Units

Author(s): Victoria Mingote, Pablo Gimeno, Luis Vicente, Sameer Khurana, Antoine Laurent and Jarod Duret
Published in: IEEE Signal Processing Letters, Issue vol. 30, 2023, Page(s) 1262-1266, ISSN 1070-9908
Publisher: Institute of Electrical and Electronics Engineers
DOI: 10.1109/lsp.2023.3313513

An Overview of the IberSpeech-RTVE 2022 Challenges on Speech Technologies

Author(s): Eduardo Lleida; Luis Javier Rodriguez-Fuentes; Javier Tejedor; Alfonso Ortega; Antonio Miguel; Virginia Bazán; Carmen Pérez; Alberto de Prada; Mikel Penagarikano; Amparo Varona; Germán Bordel; Doroteo Torre-Toledano; Aitor Álvarez; Haritz Arzelus
Published in: Applied Sciences; Volume 13; Issue 15;, Issue 5, 2023, Page(s) Pages: 8577, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app13158577

Class token and knowledge distillation for multi-head self-attention speaker verification systems

Author(s): Victoria Mingote; Antonio Miguel; Alfonso Ortega; Eduardo Lleida
Published in: Digital Signal Processing, Issue 6, 2023, ISSN 1051-2004
Publisher: Academic Press
DOI: 10.1016/j.dsp.2022.103859

Multimodal Diarization Systems by Training Enrollment Models as Identity Representations

Author(s): Victoria Mingote; Ignacio Viñals; Pablo Gimeno; Antonio Miguel; Alfonso Ortega; Eduardo Lleida
Published in: Applied Sciences, Issue 12-3, 2022, Page(s) 1141, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app12031141

Wiener Filter and Deep Neural Networks: A Well-Balanced Pair for Speech Enhancement

Author(s): Dayana Ribas; Antonio Miguel; Alfonso Ortega; Eduardo Lleida
Published in: Applied Sciences, Issue 3 ; Volume 12; Issue 18;, 2022, Page(s) Pages: 9000, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app12189000

Automatic Voice Disorder Detection Using Self-Supervised Representations

Author(s): Dayana Ribas; Miguel A. Pastor; Antonio Miguel; David Martinez; Alfonso Ortega; Eduardo Lleida
Published in: IEEE Access, Issue 6, 2023, Page(s) 14915-14927,, ISSN 2169-3536
Publisher: Institute of Electrical and Electronics Engineers Inc.
DOI: 10.1109/access.2023.3243986

Cross-Corpus Training Strategy for Speech Emotion Recognition Using Self-Supervised Representations

Author(s): Miguel A. Pastor; Dayana Ribas; Alfonso Ortega; Antonio Miguel; Eduardo Lleida
Published in: Applied Sciences, Issue 13, 2023, Page(s) 9062, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app13169062

Unsupervised Adaptation of Deep Speech Activity Detection Models to Unseen Domains

Author(s): Pablo Gimeno; Dayana Ribas; Alfonso Ortega; Antonio Miguel; Eduardo Lleida
Published in: Applied Sciences, Issue 3, 2022, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app12041832

The Domain Mismatch Problem in the Broadcast Speaker Attribution Task

Author(s): Ignacio Viñals, Alfonso Ortega, Antonio Miguel, Eduardo Lleida
Published in: Applied Sciences, Issue 11/18, 2021, Page(s) 8521, ISSN 2076-3417
Publisher: MDPI
DOI: 10.3390/app11188521