FLEX-CODESWITCH

Objectif

All current successful automatic speech recognition (ASR) systems (largely for English) are trained on hundreds of thousands of hours of training data, invariably based on individual languages. However, multilingual communities switch between two or more languages (code-switching): E What is my bank balance? Bengali/E amar bank balance kɔto? (my bank balance how much). Thus, for the recognition of speech involving code-switching, recognition rates are seriously impaired. Moreover, since code-switching across more than one language can easily violate standard Large Language Models (LLMs), this constitutes a serious setback for recognition. Recently, end-to-end systems have been used to model multilingual speech including code-switching. However, their performance is impeded by the scarcity of massive amounts of good quality code-switching speech and text training data. Errors in these transcriptions can have dangerous impacts when used in AI analytics pathways.

We have successfully built a single-word recognition system (FlexSR) based on linguistic principles (phonological features). We have shown that FlexSR can be adapted across Germanic languages and different accents by altering the lexical representation of words without additional training. It has proven to be accurate, fast, and computationally lightweight for word recognition in Dutch, English and German. The FlexSR system, including the word corpora, is smaller than 10MB. Since the extraction of phonological features makes FlexSR a powerful tool to identify words across languages, we will use additional linguistic information from our current ERC Synergy project PAAL to adapt the current FlexSR. We will extend our mispronunciation detection-based transcription verification system and build a short-phrase code-switching command/query system for banking or multimedia situations. We will focus on Bengali, Hindi and English with code-switching and accent variation across the three.