Improving Multilingual Speech Recognition for Cognitive Voice Interfaces Using Real Code-Switching Data
Suresh Kurapati, Muhamed Ihsan, M. G., M Srinju., Lakshmi Priya. N
Abstract
In recent years, Multilingual Speech Recognition (MSR) has become vital for allowing accurate and real-time cognitive voice interfaces, as it eliminates the need for distinct language identification modules. However, traditional approaches face challenges, such as phonetic confusion across languages and poor robustness in natural multilingual conversations. Hence, this research proposes an Artificial Intelligence (AI)-enabled MSR framework based on Waveform to Vector, version 2.0, with Cross-Lingual Speech Representation (Wav2Vec 2.0 -XLSR53) with real CodeSwitching (CS) training data to capture spontaneous language mixing more accurately. The gathered data were then preprocessed using silence trimming, normalization, and segmentation to standardize the input quality. This was followed by embedding generation by the Language-agnostic Bert Sentence Embedding (LaBSE) model in which every sentence pair over several languages was designed based on cosine similarity scores. After that, preprocessing was used to fine-tune the Wav2Vec 2.0 -XLSR53 model for efficient alignment among audio and transcripts. Finally, the proposed framework attained an enhanced Word Error Rate (WER), mainly in midsentence code-switching scenarios, which allowed accurate transcription for cognitive voice interfaces. Experimental calculations on Spanish, Portuguese, and Russian show that the proposed AI-MSRWav2Vec2-XLSR-CS system improves the WER by 12-15% under high-similarity conditions, significantly outperforming baselines trained solely on synthetic bilingual data.