Leveraging OpenAI Whisper Model to Improve Speech Recognition for Dysarthric Individuals

Abstract

Automatic Speech Recognition (ASR) systems are pivotal in facilitating human-technology interactions through voice commands. However, individuals with dysarthria face significant challenges in benefiting from these technologies due to their speech disorder. This paper proposes finetuning the Whisper model for Dysarthric Speech Recognition (DSR) by incorporating additional features extracted from Mel-frequency cepstral coefficients (MFCCs). By combining spectrograms and MFCCs within an attention mechanism, the model creates a richer feature representation, with spectrograms providing broader context and MFCCs highlighting crucial formant frequencies. The attention mechanism dynamically weighs the importance of each feature based on specific speech segments and dysarthric speech characteristics. Furthermore, a hierarchical attention approach is adopted, which encompasses a two-stage attention mechanism. This mechanism directs attention at both local and global levels, facilitating the capture of both fine-grained details and broader contextual information within the speech signal. This study involved the development and training of 45 speaker-adaptive dysarthric ASR systems. The proposed model achieves an average Word Recognition Accuracy (WRA) of 74.08%, showing a notable enhancement compared to the benchmark of 69.23%. The findings underscore the efficacy of the proposed approach in addressing dysarthria-related challenges in ASR systems.

Topics & Concepts