UltraSR: Silent Speech Reconstruction via Acoustic Sensing

Abstract

Silent Speech Interfaces (SSI) have been developed to convert silent articulatory gestures into speech, aiding communication in public spaces and assisting individuals with aphasia. Previous SSIs, which rely on wearable devices or cameras, often pose issues like prolonged contact or privacy risks. Recent advancements in acoustic sensing present new opportunities for gesture sensing, but they typically focus on content classification rather than reconstructing audible speech. This results in the loss of crucial speech characteristics such as rate, intonation, and emotion.In this paper, we propose UltraSR, a novel sensing system designed for accurate audible speech reconstruction by analyzing the disturbance of tiny articulatory gestures on reflected ultrasound signals. UltraSR employs a multi-scale feature extraction scheme to aggregate information from multiple views and introduces a new model that maps ultrasound to speech signals, enabling the reconstruction of audible speech from silent gestures.Instead of the laborious collection of massive training data, UltraSR constructs an inverse task to generate virtual gestures from widely available audio (e.g., phone calls) for efficient model training. Additionally, it incorporates a finetuning mechanism using unlabeled data for user adaptation.We implemented UltraSR on a portable smartphone and evaluated it in various environments. Results show that UltraSR can achieve a Character Error Rate (CER) as low as 5.22% and reduce the CER from 80.13% to 6.31% for new users with only 1 hour of ultrasound data, outperforming state-of-the-art acoustic-based approaches while preserving rich speech information.

Topics & Concepts