Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training
Chengyi Wang, Yiming Wang, Yu Wu, Sanyuan Chen, Jinyu Li, Shujie Liu, Furu Wei
Abstract
Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition.It usually requires a codebook obtained in an unsupervised way, making it less accurate and difficult to interpret.We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance and also the pre-training efficiency, either through decoding with a hybrid ASR system to generate phoneme-level alignments (named PBERT), or performing clustering on the supervised speech features extracted from an end-to-end CTC model (named CTC clustering).Both the hybrid and CTC models are trained on the same small amount of labeled speech as used in fine-tuning.Experiments demonstrate significant superiority of our methods to various SSL and self-training baselines, with up to 17.0% relative WER reduction.Our pre-trained models also show good transferability in a non-ASR speech task.