Wave-Spectrogram Cross-Modal Aggregation for Audio Deepfake Detection

Abstract

Realistic deepfake audio has posed significant security risk. To address this threat, current countermeasures attempt to extract forgery traces from either waveform signals or corresponding spectrograms. However, features extracted from single modality are susceptible to non-spoof disturbances and tend to overfit specific forgery method, thus fail to generalize on out-of-domain forgeries. In this paper, we propose a novel cross-modal deepfake audio detection framework, which leverages multi-scale representations to improve generalization in distinguish unseen synthesized utterances. As spectrogram can reveal hidden features in audio signals, our model learns discriminative and intrinsic feature representations by meticulously aligning features of multiple modalities. Specifically, we design a multiscale fusion strategy to aggregate deepfake artifacts of different scales, overcoming the challenge of aligning heterogeneous multi-modal features. Furthermore, we employ single center loss to condense the embeddings of bonafide audio, enhancing the ability of classifier in detecting out-of-domain deepfake audios. As a result, our approach outperforms the state-of-the-art studies on challenging ASVspoof2021 Deepfake dataset and In-The-Wild dataset. Extensive experiments further demonstrate the effectiveness and generalization of proposed detection framework.

Topics & Concepts