Fast-Slow Transformer for Visually Grounding Speech
Puyuan Peng, David Harwath
2022ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)23 citationsDOI
Abstract
We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.
Topics & Concepts
TransformerComputer scienceEncoderSpeech recognitionWaveformArtificial intelligenceGround truthBenchmark (surveying)VoltageEngineeringElectrical engineeringGeographyGeodesyTelecommunicationsOperating systemRadarMultimodal Machine Learning ApplicationsSpeech and Audio ProcessingSpeech Recognition and Synthesis