EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
Jaeyeon Kim, Jae‐Yoon Jung, Jinjoo Lee, Sang Hoon Woo
Abstract
We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model. Experimental results on AudioCaps and Clotho demonstrate that our model surpasses the performance of baseline models. Source code will be available at https://github.com/jaeyeonkim99/EnCLAP. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Topics & Concepts
Computer scienceClosed captioningCodecEmbeddingSpeech recognitionSpeech codingLanguage modelSource codeNatural language processingCode (set theory)Artificial intelligenceProgramming languageComputer hardwareImage (mathematics)Set (abstract data type)Music and Audio ProcessingSpeech Recognition and SynthesisSpeech and Audio Processing