Auto-captions on GIF

Yingwei Pan, Yehao Li, Jianjie Luo, Jun Xu, Ting Yao, Tao Mei

2022Proceedings of the 30th ACM International Conference on Multimedia17 citationsDOI

Abstract

In this work, we present Auto-captions on GIF (ACTION), which is a new large-scale pre-training dataset for generic video understanding. All video-sentence pairs are created by automatically extracting and filtering video caption annotations from billions of web pages. Auto-captions on GIF dataset can be utilized to pre-train the generic feature representation or encoder-decoder structure for video captioning, and other downstream tasks (e.g., sentence localization in videos, video question answering, etc.) as well. We present a detailed analysis of Auto-captions on GIF dataset in comparison to existing video-sentence datasets. We also provide an evaluation of a Transformer-based encoder-decoder structure for vision-language pre-training, which is further adapted to video captioning downstream task and yields the compelling generalizability on MSR-VTT. The dataset is available at http://www.auto-video-captions.top/2022/dataset.

Topics & Concepts

Closed captioningComputer scienceSentenceTransformerEncoderArtificial intelligenceNatural language processingAutomatic summarizationGeneralizability theoryTask (project management)Feature (linguistics)Speech recognitionInformation retrievalImage (mathematics)MathematicsLinguisticsStatisticsManagementPhysicsPhilosophyEconomicsOperating systemVoltageQuantum mechanicsMultimodal Machine Learning ApplicationsHuman Pose and Action RecognitionDomain Adaptation and Few-Shot Learning