Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning
AJ Piergiovanni, Weicheng Kuo, Anelia Angelova
Abstract
We present a simple approach which can turn a ViT en-coder into an efficient video model, which can seamlessly work with both image and video inputs. By sparsely sam-pling the inputs, the model is able to do training and in-ference from both input modalities. The model is easily scalable and can be adapted to large-scale pre-trained ViTs without requiring full finetuning. The model achieves SOTA results <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://sites.google.com/view/tubevit.
Topics & Concepts
ScalabilityComputer scienceArtificial intelligenceComputer visionImage (mathematics)Computer graphics (images)Scale (ratio)DatabasePhysicsQuantum mechanicsMultimodal Machine Learning ApplicationsHuman Pose and Action RecognitionAdvanced Vision and Imaging