Rethinking the Self-Attention in Vision Transformers
Kyungmin Kim, Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Zhicheng Yan, Péter Vajda, Seon Wook Kim
Abstract
Self-attention is a corner stone for transformer models. However, our analysis shows that self-attention in vision transformer inference is extremely sparse. When applying a sparsity constraint, our experiments on image (ImageNet- 1K) and video (Kinetics-400) understanding show we can achieve 95% sparsity on the self-attention maps while main-taining the performance drop to be less than 2 points. This motivates us to rethink the role of self-attention in vision transformer models.
Topics & Concepts
TransformerInferenceComputer scienceArtificial intelligenceComputer visionMachine learningEngineeringVoltageElectrical engineeringAdvanced Neural Network ApplicationsVisual Attention and Saliency DetectionDomain Adaptation and Few-Shot Learning