Learning Correlation Structures for Vision Transformers
Manjin Kim, Paul Hongsuck Seo, Cordelia Schmid, Minsu Cho
Abstract
We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recog-nizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate lo-cal contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effective-ness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-I K, Kinetics-400, Something-Something VI & V2, Diving-48, and FineGym.