Multiscale Super Token Transformer for Hyperspectral Image Classification
Zhe Meng, Taizheng Zhang, Feng Zhao, Gaige Chen, Miaomiao Liang
Abstract
The global modeling capability of vision transformer (ViT) has been well proven in the field of hyperspectral image (HSI) classification. However, ViT does not have the excellent local feature extraction capability compared with the convolutional neural network (CNN). Therefore, early-stage convolutions are often used to enhance ViT’s local representation ability. However, directly applying convolutions on high-dimensional HSI data increases computational overhead. Moreover, recent researches have observed that ViT may suffer from high redundancy in capturing multihead self-attention (MHSA). To address the above issues, we propose a multiscale super token transformer (MSSTT) model for HSI classification. We use a divide-and-conquer strategy to extract local features and global dependencies of HSI data at multiple granularities. Specifically, our proposed model incorporates two branches: a multiscale convolution (MSConv) branch that uses various convolutional kernels to extract diverse local features and a multiscale super token attention (MSSTA) branch for capturing global features with low redundancy. Finally, comparative experimental results with advanced methods show that the proposed MSSTT possesses better classification performance. On the Salinas (SA), Pavia University (PU), and Kennedy Space Center (KSC) datasets, the overall accuracies (OAs) of our MSSTT are 98.47%, 98.47%, and 99.38%, respectively. Code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhangtaizheng/MSSTT</uri>.