Hyperspectral Image Classification Based on a Locally Enhanced Transformer Network
Shaoguang Huang, Wei Xiao, Hongyu Chen, Siti Khairunniza Bejo, Hongyan Zhang
Abstract
Recently, transformer-based models have achieved remarkable performance in the hyperspectral image (HSI) classification. However, due to the limited training data, existing methods often show limited capability of capturing fine-grained local features. Although attempts have been made to solve this problem, the large amount of parameters imposes the risk of overfitting. In this paper, we propose a locally enhanced transformer network for HSI classification with fewer network parameters, which mainly consists of a multi-branch spatial-spectral tokenization (MSST) module and a dual-branch transformer encoder (DTE) module. The MSST generates effective spatialspectral tokens through diverse convolutions with a residual connection. The DTE consists of a global transformer branch and a locally enhanced transformer branch, which are used to capture the global and local spatial dependencies of HSI, respectively. Unlike the conventional self-attention module used in the global branch, we propose an improved multi-head selfattention (IMSA) module in the local branch by incorporating the local prior information of HSI with graph convolution, to enhance the local information extraction. To fuse the global and local features from the two branches, we introduce an adaptive strategy by using learnable weights for both branches. We devise our MSST and DTE with a shallow architecture, significantly reducing the number of parameters. Experimental results on benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art.