S2DBFT: Spectral–Spatial Dual-Branch Fusion Transformer for Hyperspectral Image Classification
Yiheng Zhang, Ziqiang Wang, Meng Huang, Ming Li, Jian Zhang, Shandong Wang, Jinglin Zhang, Heng Zhang
Abstract
Convolutional neural networks (CNNs) and Transformer-based models have achieved remarkable success in hyperspectral image (HSI) classification tasks due to their outstanding ability to extract spatial and spectral features. However, most existing methods process spatial and spectral features separately, making it difficult to effectively learn their interactive features. To address this issue, we propose a spectral-spatial dual-branch fusion Transformer (S2DBFT) for HSI classification. Initially, we construct a spectral feature extraction module (SPEEM) and a spatial feature extraction module (SPAEM) to extract low-level features. These two modules consist of a one-dimensional convolution layer and a two-dimensional convolution layer, respectively, performing shallow extraction of spectral and spatial features. Next, the two feature sets obtained are fused through a weighted fusion process. Additionally, we design a multi-head spectral-spatial self-attention (MHS3A) mechanism to enhance the interactive fusion of spectral and spatial features. Upon completion of feature fusion, a linear layer is used to obtain the sample labels. Extensive experiments on four HSI datasets demonstrate the effectiveness of the proposed S2DBFT, compared to existing state-of-the-art methods. In terms of performance evaluation, the overall accuracy and average accuracy indicate the superiority and generalizability of S2DBFT.