Multiscale Neighborhood Attention Transformer With Optimized Spatial Pattern for Hyperspectral Image Classification

Abstract

Hyperspectral images (HSIs) provide hundreds of continuous spectral bands and have been widely used for the fine identification of targets with similar appearances. In earlier studies, convolutional neural networks (CNNs) have been an effective method for HSIs classification due to their powerful feature extraction capabilities. Recently, self-attention-based vision transformer (ViT) architecture has been widely explored to fully represent global information. However, most existing transformer-based models primarily focus on global relationships and lack the ability to capture the multi-scale features which are crucial for HSIs classification. This limitation results in inferior performance for transformer-based methods compared to state-of-the-art CNN-based models. To solve this problem, a novel network called multi-scale neighborhood attention transformer (MSNAT) is proposed in this paper. Unlike previous transformer-based models, MSNAT emphasizes the neighborhood pixels within a local window size and extracts multi-scale spatial information by using different local window sizes. In addition, a spatial transformation module is integrated to generate optimized spatial input. The effectiveness of the proposed MSNAT is verified on three real hyperspectral datasets including University of Pavia (UP), University of Houston (UH), and University of Trento (UT). Experimental results demonstrate that the proposed MSNAT method outperforms both CNNs and existing transformer-based models, achieving state-of-the-art classification performance with an overall accuracy of 93.34%, 86.26%, and 96.63% on UP, UH, and UT, respectively. The source code will be available at https://github.com/xinqiao123/MSNAT.

Topics & Concepts