SiT: Scale-Interaction Transformer for Hyperspectral Image Classification

Abstract

Recently, transformers have emerged as a promising technique for capturing saliency feature dependencies that exist in hyperspectral (HS) images for the land-use and land-cover image classification tasks. However, existing transformer-based methods suffer from a significant challenge because they only take into account long-range dependencies between features in a fixed-size neighborhood. To tackle this problem, we proposed a novel scale-interaction transformer (SiT) tailored for HS image classification (HSIC) tasks. Specifically, two initial multiscale feature extraction modules are first designed for the modeling of spatial and spectral features, respectively. Furthermore, we investigate traditional transformer encoders to handle multiscale inputs and suggest a novel transformer-based encoder, called SiT encoder (SiTE), for modeling long-range dependencies of multiscale spectral and spatial features. More specifically, to enable the exploration of global correlation from multiscale perspectives in SiTE, an additional scale set is generated in addition to the query, key, and value sets. More importantly, to balance the contributions of spatial and spectral features during the final classification, we propose a novel loss function, called spectral and spatial consistency constraint (S2C2) loss, which adaptively fuses these features without introducing additional parameters. The experimental evaluations carried out on four public benchmarks showcase that the proposed SiT attains state-of-the-art (SOTA) classification performance and outperforms the previous SOTA by 2.83%, 3.04%, 2.67%, and 5.97% in terms of overall accuracy (OA) across four datasets, respectively. The code will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhonghaochen/SiT</uri>

Topics & Concepts