LGCT: Local-Global Collaborative Transformer for Fusion of Hyperspectral and Multispectral Images

Abstract

With its strong capability in modeling long-range dependencies, the Transformer achieves competitive performance in hyperspectral image (HSI) and multispectral image (MSI) fusion. However, existing Transformer-based methods face the trade-off between receptive field size and computational efficiency when dealing with spatially non-local features. Furthermore, the Transformer captures deep spectral relationships by modeling pairwise channel interactions. This global interaction may overlook features that contribute little to the overall context but are critical locally, thus affecting the accurate understanding of HSI content. To overcome these challenges, we propose a novel local-global collaborative network with Transformers (LGCT) specifically designed to achieve high-quality HSI reconstruction. The proposed LGCT includes two inverse feature streams to establish multiscale deep representations of the HSI and MSI features. The feature streams comprise collaborative Transformer blocks (CTBs) explicitly designed for the spectral and spatial domains. By combining global and local processing mechanisms, the proposed CTBs can efficiently emphasize potential crucial features that Transformer ignores when capturing deep spectral and spatial relationships, thus enabling efficient modeling of the spectral and spatial domains from details to the whole. Furthermore, to enhance the reusability of multiscale enhanced features from the spectral and spatial domains, a hierarchical and symmetric strategy is adopted to progressively fuse them to generate high-quality images. The results on both simulated and real datasets demonstrate the superior performance of the proposed method in terms of quantitative metrics and visual quality. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Hewq77/LGCT</uri>.

Topics & Concepts