A method based on hybrid cross-multiscale spectral-spatial transformer network for hyperspectral and multispectral image fusion

Abstract

• A HCMSST is proposed for HSI-MSI fusion. This network combines the robust feature extraction abilities of CNNs with the strengths of transformers in capturing long-range dependencies. By extracting image details and semantic information across multiple scales, the model achieves a holistic understanding of image content and enriches the feature representations. • We propose a novel SCDRB module. Through staggered cross-stream dense connections, the network can fully utilize hierarchical features within the same stream and establish feature interactions between the two streams. Among them, cross-stream dense connections enable the network to utilize both spectral and spatial multiscale information for a more comprehensive and effective feature representation, while staggered connections reduce the computational overhead and indirectly alleviate the gradient vanishing problem in deep networks. • To effectively explore spectral and spatial features while simultaneously utilizing information from disparate network hierarchies, we propose a cross-level dual-stream feature interaction strategy. This strategy pivots on multi-modal feature fusion across different hierarchical levels. In addition, the fused features are propagated to the corresponding branches in a pixel-wise manner to facilitate information integration and interaction. Convolutional neural networks (CNNs) have made a significant contribution to hyperspectral image (HSI) generation. However, capturing long-range dependencies can be challenging with CNNs due to the limitations of their local receptive fields, which can lead to distortions in fused images. Transformers excel at capturing long-range dependencies but have limited capacity for handling fine details. Additionally, prior work has often overlooked the extraction of global features during the image preprocessing stage, resulting in the potential loss of fine details. To address these issues, we propose a hybrid cross-multiscale spectral-spatial Transformer (HCMSST) that combines the advantages of CNNs in feature extraction and Transformers in capturing long-range dependencies. To fully extract and retain local and global information in the shallow feature extraction phase, the network incorporates CNNs with a staggered cascade-dense residual block (SCDRB). This block employs staggered residuals to establish direct connections both within and between branches and integrates attention modules to enhance the response to important features. This approach facilitates unrestricted information exchange and fosters deeper feature representations. To address the limitations of Transformer in processing fine details, we introduce multiscale spatial-spectral coding-decoding structures to obtain comprehensive spatial-spectral features, which are utilized to capture the long-range dependencies via the cross-multiscale spectral-spatial Transformer (CMSST). Further, the CMSST incorporates a cross-level dual-stream feature interaction strategy that integrates spatial and spectral features from different levels and then feeds the fused features back to their corresponding branches for information interaction. Experimental results indicate that the proposed HCMSST achieves superior performance compared to many state-of-the-art (SOTA) methods. Specifically, HCMSST reduces the ERGAS metric by 3.05% compared to the SOTA methods on the CAVE dataset, while on the Harvard dataset, it achieves a 2.69% reduction in ERGAS compared to the SOTA results.

Topics & Concepts