CESFusion: Cross-Frequency Enhanced Spatial—Spectral Fusion Network for Hyperspectral and Multispectral Image Fusion

Abstract

The fusion of hyperspectral and multispectral images involves integrating high spectral resolution hyperspectral image (HSI) and high spatial resolution multispectral image (MSI) to generate a HSI with high spatial and spectral resolution (HR-HSI). Existing HSI-MSI fusion methods primarily focus on information fusion within the spatial domain; however, few solutions have explored the employment of frequency analysis to enhance spatial resolution, limiting their capability for global perception. In this paper, we propose an efficient and novel paradigm for HSI-MSI fusion through the cross-frequency enhanced spatial-spectral fusion network, named CESFusion, exploring the complementary fusion of information between the spatial and frequency domains. Specifically, we first present the cross-frequency domain fusion module (CFFM) to perform global analysis through the Fourier transform and effectively integrate and enhance the frequency domain information from both HSI and MSI. Subsequently, we propose the spectral modeling module (SpeMM) based on state space model (SMM) to capture long-range spectral dependencies with linear complexity, and integrate it with the spatial residual block-based module (SRM) for joint spatial-spectral feature extraction. Finally, to enable sufficient interaction between the spatial and frequency domains, we adopt the cross-domain interaction module (CDIM), capturing and integrating complementary information from both domains. Moreover, a frequency-based loss function is purposely designed to further improve the restoration of global information. Extensive experiments conducted on both synthetic and real datasets demonstrate the superiority of our CESFusion, as evidenced by both quantitative and qualitative evaluation results.

Topics & Concepts