MSCSCformer: Multiscale Convolutional Sparse Coding-Based Transformer for Pansharpening
Y. Ye, Tingting Wang, Faming Fang, Guixu Zhang
Abstract
With the increasing significance of high-quality, high-resolution multispectral images (HRMS) in various domains, pansharpening, which fuses low-resolution multispectral images (LRMS) with high-resolution panchromatic images (PAN), has gained considerable attention. However, current deep learning methods have limitations in capturing global long-range dependencies and incorporating spectral characteristics across different spectral bands of multispectral images (MS). Additionally, model-based approaches do not effectively utilize the multi-scale information between LRMS and HRMS data, limiting their further performance enhancement. To address these limitations, we propose a new observation model based on Multi-Scale Convolutional Sparse Coding (MS-CSC) and design a novel Multi-Scale Hybrid Spatial-spectral Transformer (MSHST) for the unfolding networks. The MS-CSC based observation model aims to fuse multi-scale information, while the MSHST incorporates spatial self-attention to capture global long-range dependencies and spectral self-attention to capture the inter-band correlation. Experimental results demonstrate the superiority of our method over other state-of-the-art approaches in both reduced-resolution and full-resolution evaluations. Ablation experiments further validate the effectiveness of the proposed multi-scale model and MSHST. Code is available at https://github.com/Eternityyx/MSCSCformer.