A Model-Inspired Approach With Transformers for Hyperspectral Pansharpening
Yanli Shang, Jianjun Liu, Jinlong Yang, Zebin Wu
Abstract
Hyperspectral pansharpening is a method of integrating a low-resolution hyperspectral image with a high-resolution panchromatic image to produce a high-resolution hyperspectral image. In recent years, a number of hyperspectral pansharpening methods have been developed by using convolutional neural networks. However, these methods only consider local information due to limitations on the size of convolution kernels in the convolution operation. In this paper, we design a model-inspired approach with transformers for hyperspectral pansharpening. Due to the representation ability of transformers and the algorithmic explanatory ability of optimization models, our method is able to explore intrinsic relationships at a global scale on both spectral and spatial features. First, we formulate an optimization model consisting of a fidelity term and a regularization term. Then, this optimization model is solved by a half quadratic splitting algorithm, and thus divided into two sub-optimization problems, the fidelity and regularization problems. Finally, the algorithm is implemented by a transformer-based deep structure. Specifically, the fidelity problem is solved by a gradient descent algorithm further, and then implemented through a convolutional network. The regularization problem is depicted by an approximation operation and implemented via a transformer network. The experimental results on different satellite datasets demonstrate the effectiveness of the proposed method.