A Novel Spatial Fidelity With Learnable Nonlinear Mapping for Panchromatic Sharpening
Rui Wen, Liang-Jian Deng, Zhong-Cheng Wu, Xiao Wu, Gemine Vivone
Abstract
The purpose of panchromatic sharpening, i.e., pansharpening, is to fuse a low spatial resolution multispectral (LRMS) image with a high spatial resolution panchromatic (PAN) image, aiming to obtain a high spatial resolution multispectral (HRMS) image. Pansharpening models based on variational optimization consist of a spectral fidelity term, a spatial fidelity term, and a regularization term. Most of the methods assume that the existing PAN image and the homologous HRMS image satisfy the global or local linear relationship, which could be far from the real case, thus causing sub-optimal performance. Inspired by the nonlinear mapping ability of machine learning (ML) techniques, we propose a novel spatial fidelity term with learnable nonlinear mapping (LNM-SF), which trains an implicit functional operator via a specifically designed convolutional neural network (CNN) and efficiently constructs the nonlinear relationship between the known PAN and the latent HRMS images. Relying upon the above description of the spatial fidelity term, a new variational model with a learnable nonlinear mapping in the spatial fidelity term for pansharpening, named LNM-PS, is simply integrated by the conventional spectral fidelity term into the proposed LNM-SF. To effectively solve the resulting optimization problem, we develop an alternating direction method of multipliers (ADMM)-based algorithm with the fast iterative shrinkage-thresholding algorithm (FISTA) as inner solver. Extensive numerical experiments on different datasets, assessing the performance both at reduced-resolution and full-resolution, show the superiority of the proposed LNM-PS method. The code is available at https://github.com/liangjiandeng/-LNM-PS.