MD³Net: Integrating Model-Driven and Data-Driven Approaches for Pansharpening
Yinsong Yan, Junmin Liu, Shuang Xu, Yicheng Wang, Xiangyong Cao
Abstract
Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multi-modal fusion task, in this paper, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD<sup>3</sup>Net), which combines model-driven and data-driven approaches. The architecture design of MD<sup>3</sup>Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input/output predictable. In order to further explore the powerful learning ability of deep learning based approaches, we introduce the deep prior into the MD<sup>3</sup>Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared to a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet..