MLFF-GAN: A Multilevel Feature Fusion With GAN for Spatiotemporal Remote Sensing Images
Bingze Song, Peng Liu, Jun Li, Lizhe Wang, Luo Zhang, Guojin He, Lajiao Chen, Jianbo Liu
Abstract
Due to the limitation of technology and budget, it is often difficult for sensors of a single remote sensing satellite to have both high temporal resolution and high spatial (HTHS) resolution at the same time. In this paper, we proposed a new Multi-level Feature Fusion with Generative Adversarial Network (MLFF-GAN) for generating fusion HTHS images. MLFF-GAN mainly uses U-net-like architecture and its generator is composed of three stages: feature extraction, feature fusion, and image reconstruction. In feature extraction and reconstruction stage, the generator employs the encoding and decoding structure to extract three groups of multi-level features, which can cope with the huge difference of resolution between high-resolution images and low-resolution images. In the feature fusion stage, Adaptive Instance Normalization (AdaIN) block is designed to learn the global distribution relationship between multi-temporal images, and an attention module (AM) is used to learn the local information weights for the change of small areas. The proposed MLFF-GAN was tested on two Landsat and MODIS datasets. Some state-of-the-art algorithms are comprehensively compared with MLFF-GAN. We also carried on the ablation experiment to test the effectiveness of different sub-module in MLFF-GAN. The experiment results and ablation analysis show the better performances of the proposed method when compared with other methods. The code is available at https://github.com/songbingze/MLFF-GAN.