Hyperspectral and Panchromatic Images Fusion Based on the Dual Conditional Diffusion Models
Shuangliang Li, Siwei Li, Lihao Zhang
Abstract
The fusion between the low resolution hyperspectral image (LRHSI) and the panchromatic (PAN) image could obtain the high-resolution hyperspectral image (HRHSI). Recently, deep learning (DL)-based fusion methods have been explored widely due to their powerful feature learning ability. However, most DL-based methods that use the one-step fusion manner can suffer from the blurring effect. In addition, they have not fully utilized the spatial and spectral feature information of two input images, which hinders the improvement of the resulting image quality. Therefore, to fully mitigate the blurring effect and utilize two input images, we propose a dual conditional diffusion models-based fusion network (DCDMF) to obtain the fused HRHSI. The conditional diffusion model (CDM) can generate the high quality image with realistic details in an iterative denoising manner (in the inference sampling stage) other than the one-step fusion manner, which could mitigate the blurring effect greatly. To improve the spatial and spectral fidelity of the fused HRHSI, we propose the dual spatial and spectral CDM (two noise prediction networks with different conditional input) to respectively extract the image feature from the LRHSI and PAN images with different image characteristics and reconstruct the corresponding HRHSI feature and fuse them. In addition, considering the high-dimensional property of the HSI, we pre-train an auto-encoder to encode the HSI into the low-dimensional latent space with more discriminate features to reduce the computational cost. Based on the auto-encoder, we also perform the image generation process in the residual latent space to focus on learning the residual latent spatial details. Extensive experimental results on three datasets show the superiority of our method over several state-of-the-art (SOTA) methods. (The ziyuan dataset and codes could be available at https://github.com/rs-lsl/DCDMF).