DMDiff: A Dual-Branch Multimodal Conditional Guided Diffusion Model for Cloud Removal Through SAR-Optical Data Fusion

Abstract

Optical remote sensing images, as a significant data source for Earth observation, are often impacted by cloud cover, which severely limits their widespread application in Earth sciences. Synthetic aperture radar (SAR), with its all-weather, all-day observation capabilities, serves as a valuable auxiliary data source for cloud removal (CR) tasks. Despite substantial progress in deep learning (DL)-based CR methods utilizing SAR data in recent years, challenges remain in preserving fine texture details and maintaining image visual authenticity. To address these limitations, this study proposes a novel diffusion-based CR method called the Dual-branch Multimodal Conditional Guided Diffusion Model (DMDiff). Considering the intrinsic differences in data characteristics between SAR and optical images, we design a dual-branch feature extraction architecture to enable adaptive feature extraction based on the characteristics of the data. Then, a cross-attention mechanism is employed to achieve deep fusion of the multimodal feature extracted above, effectively guiding the progressive diffusion process to restore cloud-covered regions in optical images. Furthermore, we propose an image adaptive prediction (IAP) strategy within the diffusion model, specifically tailored to the characteristics of remote sensing data, which achieves a nearly 20 dB improvement in PSNR compared to the traditional noise prediction (NP) strategy. Extensive experiments on the airborne, WHU-OPT-SAR, and LuojiaSET-OSFCR datasets demonstrate that DMDiff outperforms SOTA methods in terms of both signal fidelity and visual perceptual quality. Specifically, on the LuojiaSET-OSFCR dataset, our method achieves a remarkable 17% reduction in the FID metric over the second-best method, while also yielding significant enhancements in quality assessment metrics such as PSNR and SSIM.

Topics & Concepts