SFCFusion: Spatial–Frequency Collaborative Infrared and Visible Image Fusion
Hanrui Chen, Lei Deng, Zhixiang Chen, Chenhua Liu, Lianqing Zhu, Mingli Dong, Xitian Lu, Chentong Guo
Abstract
Infrared images can provide prominent targets based on the radiation difference, making them suitable for use in all day and night conditions. On the other hand, visible images can offer texture details with high spatial resolution. Infrared and visible image fusion is promising to achieve the best of both. Conventional frequency or spatial multi-scale transformation methods are good at preserving image details. Deep learning-based methods become more and more popular in image fusion because they can preserve high-level semantic features. To tackle the challenge in extracting and fusing cross-modality and cross-domain information, we propose a Spatial-Frequency Collaborative Fusion (SFCFusion) framework that effectively fuses spatial and frequency information in the feature space. In the frequency domain, source images are decomposed into base and detail layers with existing frequency decomposition methods. In the spatial domain, a kernel-based saliency generation module is designed to preserve spatial region-level structural information. A deep learning-based encoder is employed to extract features from the source images, decomposed images and saliency maps. In the shared feature space, we achieve cross-modality spatial-frequency collaborative fusion through our proposed adaptive fusion scheme. We have conducted experiments to compare our SFCFusion with both conventional and deep learning approaches on TNO, LLVIP and M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> FD datasets. The qualitative and quantitative evaluation results demonstrate the effectiveness of our SFCFusion. We have further demonstrated the superiority of our SFCFusion in the downstream detection task. Our code will be available at https://github.com/ChenHanrui430/SFCFusion.