Infrared and Visible Image Fusion via Text-Prior Guided Frequency-Domain Decomposition

Abstract

Existing fusion methods usually neglect the effective utilization of frequency-domain information and deeper textual semantic information, which degrades the quality and semantic richness of the fused image. To address these limitations, we propose a novel Text-prior guided Frequency-Domain Decomposition Fusion network ( \(\text{TFD}^{2}\text{Fusion}\) ). Specifically, \(\text{TFD}^{2}\text{Fusion}\) features a two-stage encoder designed to enhance the representational capacity of the fused image while enriching its semantic content: (i) In the feature decomposition stage, we design a three-branch feature decomposition and aggregation architecture, which utilizes the frequency-domain feature decomposition module to extract multi-grained low- and high-frequency features in the scene and detail branches, and introduces a frequency-domain fusion module in the aggregation branch to integrate shallow visual features extracted from different convolution blocks in the frequency-domain, thereby enhancing the representation of scene and detail information; (ii) In the prior guidance stage, we propose a text-prior guided semantic amplifier, which leverages a large vision-language model to interactively integrate text-level priors into visual features, thus enriching the semantic information of the fused image. Additionally, to promote the implementation and application of the proposed text-prior guided feature decomposition fusion paradigm, we release benchmark datasets that integrate text-level priors into three mainstream datasets. Extensive experiments demonstrate the superior fusion performance of our \(\text{TFD}^{2}\text{Fusion}\) . The datasets and code are available at https://github.com/wyshao-01/TFD2Fusion .

Topics & Concepts