Multimodal Feature Disentangle-Fusion Network for Hyperspectral and LiDAR Data Classification
Yukai Pan, Nan Wu, Wei Jin
Abstract
Multimodal remote sensing (RS) data (e.g., hyperspectral and light detection and ranging (LiDAR) data) provide complementary information and have, therefore, been widely explored for land-cover classification tasks. However, due to the differences in the potential feature spaces of heterogeneous data, effectively extracting and fusing similar information between modalities and modality-specific discriminative information remains challenging. In this letter, we propose the multimodal feature disentangle-fusion network (DFNet), which introduces a disentanglement framework to tackle the challenges in feature extraction and fusion. First, DFNet uses a dual-branch CNN-Transformer disentanglement module to decompose each modality’s features into the intermodality-shared representation and intramodality-specific representation. Second, we propose a joint training strategy based on contrastive learning and knowledge distillation to constrain the solution space and enhance disentangled feature representations. In addition, we employ a gate fusion unit that uses learnable weights to measure the contributions of different modality representations. A series of experiments and ablation studies on the Houston2013 dataset demonstrate that DFNet can effectively utilize the complementary information in multimodal data.