A Contrastive Learning Enhanced Adaptive Multimodal Fusion Network for Hyperspectral and LiDAR Data Classification

Abstract

In recent years, the fusion-based classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data has garnered increasing attention from researchers. However, significant image disparities exist between HSI and LiDAR data because of their distinct imaging mechanisms, which limits the fusion of HSI and LiDAR data. Therefore, establishing effective interaction (INTER) and fusion between HSI and LiDAR data is crucial. Moreover, the classification of multimodal data with limited labeled instances is another challenging task. To address these issues, we propose a contrastive learning enhanced adaptive multimodal fusion network (CAMFNet) for the joint classification of HSI and LiDAR data. CAMFNet introduces a novel semantic similarity contrastive (SSC) loss that fully utilizes a large amount of unlabeled data to learn potential complementary information between different modalities. In addition, a hierarchical fusion strategy is adopted to fuse multimodal features. At the shallow feature fusion stage, a cross-modal feature interaction fusion (CMFIF) module is proposed to guide the extraction of multimodal features and enhance the complementary information through cross-modal interaction. At the same time, adaptive feature fusion is also used to dynamically assign corresponding weights to different modal features, achieving finer fusion. At the deep feature fusion stage, a channel split and integration (CSI) block is constructed to further optimize the fused spectral-spatial and elevation features via a negative feedback mechanism. The effectiveness of CAMFNet is validated through experiments on three standard datasets, which demonstrates that CAMFNet outperforms state-of-the-art methods.

Topics & Concepts