Dense Adaptive Grouping Distillation Network for Multimodal Land Cover Classification With Privileged Modality
Xiao Li, Lin Lei, Caiguang Zhang, Gangyao Kuang
Abstract
Multimodal land cover classification (MLCC) is a fundamental problem in remote sensing interpretation, which can obtain excellent performance on account of the complementary information between the optical and SAR modalities. However, it is usually impossible to obtain multimodal data at the same time, due to the restriction of imaging conditions. When one of the modalities data is completely missing during test phase, classical multimodal learning methods might not be able to handle the MLCC task with privileged modality. In this paper, we propose an efficient Dense Adaptive Grouping Distillation Network (DAGDNet), which learns privileged information from available modalities in the train sets, and improves the classification performance in the test sets when one modality data is scarce. More specifically, to relieve the heterogeneous gaps between different modalities and then transfer the privileged information, we propose an Interactive Gated-based Feature Grouping Module (IG-FGM), which decomposes multimodal features into modalities-shared and modality-specific components to realize the decoupling of multimodal features and grouping distillation. Furthermore, the IG-FGM is inserted into different layers of the “teacher" network to implement progressive blending of multi-modalities. Then, to adaptively highlight the importance of hierarchical features distillation and grouping distillation, we propose a Multi-stage Adaptive Distillation Learning (MS-ADL) strategy so that the weights of different distillation losses are required to change continuously along with the training process. Finally, we evaluate the superior performances of our model on representative co-registered optical and SAR datasets.