Adaptive Modality Interaction Transformer for Multimodal Knowledge Graph Completion
Yue Jian, Miao Zhang, Ziyue Qin, Chuyuan Xie, Kui Xiao, Yan Zhang, Zhifei Li
Abstract
Knowledge graphs (KGs) are frequently confronted with the challenge of incompleteness, a problem that extends to multimodal knowledge graphs (MKGs). The primary goal of multimodal knowledge graph completion (MKGC) is to predict missing entities within MKGs. However, current MKGC methods face difficulties in adequately addressing modal preferences and imbalances in modal information. To overcome these issues, we introduce AdaMKGC, an innovative hybrid model incorporating an adaptive modality interaction transformer. This model employs a dynamic attention interaction strategy and a self-enhancing sampling approach. AdaMKGC achieves a more precise utilization of multimodal information by integrating modal preference information into modal interactions. Additionally, it effectively mitigates the issue of modal imbalance through targeted sampling and adjustment for entities with deficient information. Experimental evaluations demonstrate AdaMKGC’s superior performance in overcoming these prevalent challenges. Compared to existing state-of-the-art MKGC models, AdaMKGC shows a notable enhancement of 28% in MR on the WN18-IMG dataset and an improvement of 2.7% in Hits@1 on the FB15k-237-IMG dataset. Our code is available at https://github.com/HubuKG/AdaMKGC .