LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Weiming Shen, Limao Xiong, Yuhao Zhou, Xiaohua Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Topics & Concepts
Plug-inForgettingComputer scienceStyle (visual arts)Natural language processingArtificial intelligenceProgramming languageCognitive psychologyPsychologyHistoryArchaeologyTopic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications