Nearest Neighbor Knowledge Distillation for Neural Machine Translation

Zhixian Yang, Renliang Sun, Xiaojun Wan

2022Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies12 citationsDOI Open Access PDF

Abstract

k-nearest-neighbor machine translation (kNN-MT), proposed by Although effective, kNN-MT requires conducting kNN searches through the large datastore for each decoding step during inference, prohibitively increasing the decoding cost and thus leading to the difficulty for the deployment in real-world applications. In this paper, we propose to move the time-consuming kNN search forward to the preprocessing phase, and then introduce k Nearest Neighbor Knowledge Distillation (kNN-KD) that trains the base NMT model to directly learn the knowledge of kNN. Distilling knowledge retrieved by kNN can encourage the NMT model to take more reasonable target tokens into consideration, thus addressing the overcorrection problem. Extensive experimental results show that, the proposed method achieves consistent improvement over the stateof-the-art baselines including kNN-MT, while maintaining the same training and decoding speed as the standard NMT model. 1

Topics & Concepts

Computer sciencek-nearest neighbors algorithmDecoding methodsMachine translationPreprocessorArtificial intelligenceTranslation (biology)InferenceDistillationMachine learningArtificial neural networkData miningPattern recognition (psychology)AlgorithmMessenger RNAGeneBiochemistryOrganic chemistryChemistryNatural Language Processing TechniquesMachine Learning and Data ClassificationMultimodal Machine Learning Applications