ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou
Abstract
This paper presents a new sequence-tosequence pre-training model called Prophet-Net, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
Topics & Concepts
Computer scienceOverfittingAutomatic summarizationSequence (biology)Artificial intelligencen-gramContext (archaeology)Machine learningGramScale (ratio)Language modelArtificial neural networkQuantum mechanicsPaleontologyBiologyBacteriaPhysicsGeneticsTopic ModelingNatural Language Processing TechniquesText and Document Classification Technologies