Survey of Pre-trained Models for Natural Language Processing
Jiajia Peng, Kaixu Han
Abstract
Deep learning has developed rapidly. The pre-training technology of natural language processing has also made great progress. Early natural language processing used static pre-training techniques, such as Word2Vec, GLoVe, and other word vector methods to segment text. However, this segmentation method does not consider, and the context cannot solve the problem of polysemy. The BERT model has made many typical downstream tasks a typical improvement, which has greatly promoted technological development in the field of NLP and has since entered the epoch of dynamic pre-training technology. On this basis, we analyze the shortcomings of current pre-training technology and finally looks forward to the future development trend of pre-training technology.