Impact of Transformer-Based Models in NLP: An In-Depth Study on BERT and GPT
Mustafa Salıcı, Üyesi Ercan Ölçer
Abstract
This article examines in depth the effects of BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pretrained Transformer) models in the field of natural language processing (NLP). NLP, as a sub-branch of computer science and artificial intelligence, has an important place in the processes of understanding, processing and producing human language. Transformer-based models have provided revolutionary advances in many NLP tasks, especially language modeling, sentiment analysis, machine translation and question answering. In this article, the technical structures of BERT and GPT models, their performance in low-resource languages such as Turkish and the difficulties encountered in these languages are discussed in detail. The BERT model stands out with its ability to evaluate the contextual meaning of the language from both directions using bidirectional context and offers high success rates especially in tasks such as text classification, named entity recognition (NER) and sentiment analysis. The GPT model, on the other hand, provides superiority in text production with its autoregressive structure and gives successful results in tasks such as creative writing and chatbots. However, the application of these models to morphologically rich and agglutinative languages such as Turkish poses certain challenges. These challenges arise due to the structural features of the language and the limitations of the datasets. The article also discusses the development of Turkish-specific models such as BERTurk, data augmentation techniques, and the success of modifications appropriate to the characteristics of the language in overcoming these challenges. In conclusion, the development of BERT and GPT models in the field of NLP has been a major step in terms of natural language processing and modeling. However, further research and development is required for these models to perform better in languages such as Turkish. The article predicts that future developments of these models will enable language technologies to reach wider audiences and be used in various applications.