Large Language Models are Not Models of Natural Language: They are Corpus Models
Csaba Veres
Abstract
Natural Language Processing (NLP) has become one of the leading application areas in the current Artificial Intelligence boom. <i>Transfer learning</i> has enabled large <i>deep learning</i> neural networks trained on the <i>language modeling</i> task to vastly improve performance in almost all language tasks. Interestingly, when the models are trained with data that includes software code, they demonstrate remarkable abilities in generating functioning computer code from natural language specifications. We argue that this creates a conundrum for claims that neural models provide an alternative theory to <i>generative phrase structure grammars</i>in explaining how language works. Since the acceptable syntax of programming languages is determined by phrase structure grammars, successful neural models are apparently uninformative about the theoretical foundations of programming languages, and by extension, natural languages. We argue that the term <i>language model</i> is misleading because deep learning models are not theoretical models of language and propose the adoption of <i>corpus model</i> instead, which better reflects the genesis and contents of the model.