How Context Affects Language Models' Factual Predictions
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander Miller, Sebastian Riedel
Abstract
When pre-trained on large unsupervised textual corpora, language models are able to \nstore and retrieve factual knowledge to some extent, making it possible to use them directly for zero-shot cloze-style question answering. However, storing factual knowledge in a \nfixed number of weights of a language model clearly has limitations. Previous approaches \nhave successfully provided access to information outside the model weights using supervised architectures that combine an information retrieval system with a machine reading \ncomponent. In this paper, we go a step further and integrate information from a retrieval \nsystem with a pre-trained language model in a purely unsupervised way. We report that \naugmenting pre-trained language models in this way dramatically improves performance \nand that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline. Furthermore, processing query and context with different \nsegment tokens allows BERT to utilize its Next Sentence Prediction pre-trained classifier \nto determine whether the context is relevant or not, substantially improving BERT’s zeroshot cloze-style question-answering performance and making its predictions robust to noisy \ncontexts.