Probing for Referential Information in Language Models

Abstract

Language models keep track of complex linguistic information about the preceding context -including, e.g., syntactic relations in a sentence. We investigate whether they also capture information beneficial for resolving pronominal anaphora in English. We analyze two state of the art models with LSTM and Transformer architectures, respectively, using probe tasks on a coreference annotated corpus.

Topics & Concepts