Do Neural Language Models Overcome Reporting Bias?
Vered Shwartz, Yejin Choi
Abstract
Mining commonsense knowledge from corpora suffers from reporting bias, over-representing the rare at the expense of the trivial We study to what extent pre-trained language models overcome this issue. We find that while their generalization capacity allows them to better estimate the plausibility of frequent but unspoken of actions, outcomes, and properties, they also tend to overestimate that of the very rare, amplifying the bias that already exists in their training corpus.
Topics & Concepts
GeneralizationComputer scienceArtificial intelligenceLanguage modelNatural language processingMachine learningArtificial neural networkDeep neural networksMathematicsMathematical analysisTopic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications