CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Patrick Huber, Armen Aghajanyan, Barlas Oğuz, Dmytro Okhonko, Scott Yih, Sonal Gupta, Xilun Chen
2022Findings of the Association for Computational Linguistics: NAACL 202213 citationsDOIOpen Access PDF
Abstract
We propose a novel open-domain questionanswering dataset based on the Common Crawl project. With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English data-points), we use our largescale, natural, diverse and high-quality corpus to in-domain pre-train popular language models for the task of question-answering. In our experiments, we find that our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks 1 .
Topics & Concepts
Question answeringComputer scienceOpen domainTask (project management)Language modelDomain (mathematical analysis)Artificial intelligenceNatural language processingScale (ratio)Training setInformation retrievalResource (disambiguation)Natural languageNatural language understandingMathematicsMathematical analysisComputer networkQuantum mechanicsPhysicsManagementEconomicsTopic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications