Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science
Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sang-Hoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A. Persson, Gerbrand Ceder, Anubhav Jain
Abstract
-based models by 1%∼12%, implying that domain-specific pre-training provides measurable advantages. Despite relative architectural simplicity, the BiLSTM model consistently outperforms BERT, perhaps due to its domain-specific pre-trained word embeddings. Furthermore, MatBERT and SciBERT models outperform the original BERT model to a greater extent in the small data limit. MatBERT's higher-quality predictions should accelerate the extraction of structured data from materials science literature.
Topics & Concepts
Named-entity recognitionComputer scienceBottleneckTransformerDomain (mathematical analysis)Training setArtificial intelligenceNatural language processingMachine learningTask (project management)Mathematical analysisEconomicsPhysicsManagementEmbedded systemQuantum mechanicsVoltageMathematicsTopic ModelingMachine Learning in Materials ScienceNatural Language Processing Techniques