IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization
Fajri Koto, Jey Han Lau, Timothy Baldwin
2021Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing81 citationsDOIOpen Access PDF
Abstract
We present INDOBERTWEET, the first largescale pretrained model for Indonesian Twitter that is trained by extending a monolinguallytrained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets. 1
Topics & Concepts
InitializationVocabularyComputer scienceIndonesianBenchmark (surveying)Focus (optics)Natural language processingArtificial intelligenceDomain adaptationLanguage modelEmbeddingLayer (electronics)Domain (mathematical analysis)Word (group theory)Adaptation (eye)Speech recognitionLinguisticsPsychologyMathematicsPhilosophyOrganic chemistryChemistryOpticsGeographyNeuroscienceMathematical analysisProgramming languageGeodesyClassifier (UML)PhysicsTopic ModelingNatural Language Processing TechniquesText Readability and Simplification