An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
Wuwei Lan, Chen Yang, Wei Xu, Alan Ritter
Abstract
Multilingual pre-trained Transformers, such as mBERT However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study Giga-BERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings.
Topics & Concepts
Computer scienceArabicTransformerNatural language processingRelationship extractionArtificial intelligenceNamed-entity recognitionTransfer of learningInformation extractionTransfer (computing)Zero (linguistics)Task (project management)LinguisticsEngineeringVoltageElectrical engineeringSystems engineeringParallel computingPhilosophyTopic ModelingNatural Language Processing TechniquesText Readability and Simplification