Efficient Large Scale NLP Feature Engineering with Apache Spark
Armin Esmaeilzadeh, Maryam Heidari, Reyhaneh Abdolazimi, Parisa Hajibabaee, Masoud Malekzadeh
Abstract
Feature engineering is a computationally time-consuming process in the end-to-end machine learning pipeline. Large amounts of text data are being generated on many heterogeneous sources and platforms on the internet. The compute resources needed to extract valuable features from these big datasets are increasing significantly. In this research, we evaluate the runtime of the RDD and the Spark-SQL APIs of the Apache Spark framework to extract text features from the corpus of english Wikipedia. As a result, we demonstrate the significant runtime performance of the SparkSQL compared to RDD API.
Topics & Concepts
SPARK (programming language)Computer sciencePipeline (software)Feature engineeringBig dataFeature (linguistics)Artificial intelligenceSQLProcess (computing)Scale (ratio)The InternetDatabaseNatural language processingData miningDeep learningProgramming languageWorld Wide WebPhysicsLinguisticsPhilosophyQuantum mechanicsWeb Data Mining and AnalysisTopic ModelingScientific Computing and Data Management