Big data and Spark: Comparison with Hadoop

Yassine Benlachmi, Moulay Lahcen Hasnaoui

20202020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4)16 citationsDOI

Abstract

Now a days, data is generating by different industries on a daily basis which amount growing from terabyte to petabyte. To handle this large amount of data, big data technology is great revolution and has impacted the trends of the applied science. For the analysis of the large amount of data, Hadoop MapReduce technology is used in parallel multiple nodes. The numerous amount of data and information is stored through HDFS, Map and Reduce are two main functions of the MapReduce. There are many loop holes in MapReduce so for fast queries and real time data stream, Spark is introduced. The base of the Spark is DAG and RDD approaches. The main purpose of this research is to make comparison between the basic features of Hadoop and Spark on which basis their performance will be evaluate.

Topics & Concepts

PetabyteTerabyteSPARK (programming language)Big dataComputer scienceDatabaseBase (topology)Operating systemData miningProgramming languageMathematicsMathematical analysisPrivacy-Preserving Technologies in DataIoT and Edge/Fog ComputingCloud Computing and Resource Management