Novel Distributed Architecture for Frequent Pattern Mining using Spark Framework
Keerthi Samudrala, Jaswanth Kolisetty, Abhiram Shri Chakravadhanula, Bharat Preetham, Rajiv Senapati
Abstract
The Apriori algorithm is one of the association rule mining algorithms which is commonly used in retail data analysis to find patterns in itemsets. However, as the amount of data generated by retailers continues to increase, traditional approaches may no longer be sufficient. To address this challenge, In this paper, we have proposed a distributed architecture implementation of the Apriori algorithm in the Hadoop ecosystem using Spark. The approach involves three phases: categorising and partitioning customer data based on seasons, clustering customers based on behaviour and preferences, and applying the Apriori algorithm to obtain frequent itemset patterns and association rules. By implementing this approach in a distributed architecture, we are able to efficiently analyse large datasets and make accurate predictions about customer needs and preferences, which has important implications for retail strategy and we anticipate that this approach will be particularly useful in the context of retail data analysis where large amounts of data must be processed quickly and accurately.