A distributed real-time recommender system for big data streams
Heidy Hazem, Ahmed Awad, Ahmed H. Yousef
Abstract
Recommender Systems (RS) play a crucial role in our lives. As users become continuously connected to the internet, they are less tolerant of obsolete recommendations made by an RS. Online RS has to address three requirements: continuous training and recommendation, handling concept drifts, and the ability to scale. Streaming RS proposed in the literature address the first two requirements only. That is because they run the training process on a single machine. To tackle the third challenge, we propose a Splitting and Replication mechanism for distributed streaming RS. Our mechanism is inspired by the shared-nothing architecture that underpins contemporary big data processing systems. We have applied our mechanism to two well-known approaches for online RS, namely, matrix factorization and item-based collaborative filtering. We conducted experiments comparing the performance with the baseline (single machine). Evaluating different data sets, experiments show online recall improvement by 40% with more than 50% less memory consumption.