Extending Kafka Streams for Complex Event Recognition
Samuele Langhi, Riccardo Tommasini, Emanuele Della Valle
Abstract
Streaming Analytics (SA) and Complex Event Recognition (CER) are of paramount importance in searching for an ultimate Big Data solution that can simultaneously address Data Velocity, Variety, and Volume. Indeed, the growing popularity of streaming data has pushed the boundaries of existing data systems, fostering the rise of Stream Processing Engines (SPE). However, Data Velocity never appears isolated. Streams are huge, heterogeneous, and noisy as they come from multiple sources. Horizontally-scalable SPEs like Flink and KSQL-DB allow continuous stream analytics using SQL-like languages. On the other hand, CER engines like OracleCEP and DroolFusion use regular languages for (parallel) pattern detection over heterogeneous streams. This paper takes a first step towards a unifying solution. To this extent, we present KELPr, an in-memory distributed CER engine designed extending the Dual Streaming Model and implemented on top of Kafka Streams.