Employing Real-time Stream Process Utilising IoT Data Analytics
Shireesha Gorgilli, Vamsi Krishna Koganti, Ahmad Jamal, Arpit Jain, Alok Sati, Khemraj Sharma
Abstract
The enormous growth in Internet of Things (IoT) devices contributed to an immense rise in real-time data production. This has made it important to have fast stream processing systems for handling large amounts of time-sensitive data. Due to their inherent delay and restricted ability to scale, traditional batch processing systems are not effective in these types of scenarios. Employing technologies like Apache Kafka to handle data ingestion, Apache Flink to process distributed stream, InfluxDB to store time-series, along with Grafana for real-time visualization, this study suggests the modular, minimal latency, high-throughput real-time stream processing design developed for IoT applications. For adaptive analytics and anomaly detection, the system combines cloud-based deep learning (LSTM) with lightweight edge-based Machine Learning (ML) models.The simulated IoT ecosystem was created using high-frequency sensor data produced by both actual and resembled devices to assess the architecture. Experimental findings reveal that the system greatly outperforms conventional batch systems by regularly achieving sub-second latency (0.6s–0.8s), scalable throughput (up to 10,000 events/second), along enhanced event identification accuracy (from 85% to 93%). The results illustrate that the suggested design may provide reliable, scalable insights for smart city, automated manufacturing and environmental monitoring applications. For intelligent IoT infrastructures to be ready for the future, the study shows that real-time processing models are necessary and feasible for implementation.