Fast Anomaly Detection for IoT Services Based on Multisource Log Fusion
Xingguo Jiang, Hong Luo, Yan Sun, Mohsen Guizani
Abstract
With the fast development of Internet of Things (IoT), anomaly detection has recently become a common concern for IoT smart applications. The circuit detection mode of all services to detect application anomaly is widely adopted. However, regularly collecting key performance indicators (KPIs) of all services under different clouds is a time-consuming task. Moreover, too many alerts in a short period of time will affect the processing speed of engineers. Recently, some researches of inferring the key service on the call paths can solve the above problems. However, it is still a challenge to locate dynamic and scattered key services accurately. In this article, we propose a service inference-based anomaly detection approach (SIADA), which integrates three sources of logs: 1) call; 2) business; and 3) metric. SIADA leverages the deep graph representation with context-aware multigraph fusion based on a recurrent encoder. This should infer key services and adopt variational autoencoder (VAE) with the flow model to detect multivariate time-series anomalies for key services. We have conducted extensive experiments on the public data set MicroSS. SIADA achieves the best average accuracy of 92% in service inference and the best average F1-score of 0.98 in anomaly detection, which has improved by 12.17% and 6.42% compared with the best SOTA baseline, respectively. Moreover, the total detection time, network transmission, and average alert number are reduced by 42.12%, 81.87%, and 83.5%, respectively.