Anomaly Detection in Resource Constrained Environments With Streaming Data
Prarthi Jain, Seemandhar Jain, Osmar R. Zai͏̈ane, Abhishek Srivastava
Abstract
Isolation Forest (or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> ) is a well-known technique for anomaly detection. It is, however, a bulky approach that assumes the luxury of large storage space and is also ineffective with dynamic streaming data so common nowadays in varied application domains. In this work, we present the Preprocessed Isolation Forest ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> ) approach for anomaly detection that works well in resource constrained environments and is also effective on streaming data. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> is largely based on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> algorithm and to effectively handle the streaming data includes a pre-processing stage. In the pre-processing stage, Principal Component Analysis (PCA) is first harnessed to significantly reduce the dimension and bulk of the data. Subsequently, the streaming characteristic of the data is handled through a sliding window mechanism that creates sequential blocks of data for systematic processing. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> is able to identify anomalies as effectively as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> and other state-of-the-art anomaly detection techniques but has substantially low storage and prediction complexity. We conduct empirical evaluation of the proposed approach with standard data sets and show that it performs comparably with standard techniques in terms of Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and is able to work with high-dimensional, streaming data. Subsequently, we do a real-world hardware implementation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> and demonstrate that the approach is realistic and practicable in resource-constrained environments.