Automated Log Classification Using Deep Learning
Shekar Ramachandran, Rupali Agrahari, Priyanka Mudgal, Harshita Bhilwaria, Garth Long, Arisha Kumar
Abstract
System logs are a rich source of information. The complexity and amount of data contained in log files is increasing rapidly, especially with the development of cloud computing in clustered environments. Log analysis is effective to find the root cause of an issue. Because of the huge volume of unstructured logs, manual inspection becomes difficult and time consuming. Hence, there is a need to automate the process of log analysis. Current state-of-the-art deep learning-based techniques are trained on generic datasets and do not perform well on specific system logs. The proposed invention is focused to predict errors from large system specific logs by using manual vectorization technique called “LogWord2Vec” and combining it with data-cleaning, augmentation, embedding, and deep learning method together. Unlike other solutions, our method does not require a large dataset, and works well on medium to small sized datasets. The proposals described in this disclosure help in improving both the accuracy and the efficiency of log parsing systems. We have used combination of Convolution neural network (CNN) layer and long short-term memory (LSTM) layer to build the model and obtained accuracy of 99.19% with 0.81% of misclassification.