Logram: Efficient Log Parsing Using n-Gram Dictionaries
Hetong Dai, Heng Li, Che Shao Chen, Weiyi Shang, Tse-Hsun Chen
Abstract
Software systems usually record important runtime information in their logs. Logs help practitioners understand system runtime behaviors and diagnose field failures. As logs are usually very large in size, automated log analysis is needed to assist practitioners in their software operation and maintenance efforts. Typically, the first step of automated log analysis is log parsing, i.e., converting unstructured raw logs into structured data. However, log parsing is challenging, because logs are produced by static templates in the source code (i.e., logging statements) yet the templates are usually inaccessible when parsing logs. Prior work proposed automated log parsing approaches that have achieved high accuracy. However, as the volume of logs grows rapidly in the era of cloud computing, efficiency becomes a major concern in log parsing. In this work, we propose an automated log parsing approach, <i>Logram</i> , which leverages <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> -gram dictionaries to achieve efficient log parsing. We evaluated <i>Logram</i> on 16 public log datasets and compared <i>Logram</i> with five state-of-the-art log parsing approaches. We found that <i>Logram</i> achieves a higher parsing accuracy than the best existing approaches (i.e., at least 10 percent higher, on average) and also outperforms these approaches in efficiency (i.e., 1.8 to 5.1 times faster than the second-fastest approaches in terms of end-to-end parsing time). Furthermore, we deployed <i>Logram</i> on <i>Spark</i> and we found that <i>Logram</i> scales out efficiently with the number of <i>Spark</i> nodes (e.g., with near-linear scalability for some logs) without sacrificing parsing accuracy. In addition, we demonstrated that <i>Logram</i> can support effective online parsing of logs, achieving similar parsing results and efficiency to the offline mode.