Leveraging Large Language Models for Scalable and Explainable Cybersecurity Log Analysis
Giulia Palma, Gaia Cecchi, Mario Caronna, Antonio Rizzo
Abstract
The increasing complexity and volume of cybersecurity logs demand advanced analytical techniques capable of accurate threat detection and explainability. This paper investigates the application of Large Language Models (LLMs), specifically qwen2.5:7b, gemma3:4b, llama3.2:3b, qwen3:8b and qwen2.5:32b to cybersecurity log classification, demonstrating their superior performance compared to traditional machine learning models such as XGBoost, Random Forest, and LightGBM. We present a comprehensive evaluation pipeline that integrates domain-specific prompt engineering, robust parsing of free-text LLM outputs, and uncertainty quantification to enable scalable, automated benchmarking. Our experiments on a vulnerability detection task show that the LLM achieves an F1-score of 0.928 ([0.913, 0.942] 95% CI), significantly outperforming XGBoost (0.555 [0.520, 0.590]) and LightGBM (0.432 [0.380, 0.484]). In addition to superior predictive performance, the LLM generates structured, domain-relevant explanations aligned with classical interpretability methods. These findings highlight the potential of LLMs as interpretable, adaptive tools for operational cybersecurity, making advanced threat detection feasible for SMEs and paving the way for their deployment in dynamic threat environments.