Self-learning model fusion for network anomaly detection: A hybrid CNN-LSTM-transformer framework
Jun Wang, Ning Huang, Houzhong Zhang, L. J. Liu, Qiang Fu, Kerang Cao, Xiwang Guo, Hoe-Kyung Jung
Abstract
The rapid evolution of cyber threats poses significant challenges to the adaptability and performance of anomaly detection systems. This study presents an innovative hybrid deep learning framework that integrates Convolutional Neural Networks (CNN), Long Short-Term Memory networks (LSTM), and Transformer models with a novel self-learning mechanism to enhance network traffic anomaly detection. Our key contributions include: (1) a synergistic two-stage model fusion architecture that captures both spatial and temporal traffic patterns; (2) an adaptive learning mechanism with multi-metric drift detection that autonomously responds to evolving threats; and (3) a knowledge preservation strategy that maintains detection capabilities while adapting to new attack patterns. The proposed CNN-LSTM model achieves F1-scores of 0.9778 and 0.9695 on the UNSW-NB15 and CICIDS2017 datasets respectively for binary classification of normal vs. anomalous traffic. The LSTM-Transformer model further classifies specific anomaly types with accuracies of 0.9632 and 0.9528 on these datasets, representing significant improvements over recent methods. Experiments demonstrate the framework's robustness, maintaining an average accuracy of 0.955 ([Formula: see text] 0.005) over a 15-day simulated period with multiple induced concept drifts. The self-learning mechanism, with multi-metric drift detection and an efficient model update strategy, enables the system to detect drifts and recover performance within 23.4 ± 0.20 hours post-drift, while achieving a 92.8% detection rate for zero-day attacks. The proposed framework offers a promising direction for developing efficient and autonomous cybersecurity systems capable of handling dynamic and evolving threat landscapes.