A Self-Adaptive Intrusion Detection System for Zero-Day Attacks Using Deep Q-Networks
Mouhammd Alkasassbeh, Ebtehal H. Omoush, Mohammad Almseidin, Amjad Aldweesh
Abstract
Zero-day attacks remain among the most formidable threats to modern cybersecurity, exploiting undisclosed vulnerabilities that bypass conventional detection mechanisms. In this paper, we propose a self-learning intrusion detection system (IDS) based on Deep Q-Networks (DQNs), a reinforcement learning approach capable of autonomously classifying network traffic without dependence on predefined signatures or labeled training data. Leveraging the UGRansome dataset, the model is evaluated under two experimental scenarios: a conventional random split and a more rigorous zero-day split, where ransomware families in the test set are entirely excluded from training. In binary classification, the proposed system achieves 97.6% accuracy (F1-score: 0.98) in the random split and 95.9% accuracy (F1-score: 0.96) in the zero-day setup, consistently prioritizing high recall for ransomware detection. In multiclass classification, it attains 97.0% and 92.0% accuracy in the random and zero-day splits, respectively. These results underscore the potential of reinforcement learning as a robust and adaptive foundation for real-time intrusion detection in evolving threat landscapes.