Machine Learning on Public Intrusion Datasets: Academic Hype or Concrete Advances in NIDS?
Marta Catillo, Antonio Pecchia, Umberto Villano
Abstract
The number of papers on network intrusion detection based on machine and deep learning is growing at an unprecedented rate. Most of these papers follow a well-consolidated pattern: (i) proposal of an intrusion detection system based on machine (deep) learning, (ii) learning-testing with one (more) public intrusion dataset(s), (iii) achievement of outstanding detection performance. Is the intrusion detection problem solved? Unfortunately, no. This paper shares a deep reflection on the major limitations of public intrusion datasets and related machine learning experiments, which greatly diminish the findings documented by the literature. At the end of the day, in spite of the academic hype and the increasingly-complex machine and deep learning exercises around, the role of public datasets in advancing intrusion detection of real-world networks remains questionable. The way existing intrusion datasets are collected, released and used by the community should be approached with extreme caution. This paper provides concrete hints for the construction of future intrusion detection datasets and more rigorous machine learning experiments.