HyperEye: A Lightweight Features Fusion Model for Unknown Encrypted Malware Traffic Detection
Xiaodong Zang, Zilong Zheng, Haosheng Zheng, Xuan Liu, Muhammad Khurram Khan, Weiwei Jiang
Abstract
Recently, effectively identifying encrypted malicious traffic without decryption in consumer applications relies heavily on high-quality labeled traffic datasets. However, this harms models for incorrect labeling and requires more efficient real-time identification of encrypted unknown ones. This paper proposes HyperEye, a real-time, unsupervised, encrypted malicious traffic detection system. It can detect unknown traffic patterns by analyzing the fused traffic features in-depth. Precisely, we extract protocol-agnostic numerical and protocol-specific text features and devise a cross-term fusion algorithm to obtain a comprehensive traffic behavior description. We designed a genetic algorithm-based DBSCAN (GA-DBSCAN) parameter optimization algorithm to enhance the quality and stability in identifying malicious traffic. Extensive experimental results with open-world and real-world datasets demonstrate that our work outperforms other state-of-the-art malware detection systems, achieving an average 11.95% improvement in the F1-score. Besides, experimental results with the real-world dataset demonstrate that our system applies to the dynamic nature of consumer applications and can safeguard users’ data and privacy.