A Proposed New Endpoint Detection and Response With Image-Based Malware Detection System
Tran Hoang Hai, Vu Van Thieu, Tran Thai Duong, Hong Hoa Nguyen, Eui‐Nam Huh
Abstract
Due to increased reliance on technology and cloud-based services, cyber risks are more common. Advanced persistent threats make it difficult to detect attacks, hence Endpoint Detection and Response (EDR) was adopted in 2013. EDR uses a scanning application on each endpoint machine to monitor and capture events and logs. However, EDR is vulnerable to attacks by malware, so a lightweight malware detector is needed. Image-based malware classification is a technique for classifying malware based on its representative image, but previous studies have not been integrated with EDR. This research aims to integrate EDR with an image-based malware classifier. A basic EDR implementation named Deep Ocean Protection System (DOPS) has been developed with two pre-trained models (Mobilenet V2 and Inception V3) fine-tuned with MalImg and BODMAS datasets. The models were evaluated with the DikeDataset and Mobilenet V2 fine-tuned with BODMAS 4.0.0 performed best in terms of loading and prediction time with a high AUC score of 0.8615. Inception V3 fine-tuned with BODMAS 4.0.0 also achieved a remarkable AUC score of 0.9392. These results show the potential of integrating image-based malware detection with EDR.