Benchmarking and Evaluating Large Language Models in Phishing Detection for Small and Midsize Enterprises: A Comprehensive Analysis
Jun Zhang, Peiqiao Wu, Jeffrey London, Dan Tenney
Abstract
The proliferation of Generative Artificial Intelligence (GenAI) has driven significant innovation but also introduced new security risks, particularly in social engineering attacks such as phishing. Despite the potential for misuse of GenAI in such attacks, research on its use for defense against these threats is limited. To address this issue, this study addresses the unique cybersecurity needs of small and midsize enterprises (SMEs) by utilizing high-quality email datasets, categorized into human and AI-generated phishing and legitimate emails, to evaluate the effectiveness of different GenAI models. The results demonstrated that the open-source Llama-3-8b-instruct model outperformed other alternatives, achieving the highest accuracy and F1-score, while offering a cost-effective and flexible solution for SMEs. This approach lies in its focus on base models with default parameters, emphasizing ease of implementation without the need for additional training or fine-tuning. Compared to existing methods, this approach simplifies adoption while maintaining robust detection capabilities. The proposed prompt template enables LLMs to provide explanations and suggestions, assisting users in making informed decisions. Practical recommendations for SMEs include deployment strategies and cost-effective management of human factors in cybersecurity. However, the study acknowledges limitations in its reliance on base models and emphasizes the need for further research on fine-tuning and parameter optimization. These findings suggest the feasibility of using LLMs for real-world cybersecurity applications against phishing attacks in the context of SMEs, although certain risks and challenges remain.