LLM-BotGuard: A Novel Framework for Detecting LLM-Driven Bots With Mixture of Experts and Graph Neural Networks
Jinglong Duan, Weihua Li, Quan Bai, Minh Nguyen, Xiaodan Wang, Jianhua Jiang
Abstract
Detecting social media bots has become increasingly critical due to their detrimental impact on online environments. With the emergence of sophisticated large language models (LLM) such as ChatGPT, bot detection faces new challenges. These bots based on LLMs exhibit human-like behaviors, and it is difficult for traditional detection approaches to identify them effectively. Such conventional methods struggle with the advanced features associated with LLM-driven bots, which possess contextual understanding and mimic human interaction patterns. The significance of detecting LLM-driven bots lies in their increased difficulty of detection and their potential to inflict more covert harm compared with traditional bots. To address these challenges, we propose LLM-BotGuard, a novel detection model that is capable of capturing the unique features of LLM-driven bots alongside other bot characteristics through three key modules, i.e., pattern-informed feature extraction module, mixture of experts module, and graph module with graph sample and aggregation networks. Extensive experiments have been conducted to evaluate the performance of the proposed LLM-BotGuard. The results demonstrate that LLM-BotGuard significantly outperforms baseline methods in detecting LLM-driven bots. The proposed LLM-BotGuard offers a robust solution for identifying sophisticated LLM-driven bots in online social networks.