Evaluating large transformer models for anomaly detection of resource-constrained IoT devices for intrusion detection system
Ahmad Almadhor, Shtwai Alsubai, Natalia Kryvinska, Abdullah Al Hejaili, Mohamed Arselene Ayari, Belgacem Bouallègue, Sidra Abbas
Abstract
The rapid growth of the Internet of Things (IoT) has revolutionised industries but also introduced critical security threats, making robust Intrusion Detection Systems (IDS) essential. Traditional signature-based IDS struggles with evolving threats, while AI-driven approaches, such as machine learning (ML) and deep learning (DL), show promise but face challenges in terms of scalability and adaptability. Large Transformer Models (LTMs) offer a novel solution by enhancing anomaly detection, automating threat analysis, and improving real-time IoT security through advanced contextual understanding. In this research, we propose an LTM-based IDS for real-time detection of IoT attacks. Integrating LTMs into IoT security can improve intelligence, automation, and threat mitigation. We propose transformer-based deep learning models such as Fine-Tuned Bidirectional Encoder Representations from Transformers Model (BERT), Distilled Bidirectional Encoder Representations from Transformers (DistilBERT), and Robustly Optimised BERT Pretraining (RoBERTa). Attack categories in the RT_IoT2022 dataset were encoded into numerical labels, followed by comprehensive data preprocessing, including random sampling and handling of missing values. To improve interpretability, the data was transformed into text format to ensure compatibility with BERT-based models. Subsequently, the dataset was split and converted into the Hugging Face Dataset format, allowing for seamless integration with Natural Language Processing (NLP) models for IoT attack detection. Then, we apply fine-tuning multiple transformer architectures, including BERT, DistilBERT, and RoBERTa, for IoT attack classification, optimising hyperparameters for efficient learning. The BERT model demonstrated strong performance, achieving its lowest training loss of 0.0211 at [Formula: see text] epoch and the lowest validation loss of 0.0677 at [Formula: see text] epoch. These results indicate effective learning and good generalisation capability.