Leveraging Transformer Models in the Cyberbullying Text Classification System for the Low-resource Bengali Language
Md. Nesarul Hoque, Md. Hanif Seddiqui
Abstract
Cyberbullying is when a perpetrator attacks a person or a group of people using insulting or dehumanizing messages sent over digital networks. The availability of the internet and unrestricted usage of social networking sites are the reasons for the sharp rise in cyberbullying occurrences every day. It degrades the victim mentally and socially. Therefore, classifying cyberbullying text is currently an important research topic for scholars. A limited number of articles are recognized for identifying cyberbullying material for the low-resource Bengali language, in contrast to English and other high-resource languages, due to resource limitations. In our research, we have attempted to develop an effective Bengali cyberbullying text classification system. At first, we investigate a benchmark dataset of 44001 entries for five cyberbullying classes: Not Bully, Troll, Sexual, Religious, and Threat. After that, we exploit five cutting-edge transformer models, including multilingual Bidirectional Encoder Representations from Transformers (mBERT), Bangla-BERT-Base, BanglaBERT, DistilmBERT, and XLM-RoBERTa. Finally, we have proposed a new combined-transformer model, Transformer-ensemble, obtaining outstanding results with 87.54% accuracy and 87.52% F1-score. This method outperforms the state-of-the-art approaches for identifying five Bengali cyberbullying classes. Thus, our proposed system can play a vital role in reducing cyberbullying occurrences on digital platforms.