Lightweight Version Bidirectional Encoder Representation from Transformer Low-Resource Language Text Classification
K. Thiyagarajan, A C Ramachandra, Baker Karim, Amit Chaudhary, Nath A. Viswanath
Abstract
Training a Deep-Learning (DL) text classification model usually requires a large amount of labelled data, yet labelling data are usually labour-intensive and time-consuming. However, it becomes particularly challenging when working with low-resource languages, where labelled data are limited. To overcome this constraint, few-shot text classification allows models to generate precise predictions from a small amount of labelled data. In this research, paper aim to develop a meta-learning framework integrated with transformer architectures to enhance the performance in low-resource text classification settings. This architecture proposed A-Lightweight version bidirectional encoder representations from transformer (ALBERT), a lightweight version of bidirectional encoder representations from transformer (BERT) and introduced a self-supervised learning phase to help the model gain more useful representations before meta-training. The experimental results show strong improvements over standard baselines. This model achieved 88% accuracy on the CLINC150 dataset and 86% on the Ewe dataset, outperforming the other methods in both accuracy and F1 score.