An Empirical Evaluation of the Zero-Shot, Few-Shot, and Traditional Fine-Tuning Based Pretrained Language Models for Sentiment Analysis in Software Engineering
Md Shafikuzzaman, Md Rakibul Islam, Alex C. Rolli, Sharmin Akhter, Naeem Seliya
Abstract
Recent advances in natural language processing (NLP) have led to the development of revolutionized pretrained language models (PLMs) impacting various NLP tasks, including sentiment analysis in software engineering. Choosing the right PLMs is crucial to effectively leverage these advanced PLMs. This paper presents the largest comparative evaluation of the PLMs for sentiment analysis in software engineering. Specifically, the study initially quantifies the performances of four traditionally fine-tuned PLMs, five zero-shot PLMs including <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GPT-4</monospace> and <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GPT-3</monospace> models, and three few-shot PLMs on six domain-specific datasets. The performances of the selected PLMs are also compared against two software engineering domain-specific traditionally fine-tuned PLMs and two state-of-the-art tools. The quantitative analysis reveals varying strengths across the different PLM types. The traditionally fine-tuned domain-specific PLM <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">seBERT</monospace> achieves the best results in the larger datasets, whereas the few-shot PLMs, such as <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">All-DistillRoBERTa</monospace> , show the best performances in the smaller datasets. A qualitative error analysis with the help of an Explainable AI technique uncovers existing challenges faced by PLMs in sentiment analysis for software engineering. The comprehensive quantitative and qualitative experiments significantly enrich knowledge in sentiment analysis in software engineering through reproducible insights.