Detecting Code Smells using ChatGPT: Initial Insights
Luciana Lourdes Silva, Janio Rosa da Silva, João Eduardo Montandon, Marcus V. A. Andrade, Marco Túlio Valente
Abstract
This paper presents initial insights into the effectiveness of ChatGPT in detecting code smells in Java projects. We utilize a large dataset comprising four code smells—Blob, Data Class, Feature Envy, and Long Method—classified into three severity levels. To assess ChatGPT’s proficiency, we employ two different prompts: (i) a generic prompt and (ii) a prompt specifying the smells selected for our research. We evaluate ChatGPT’s abilities using metrics such as precision, recall, and F-measure. Our results reveal that the odds of ChatGPT providing a correct outcome with a specific prompt are 2.54 times higher compared to a generic one. Furthermore, ChatGPT is more effective at detecting smells with critical severity (F-measure = 0.52) than those with minor severity (F-measure = 0.43). Finally, we discuss the implications of our findings and suggest future research directions for leveraging large language models to detect code smells.