Using generative artificial intelligence as an automated essay scoring tool: a comparative study
Ngoc My Bui, Jessie S. Barrot
Abstract
This study examines the reliability and consistency of two popular GenAIs (i.e. ChatGPT-4 and Gemini) for automated scoring of essays in comparison with experienced human raters. To address this objective, we subjected the 120 essays to two rounds of scoring by ChatGPT-4, Gemini, and human raters. Our study reveals significant limitations in their consistency and reliability compared to human raters. Specifically, we found that GenAIs are stricter in scoring essays, particularly ChatGPT-4. Given this scoring variability, the correlations between these raters ranged from negligible to moderate. In terms of scoring consistency, only the human raters demonstrated high intraclass consistency. These results were attributed to several factors, such as GenAI’s sensitivity to rubrics and writing criteria, limited training data, lack of contextual understanding, and model updates. Implications for second language (L2) writing teaching, assessment, tool enhancement, and future studies are discussed.