A study of artificial intelligence in writing assessment for secondary school students: a comparative analysis based on the GPT-4 and human raters
Nan Xiao, C. Z. Yuan, YuTing Pei, Xue Wang, YuLe Cai
Abstract
This study examines the feasibility of AI in secondary school writing assessment by comparing GPT-4’s scoring performance with human raters. Using a randomized block design, 60 middle school students’ essays were evaluated by three experienced teachers and GPT-4. Results showed: (1) high overall agreement between AI and human raters (ICC = 0.84, 95% CI [0.79, 0.89]), with strongest correlation in high-level essays (r = 0.92, p < 0.001); (2) AI demonstrated superior efficiency, averaging 0.8 seconds per essay with stable performance (CV = 1.2%); (3) AI scored lower than humans in assessing creativity (M = 76.5 vs. 82.3, p < 0.05), revealing limitations in evaluating unconventional expressions. The study proposes a tiered system: AI for standardized scoring and humans for creative judgment. These findings support the cautious integration of AI in educational assessment..