Better by You, better than Me? ChatGPT-3 as writing assistance in students’ essays.
Željana Bašić, Ana Banovac, Ivana Kružić, Ivan Jerković
Abstract
Aim: To compare students' essay writing performance with or without employing ChatGPT-3 as a writing assistant tool.Materials and methods: Eighteen students participated in the study (nine in control and nine in the experimental group that used ChatGPT-3). We scored essay elements with grades (A-D) and corresponding numerical values (4-1). We compared essay scores to students' GPTs, writing time, authenticity, and content similarity. Results: Average grade was C for both groups; for control (2.39 ± 0.71) and for experimental (2.00 ± 0.73). None of the predictors affected essay scores: group (P = 0.184), writing duration (P = 0.669), module (P = 0.388), and GPA (P = 0.532). The text unauthenticity was slightly higher in the experimental group (11.87% ±13.45 to 9.96% ± 9.81%), but the similarity among essays was generally low in the overall sample (the Jaccard similarity index ranging from 0 to 0.054). In the experimental group, AI classifier recognized more potential AI-generated texts. Conclusions: This study found no evidence that using GPT as a writing tool improves essay quality since the control group outperformed the experimental group in most parameters.