Human vs. AI Authorship: Does it Matter in Evaluating Creative Writing? A Pilot Study Using ChatGPT
Miguel Landa–Blanco, Maitée Agüero-Flores, Miguel Ángel Mercado
Abstract
The current study analyzed whether people rated creative writing texts differently if they believed an Artificial Intelligence (AI) (ChatGPT) or a person was the author. A pilot experiment was designed. AI-generated texts were presented to a control and an experimental group. The stimuli included three poems and a short story. In all cases, the texts were created by ChatGPT; however, participants in the control group were told the texts were written by a person, while the experimental group was told the text was generated by ChatGPT. No statistically significant differences were found when comparing the control and experimental groups' scores regarding perceived creativity and originality of the texts, nor in how much enjoyment it caused and how likely participants were to recommend the texts to someone else. Therefore, preliminary evidence from this pilot study suggests that readers do not evaluate different, in terms of creative writing, a text attributed to a human authorship than one believed to have been written by an AI.