From red ink to algorithms: investigating the use of large language models in academic writing feedback
Marina Jovic, Stavros Papakonstantinidis, Robert Kirkpatrick
Abstract
Abstract This paper explores the emerging role of large language models (LLMs) in academic feedback for student writing. It examines feedback provided by two human graders and two LLMs, ChatGPT-4.5 and Claude-3.7 Sonnet, on argumentative and reflective essays written by students from diverse linguistic backgrounds. Our investigation addresses two research questions: how human and AI evaluators differ in assessing distinct academic essay genres and how conversational prompting and genre influence the equity and actionability of AI-generated feedback. A six-dimension evaluation rubric reveals that human graders excel at understanding context and offering holistic guidance, while AI-generated feedback demonstrates superior structure and efficiency, particularly in identifying mechanical and organizational issues. AI systems using conversational prompting significantly outperformed both human graders and zero-shot AI, with notable gains in future orientation and conciseness for argumentative essays. Both human and AI evaluators generally produced higher-quality feedback for reflective essays, with 1.3 + point advantages. While AI systems applied more consistent evaluation criteria across different student groups, they showed limitations in deeper evaluation of argument structure and personalized revision strategies. The paper demonstrates how educators can enhance AI-generated feedback through tailored conversational prompting strategies, suggesting that a blended approach combining AI’s structural consistency with human expertise in higher-order analysis offers the most effective educational outcomes.