The Promises and Pitfalls of LLMs as Feedback Providers: A Study of Prompt Engineering and the Quality of AI-Driven Feedback
Lucas Jasper Jacobsen, Kira Elena Weber
Abstract
Artificial intelligence (AI) in higher education (HE) is reshaping teaching and learning, and feedback provided by large language models (LLMs) seems to have an impact on student learning. However, few empirical studies have compared the quality of LLM feedback with the feedback quality of real persons. Therefore, this study addresses the following questions: What prompts are needed to ensure high-quality LLM feedback in HE? How does feedback from novices, experts, and LLMs differ in terms of quality and content accuracy? We developed a learning goal with three errors and a theory-based manual to evaluate prompt quality. Specifically, three prompts of varying quality were created and used to generate feedback from ChatGPT-4. We provided the highest-quality prompt to novices and experts. Our results showed that only the best prompt produced consistently high-quality feedback. Additionally, LLM and expert feedback were significantly better than novice feedback, with LLM feedback being both faster and better than expert feedback in the categories of explanation, questions, and specificity. This suggests that LLM feedback can be a high-quality and efficient alternative to expert feedback. However, we postulate that prompt quality is crucial, highlighting the need for prompting guidelines and human expertise.