Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course
Cheng-Han Chiang, Wei‐Chih Chen, Chun-Yi Kuan, Chienchou Yang, Hung-yi Lee
Abstract
Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research.However, it is unclear whether these LLM-based evaluators can be applied in real-world classrooms to assess student assignments.This empirical report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students.Based on student responses, we find that LLM-based assignment evaluators are generally acceptable to students when students have free access to these LLM-based evaluators.However, students also noted that the LLM sometimes fails to adhere to the evaluation instructions.Additionally, we observe that students can easily manipulate the LLM-based evaluator to output specific strings, allowing them to achieve high scores without meeting the assignment rubric.Based on student feedback and our experience, we provide several recommendations for integrating LLM-based evaluators into future classrooms.Our observation also highlights potential directions for improving LLM-based evaluators, including their instruction-following ability and vulnerability to prompt hacking.