AI-assisted marking: Functionality and limitations of ChatGPT in written assessment evaluation
Joan Li, Nikhil Kumar Jangamreddy, Ryuto Hisamoto, Ruchita Bhansali, Amalie Dyda, Luke Zaphir, Mashhuda Glencross
Abstract
Generative artificial intelligence technologies, such as ChatGPT, bring an unprecedented change in education by leveraging the power of natural language processing and machine learning. Employing ChatGPT to assist with marking written assessment presents multiple advantages including scalability, improved consistency, eliminating biases associated with human subjectivity. This work aimed to evaluate the usefulness, reliability and accuracy of ChatGPT in marking written assessments of varied types and to identify its limitations and challenges. ChatGPT was instructed using a set of prompts to mark the assessment based on a rubric. ChatGPT was able to evaluate and assess both coding and reflective assessments and to distinguish between assignments of different quality, demonstrating high consistency and accuracy for higher quality assessments, comparable to a human marking. ChatGPT was also able to generate textual detailed justifications based on the rubric and assessment task description. There was a significant difference in the outcomes generated by different prompts. These preliminary findings suggest that utilising ChatGPT as a marking assistant can increase written assessment marking efficiency, reduce cost and potentially decrease the unfairness and bias by providing a moderating perspective. Implications for practice or policy: Assessment designers could reconsider the design, purpose and objectives of written assessments and leverage ChatGPT effectively for teaching and learning. Assessors might consider adapting the technology as a grading aid, to support a human-in-the-loop grading process, providing additional resources and time, moderating and refining individual feedback, to increase consistency and quality. Curriculum and programme leaders could develop guidelines around the ethical use of generative AI-assisted assessment practice, monitor and regulate the ongoing evaluation and refinement.