A Review of Automatic Item Generation Techniques Leveraging Large Language Models

Abstract

Over a decade ago, automatic item generation (AIG) was introduced to meet the increasing need for high-quality items in educational measurement. Around the same time, a new area of research in computer science began to develop questions for educational use. Historically, researchers from these two domains had little knowledge or communication with one another. However, the development of pre-trained large language models (LLMs) has sparked the interest of researchers from both domains in applying these models for automatically creating items. With similar objectives and methodologies, these two research domains appear to be converging on how to address the problems in this field. The purpose of this study is to provide a review of the current state of research by synthesizing existing studies on the use of LLMs for AIG. By combining research from both domains, we examine the utility and potential of LLMs for AIG. We performed a comprehensive literature review in seven research databases, selected studies based on predefined criteria, and summarized 60 relevant studies that employed LLMs in the AIG process. We identified the most commonly used LLMs in current AIG literature, their specific applications in the AIG process, and the characteristics of the generated items. We found that LLMs are flexible and effective in generating various types of items based on different languages and subject domains. However, many studies have overlooked the quality of the generated items, indicating a lack of a solid educational foundation. This review emphasizes the urgent need for greater integration of learning and measurement theories in future AIG research. We share two suggestions to enhance the educational foundation for leveraging LLMs in AIG, advocating for interdisciplinary collaborations to exploit the utility and potential of LLMs.

Topics & Concepts