Enhancing Engineering Education Through LLM-Driven Adaptive Quiz Generation: A RAG-Based Approach

Abstract

This research-to-practice study aims to develop an Artificial Intelligence (AI) MCQ generation system for engineering students, with a focus on adaptive learning, educational technology, and innovative assessment tools, to enhance personalized learning. Engineering education faces significant academic performance challenges, with first-year retention rates in STEM fields ranging between 27% to 46%, largely due to poor academic achievements. Multiple Choice Questions (MCQs) identify misconceptions, reinforce knowledge retention, and offer efficient assessment methods for engineering education. This interactive method improves attention and memory retention, reinforces knowledge, and improves comprehension. In this context, the emergence of Large Language Models (LLMs) such as GPT-4 has marked a significant advancement. Our literature review method employed a systematic approach, analyzing peer-reviewed articles, conference papers, and authoritative reports to uncover the trends and challenges in AI-driven quiz generation. The notable gap identified in our literature review is the lack of LLM-based adaptive quiz generation methods specifically for engineering education. Our methodology involved sourcing relevant structured datasets, data pre-processing, embedding generation, vector database storage, hybrid-search retrieval, LLM query results feed, prompt engineering, and context-based response. In this research, we adopted Vectara as a vector database tool for its automatic data ingestion capabilities and seamless integration with generative AI applications. Prompt engineering involves a dual-prompt approach, where the Contextual Question Prompt formulates questions based on user topics and chat history, while the Answer Question Prompt manages MCQ responses with explanations, ensuring relevant and contextually accurate interactions. Evaluation includes topic relevancy, answer relevancy, and a contextual relevancy score. Preliminary results indicate promising results for the generation of accurate and contextually appropriate questions with minimal hallucinations. The quiz generation system was deployed using Streamlit cloud-based architecture to showcase the functionality. Looking forward, we aim to expand the dataset to include more diverse engineering disciplines and to refine the retrieval algorithms to better handle complex diagrams and mathematical expressions commonly found in engineering texts.

Topics & Concepts