Exploring the use of retrieval-augmented generation models in higher education: A pilot study on artificial intelligence-based tutoring
Renáta Németh, Annamária Tátrai, Miklós Szabó, Péter Tibor Zaletnyik, Árpád Tamási
Abstract
The role of artificial intelligence in education is growing, but its impact on teaching and learning remains unclear. This paper examines an artificial intelligence tutor like ChatGPT, enhanced with retrieval-augmented generation, in a pilot project at two universities. The advantage of retrieval-augmented generation in higher education is that lecturers can take control of the underlying literature used by the large language model. Four courses from three programs with different learning objectives and pedagogical methods were involved to provide as diverse a context as possible. We created an artificial intelligence tutor for each course, capable of answering students' questions by referring to the educational resources provided. Both students and lecturers reported positive experiences. Qualitative analysis of the chats shows a high level of engagement and motivation. Based on expert coding of a random sample, 1.5 % of answers were incorrect and 16.5 % were outside the context provided to the large language model. We found that retrieval-augmented generation reduces hallucinations for topics that are sufficiently explained in the material. Both the online survey results and the temporal pattern of the queries showed various learning strategies. Students used the tutor effectively, intuitively finding the most appropriate use for themselves. A comparison of different courses shows that the same tool can work differently in different contexts. Our results show that the artificial intelligence tutor does not simply impart knowledge, but mediates the relationship between student, lecturer, and course material. This mediation can facilitate learning, but it can also pose new challenges.