Using Cipherbot: An Exploratory Analysis of Student Interaction with an LLM-Based Educational Chatbot
Joni Salminen, Soon-Gyo Jung, Johanne Medina, Kholoud K. Aldous, Jinan Y. Azem, Waleed Akhtar, Bernard J. Jansen
Abstract
Cipherbot, an educational chatbot using large language models to answer student questions concerning learning materials uploaded by the educator, was pilot tested in a classroom setting. Forty-four students used Cipherbot for seven weeks, sending 8077 messages. The average number of messages sent per student was 184 (SD = 80), with an average length of 98 characters (SD = 80). The engagement followed a non-normal distribution, with few power users, implying that most students are still hesitant to adopt tools like Cipherbot. Cipherbot was able to answer 82.5% of the student questions, demonstrating a scalable ability to address students' learning queries, with some room for improvement.