Towards an AI tutor for undergraduate geotechnical engineering: a comparative study of evaluating the efficiency of large language model application programming interfaces
Amir Tophel, Liuxin Chen, Umidu Hettiyadura, Jayantha Kodikara
Abstract
Abstract This study investigates the efficiency of large language model (LLM) application programming interfaces (APIs)—specifically GPT-4 and Llama-3—as AI tutors for undergraduate Geotechnical Engineering education. As educational needs in specialised fields like Geotechnical Engineering become increasingly complex, innovative teaching tools that provide personalised learning experiences are essential. Unlike previous studies on AI-driven education, our research uniquely focuses on assessing the role of retrieval-augmented generation (RAG) in improving the accuracy of LLM-generated solutions to Geotechnical problems. A dataset of 391 questions from the related textbook written by Das and Sobhan (Das B, Sobhan K. Principles of Geotechnical engineering, Eight Edition. In: Cengage Learning. 2014) was used for evaluation, with solutions sourced from the textbook’s manual. Performance benchmarking focused on 20 challenging questions previously identified by Chen et al. (Chen et al. in Geotechnics 4:470–498, 2024) as problematic for GPT-4 in Zero Shot tasks. GPT-4 with API support demonstrated superior accuracy, achieving accuracy rates of 95% at a temperature setting of 0.1, 82.5% at 0.5, and 60% at 1. In comparison, Llama-3 achieved an accuracy of 25% in Zero Shot tasks and 45% with API support at a temperature setting of 0.1. The findings highlight GPT-4’s potential as an AI tutor for Geotechnical Engineering education while demonstrating the need for domain-specific optimisation and advanced formula integration techniques. This study contributes to the ongoing discourse on AI in education by providing empirical evidence supporting the deployment of LLMs as personalised, adaptive teaching aids in engineering disciplines. Future work should explore optimised formula integration strategies, expanded domain knowledge bases, and long-term student learning outcomes.