Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, Gustavo Soares
Abstract
Generative AI and large language models hold great promise in enhancing computing education by powering next-generation educational technologies. State-of-the-art models like OpenAI’s ChatGPT [8] and GPT-4 [9] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, a digital assistant for an educator, and a digital peer for collaborative learning [1, 2, 7]. In our work, we seek to comprehensively evaluate and benchmark state-of-the-art large language models for various scenarios in programming education.
Topics & Concepts
Computer scienceBenchmarkingGenerative grammarArtificial intelligenceNatural language processingBusinessMarketingTeaching and Learning ProgrammingParallel Computing and Optimization TechniquesScientific Computing and Data Management