Using Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories: Challenges, Opportunities, and Critiques
Murtaza Ali, P. V. Rao, Yifan Mai, Benjamin Xie
Abstract
BACKGROUND AND CONTEXT. The pace of advancement of large language models (LLMs) motivates the use of existing infrastructure to automate the evaluation of LLM performance on computing education tasks. Concept inventories are well suited for evaluation because of their careful design and prior validity evidence.
Topics & Concepts
BenchmarkingComputer scienceEngineering managementData scienceBusinessEngineeringMarketingOnline Learning and Analytics