Evaluation of Generative AI Models in Python Code Generation: A Comparative Study
Dominik Palla, Antonín Slabý
Abstract
This study evaluates generative AI models for Python code generation, focusing on their syntax accuracy, response time, completeness, and reliability. The models tested include OpenAI’s GPT series (GPT-4 Turbo, GPT-4o, GPT-4o Mini, GPT-3.5 Turbo), Google’s Gemini (1.0 Pro, 1.5 Flash, 1.5 Pro), Meta’s LLaMA (3.0 8B, 3.1 8B), and Anthropic’s Claude models (3.5 Sonnet, 3 Opus, 3 Sonnet, 3 Haiku). Ten coding tasks of varying complexity were designed and tested across three iterations per model to measure performance and consistency. The Claude models achieved the highest accuracy and reliability scores, demonstrating their suitability for precision-demanding tasks. In contrast, the Gemini models revealed limitations in handling complex tasks effectively. Cost-effective models like Claude 3 Haiku and Gemini 1.5 Flash proved budget-friendly, maintaining adequate accuracy for simpler tasks. Overall, the study highlights key strengths and trade-offs of each model, providing practical guidance for developers selecting generative AI for automated coding. Unlike previous studies that primarily focus on single-metric benchmarking, this study introduces a multi-dimensional evaluation framework considering response accuracy, reliability, cost efficiency, and exception handling. Future research could broaden the scope to other programming languages and advanced metrics such as code optimization and security.