AI Writes, We Analyze: The ChatGPT Python Code Saga
Md Fazle Rabbi, Arifa Islam Champa, Minhaz F. Zibran, Md Rakibul Islam
Abstract
In this study, we quantitatively analyze 1,756 AI-written Python code snippets in the DevGPT dataset and evaluate them for quality and security issues. We systematically distinguish the code snippets as either generated by ChatGPT from scratch (ChatGPT-generated) or modified user-provided code (ChatGPT-modified). The results reveal that ChatGPT-modified code more frequently displays quality issues compared to ChatGPT-generated code. The findings provide insights into the inherent limitations of AI-written code and emphasize the need for scrutiny before integrating such pieces of code into software systems.
Topics & Concepts
Python (programming language)Computer scienceProgramming languageSoftware Engineering ResearchSoftware System Performance and ReliabilityMachine Learning and Data Classification