Zero-shot Prompting for Code Complexity Prediction Using GitHub Copilot
Mohammed Latif Siddiq, Abdus Samee, Sk Ruhul Azgor, Md. Asif Haider, Shehabul Islam Sawraz, Joanna C. S. Santos
Abstract
Code generation models are gaining popularity because they can produce correct code from a prompt, speeding up the software development process. GitHub Copilot is currently one of the most commonly used tools for code generation. This tool is based on GPT3, a Large Language Model (LLM), and can perform zero-shot prompting tasks i.e., tasks for which the model is not specifically trained. In this paper, we describe a preliminary study that investigates whether GitHub Copilot can predict the runtime complexity of a given program using zero- shot prompting. In our study, we found that GitHub Copilot can correctly predict the runtime complexity 45.44% times in the first suggestion and 56.38 % times considering all suggestions. We also compared Copilot to other machine learning, neural network, and transformer-based approaches for code complexity prediction. We observed that Copilot outperformed other approaches for predicting code with linear complexity <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{O}(n)$</tex> .