The GitHub Recent Bugs Dataset for Evaluating LLM-Based Debugging Applications
Jae Yong Lee, Sungmin Kang, Juyeon Yoon, Shin Yoo
Abstract
While Large Language Models (LLMs) have demon-strated strong natural language and code processing capabilities, concern has been raised as to whether existing bug benchmarks are included in their training data. We examine the training data of the open-source LLM StarCoder, and find it likely that data from the widely used Defects4J benchmark was included, raising the possibility of its inclusion in the training data of the GPT model as well. This makes it difficult to tell how well LLM-based results on Defects4J would generalize, as for any results it would be unclear whether a technique's performance is due to LLM generalization or memorization. To remedy this issue and facilitate continued research on LLM-based SE, we present the GitHub Recent Bugs (GHRB) framework, which continuously gathers real-world Java bugs for use in evaluation of LLM-based techniques. To date, we have gathered 89 bugs reported after the GPT-3.5 training data cutoff point of September 2021.