Ecosystem of Large Language Models for Code

Abstract

The extensive availability of publicly accessible source code and the advances in language models, coupled with increasing computational resources, have led to a remarkable rise of large language models for code (LLM4Code). These models do not exist in isolation but rather depend on and interact with each other, forming a complex ecosystem that is worth studying. It motivates us to introduce a pioneering analysis of the LLM4Code ecosystem . Utilizing Hugging Face [ ]—the premier hub for transformer-based models—as our primary source, we manually curate a list of datasets and models focused on software engineering tasks. We first identify key datasets, models, and users in the ecosystem and quantify their contributions and importance. We then examine each model’s documentation to trace its base model and understand the process for deriving new models. We categorize LLM4Code model reuse into nine categories, with the top three being fine-tuning , architecture sharing , and quantization . Additionally, we examine documentation and licensing practices, revealing that LLM4Code documentation is less detailed than that of general AI repositories on GitHub. The license usage pattern is also different from other software repositories, and we further analyze potential license incompatibility issues. To analyze the rapidly growing LLM4Code, we explore the potential of using LLMs to assist in constructing and analyzing the ecosystem. Advanced LLMs from OpenAI identify LLM4Code with 98% accuracy, infer base models with 87% accuracy, and predict reuse types with 89% accuracy. We employ LLMs to expand the ecosystem and find that conclusions from the manually curated dataset align with those from the automatically created one. Based on our findings, we discuss the implications and suggestions to facilitate the healthy growth of LLM4Code.

Topics & Concepts