Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation
Sarah Fakhoury, Aaditya Naik, Georgios K. Sakkas, Saikat Chakraborty, Madan Musuvathi, Shuvendu K. Lahiri
Abstract
We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.
Topics & Concepts
Computer scienceWorkflowCode (set theory)Set (abstract data type)Empirical researchTest (biology)Code generationNatural language generationNatural languageHuman–computer interactionProgramming languageSoftware engineeringArtificial intelligenceComputer securityDatabaseKey (lock)EpistemologyPaleontologyBiologyPhilosophySoftware Engineering ResearchSoftware Testing and Debugging TechniquesAdvanced Malware Detection Techniques