Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs

Abstract

Code analysis is a fundamental problem in Software Engineering (SE), playing a crucial role in tasks such as debugging, performance optimization, and security assessment. Human developers typically approach code analysis through syntax parsing, static semantics inference, and dynamic reasoning. Traditional code analysis tools, while effective, are often limited by language specificity, complex configurations, and lack of cross-language generalization. Recently, large language models (LLMs) have emerged as promising tools for automating various code-related tasks. However, their capabilities for fundamental code analysis remain underexplored. Understanding these capabilities is crucial for assessing and enhancing LLM-based programming tools. We structure our study around three aspects of code analysis aligned with human practices: syntax parsing, static semantics inference, and dynamic reasoning. We evaluate 21 state-of-the-art LLMs across nine representative tasks spanning multiple programming languages (C, Java, Python, Solidity), including Abstract Syntax Tree (AST) generation, Control Flow Graph (CFG) construction, data dependency analysis, taint analysis, and flaky test reasoning. We apply a rigorous three-layer evaluation protocol (combining automated metrics, expert adjudication, and consistency validation) to 3,124 code samples. The protocol achieves high inter-rater reliability (Cohen's \(\kappa=0.844\) – \(0.936\) ) and strong human-machine agreement (Gwet's AC1 = 0.500–0.727, F1 = 0.791–0.882), ensuring methodological rigor. Our results reveal that while best-performing LLMs excel in syntax parsing (AST 90%+, expression matching 84–100%) and show initial promise in static analysis, their dynamic reasoning capabilities remain limited ( \(<\) 70%) with high data-shift sensitivity (F1 scores varying 0–1.0 across projects). This capability hierarchy (strong syntax parsing, moderate static analysis, weak dynamic reasoning) appears consistently across different model families and scales, suggesting fundamental rather than transient limitations. These findings offer new insights into how LLMs can complement traditional code analyzers: LLMs provide cross-language generalization but produce non-deterministic outputs requiring validation, while traditional tools provide deterministic guarantees but require language-specific configuration. We contribute a validated evaluation framework with systematic comparison against traditional analyzers (Tree-sitter, Soot, Joern), task-specific applicability tiers for deployment guidance, and clarify directions for future enhancement of LLM-based code analysis. Our open benchmark is released at https://github.com/mathieu0905/llm_code_analysis.git .

Topics & Concepts