Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs

Code analysis is a fundamental problem in Software Engineering (SE), playing a crucial role in tasks such as debugging, performance optimization, and security assessment. Human developers typically approach code analysis through syntax parsing, static semantics inference, and dynamic reasoning. Traditional code analysis tools, while effective, are often limited by language specificity, complex configurations, and lack of cross-language generalization. Recently, large language models (LLMs) have emerged as promising tools for automating various code-related tasks. However, their capabilities for fundamental code analysis remain underexplored. Understanding these capabilities is crucial for assessing and enhancing LLM-based programming tools. We structure our study around three aspects of code analysis aligned with human practices: syntax parsing, static semantics inference, and dynamic reasoning. We evaluate 21 state-of-the-art LLMs across nine representative tasks spanning multiple programming languages (C, Java, Python, Solidity), including Abstract Syntax Tree (AST) generation, Control Flow Graph (CFG) construction, data dependency analysis, taint analysis, and flaky test reasoning. We apply a rigorous three-layer evaluation protocol (combining automated metrics, expert adjudication, and consistency validation) to 3,124 code samples. The protocol achieves high inter-rater reliability (Cohen's \(\kappa=0.844\) – \(0.936\) ) and strong human-machine agreement (Gwet's AC1 = 0.500–0.727, F1 = 0.791–0.882), ensuring methodological rigor. Our results reveal that while best-performing LLMs excel in syntax parsing (AST 90%+, expression matching 84–100%) and show initial promise in static analysis, their dynamic reasoning capabilities remain limited ( \(