Five Lines, One Question, A Micro-Benchmark for Evaluating Structural Code Understanding

While large language models demonstrate impressive fluency in code generation, their ability to perform precise, structural reasoning about code remains poorly understood. This paper introduces Five Lines, One Question (5L1Q), a minimalist benchmark that isolates fundamental reasoning capabilities by presenting models with trivial five-line code snippets and atomic questions about their structure. Our evaluation of state-of-the-art models reveals a striking reasoning-comprehension gap: although models excel at pattern-based tasks like token localization (up to 92% accuracy), their performance dramatically degrades on tasks requiring structural reasoning such as change detection (as low as 22% accuracy) and symbolic substitution. This consistent failure pattern, where models struggle with operations that are trivial for traditional program analysis tools, suggests current architectures lack robust internal mechanisms for representing and manipulating code structure. Our benchmark provides a lightweight, interpretable diagnostic tool for evaluating this critical dimension of code understanding, offering a pathway toward models that combine generative fluency with genuine analytical capability.