A Localization Framework for Reasoning Faults in LLM-based Code Agents

Autonomous LLM-based code agents are rapidly advancing, yet their practical utility is hindered by a critical gap in our understanding of their failures. When an agent fails, the root cause is often not a simple code bug but a complex flaw in its internal reasoning process, which is captured in unstructured "hypothesis" logs. To date, there has been no annotated dataset to systematically study this phenomenon, especially on complex, competitive programming-style tasks. To address this gap, we introduce APPS-Failure-DB, a new benchmark dataset of 2450 unique, validated reasoning failures from an LLM agent (Qwen 7B Coder) tasked with solving problems from the APPS dataset. We generated this dataset using a novel "Oracle-Annotated Pipeline," where a blind Oracle Agent (GPT-4o) validates and provides ground-truth annotations for each logical failure. Building on this dataset, we propose and evaluate a new, tool-assisted localization framework that is the first to link the unstructured text of an agent’s hypothesis log to specific code-level faults. Our framework combines dynamic analysis (coverage tracing) with a semantic "Tracer Agent" to build a traceability map, enabling a final "Debugger Agent" to perform precise localization. Our results demonstrate that we can characterize failure patterns in agent reasoning, with 73.82% of failures occurring in the Execution-Phase, and achieve a reasoning-phase localization accuracy of 67.48%. While code-line-level localization remains challenging with our 7B parameter model (F1-Score of 0.0714), our framework establishes a foundation for future work with larger models. This work contributes both a new dataset for the research community and a novel framework for diagnosing agent reasoning failures.