AutoFlow: An LLM-Agent Framework for Self-Correcting, Multi-Step Data Pipeline Synthesis

The automation of complex data engineering workflows represents a critical frontier for enterprise efficiency. While Large Language Models (LLMs) have demonstrated proficiency in generating code for isolated data tasks, their application to real-world enterprise scenarios characterized by large-scale, multi-step pipelines requiring robust verification remains fundamentally limited. Existing approaches typically treat tasks atomically, lacking mechanisms for strategic decomposition, state persistence across steps, and autonomous error correction. This gap leads to a sharp decline in LLM accuracy and reliability when faced with intricate workflows such as preparing a dataset for regulatory reporting, which may involve schema alignment, temporal joins, business rule application, and quality validation in a specific, dependent sequence. To bridge this gap, we introduce AutoFlow, a novel LLM-Agent framework for the self-correcting synthesis of end-to-end data pipelines. AutoFlow conceptualizes pipeline creation as a reinforced planning-and-verification problem. The system employs a meta-reasoning LLM agent to dynamically decompose a high-level natural language specification into a directed acyclic graph (DAG) of subtasks. Each node’s execution is handled by a specialized tool-calling agent, and crucially, the output state of each step is programmatically verified against a set of constraints. Failed verifications trigger a feedback loop where a corrective agent analyzes the error context including code, output data samples, and constraint violations and proposes a revised action, enabling iterative self-repair without human intervention. The design of AutoFlow is grounded in and extends principles from recent advancements in LLM-based agents and program synthesis. The framework’s dynamic decomposition and planning module is inspired by the "Chain of Thought" prompting paradigm and its extensions for tool use, notably as explored in "ReAct: Synergizing Reasoning and Acting in Language Models" (Yao et al., 2022). While ReAct interleaves reasoning traces with actions for question answering, AutoFlow adapts this synergy for complex task planning, where the "reasoning" involves generating a validated execution plan and the "acting" involves invoking a curated set of data operations (e.g., pandas transformations, SQL executions, quality checks). Furthermore, AutoFlow’s core innovation in state-aware, self-correcting execution directly addresses limitations observed in earlier program synthesis systems. It builds upon the conceptual foundation of "Self-Debugging: Teaching Large Language Models to Debug Their Own Code" (Chen et al., 2023), which demonstrates that an LLM can improve code generation by leveraging execution feedback and error messages. AutoFlow generalizes this self-debugging capability from single-function code to a multi-agent workflow context. It formalizes verification not just by runtime errors, but through rich, domain-specific constraints (e.g., "column customer_id must be non-nullable and unique," "output row count must match the source after left join"), creating a more robust feedback signal for the corrective loop. We implement a prototype of AutoFlow with a configurable toolkit for tabular data operations and evaluate it against a novel benchmark, EntEval-Complex, comprising 50 high-fidelity, multi-step data engineering tasks derived from real business intelligence and data warehousing scenarios. Tasks require an average of 4.7 interdependent steps. We benchmark AutoFlow against three strong baselines: (1) direct LLM prompting with the full task description, (2) a fixed Chain-of-Thought (CoT) prompting strategy with step-by-step instructions, and (3) a naive LLM-agent with tool access but no integrated verification or correction loop. Our evaluation employs a dual metric suite: syntactic correctness (executability of the final end-to-end script) and semantic accuracy (the fidelity of the output data to a ground-truth result). Results demonstrate that AutoFlow significantly outperforms all baselines in semantic accuracy, achieving a score of 78.4% compared to 41.2% for direct prompting, 55.7% for CoT, and 62.1% for the naive agent. The self-correction module proves critical, resolving 67% of intermediate verification failures autonomously. Analysis reveals that failures in the strongest baseline (naive agent) are predominantly due to error propagation across steps and an inability to recover from semantic mismatches flaws that AutoFlow’s architecture is explicitly designed to mitigate. Furthermore, we present detailed case studies illustrating AutoFlow’s ability to recover from complex errors such as misinterpreted join logic or misapplied aggregation windows, showcasing its iterative refinement process. This work makes three primary contributions. First, we identify and formalize the core challenge of autonomous, reliable multi-step pipeline synthesis as a distinct problem in AI-for-data engineering. Second, we introduce the AutoFlow framework, featuring a novel closed-loop architecture with integrated planning, execution, state verification, and self-correction. Third, we release the EntEval-Complex benchmark to spur further research in this domain. Our findings indicate that LLM-based automation for enterprise data engineering must move beyond single-step code generation toward systems that manage complexity through iterative, feedback-driven refinement. AutoFlow provides a foundational step in this direction, highlighting the necessity of verification as a first-class citizen in the pipeline synthesis process. Future work will explore integrating external knowledge graphs for constraint generation, optimizing the planning algorithm for computational efficiency, and extending the framework to support streaming data pipelines.