Human-in-the-Loop Agentic Data Pipelines: Designing Safe and Governable Autonomous Operations in Data Engineering
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Modern data engineering systems face increasing operational complexity as data volumes, schema diversity, and pipeline interdependencies grow. Autonomous agentic systems—AI agents capable of reasoning, planning, and executing data operations—promise to reduce manual toil and improve responsiveness to anomalies and schema changes. However, fully autonomous actions in production data systems introduce significant risks including schema drift propagation, data corruption, incorrect backfills, and cascading failures that can compromise data integrity and downstream analytics. This paper proposes a layered architectural framework for human-in-the-loop (HITL) agentic data pipelines that balances automation benefits with safety guarantees. We define core design principles—approval gating, policy enforcement, auditability, and rollback mechanisms—and present a five-layer reference architecture comprising data pipeline, observability, agent reasoning, human governance, and execution orchestration layers. Through concrete operational scenarios including schema evolution handling, anomaly remediation, and late-data recomputation, we demonstrate how explicit human oversight at high-uncertainty decision boundaries can prevent catastrophic failures while preserving automation gains. We analyze trade-offs between automation velocity and safety, discuss limitations of current approaches, and identify future research directions in policy learning, trust calibration, interpretability, and formal verification of agent actions.
Fields of Study & Subjects
Software System Performance and Reliability
Advanced Database Systems and Queries
Scientific Computing and Data Management
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 30 March 2026
- Language
- EN
- Document type
- Preprint