Self-Healing Data Operations in Modern Data Pipelines

Read full text Summarise with VIRAT
Authors: Pathum T Fernando
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

My research focuses on self-healing data operations in modern data pipelines, with the goal of designing autonomous, resilient, and trustworthy data infrastructure for large-scale analytics and machine learning systems. I investigate how data pipelines can move beyond reactive monitoring toward intelligent, closed-loop self-healing mechanisms that autonomously detect anomalies, diagnose root causes, and execute remediation actions with minimal human intervention. The work integrates concepts from data observability, AIOps, anomaly detection, causal root cause analysis, and automated remediation within distributed batch and real-time data ecosystems. A key contribution of my research is the formulation of layered architectural models that unify infrastructure telemetry, data quality signals, dependency graphs, decision engines, and learning feedback loops to enable continuous reliability improvement. I place strong emphasis on quantitative evaluation methodologies, using synthetic and empirical experiments to measure improvements in reliability metrics such as mean time to detect (MTTD), mean time to repair (MTTR), automation rate, and data quality service levels. My research also addresses governance, security, explainability, and zero-trust data principles, which are critical for deploying autonomous data systems in regulated and mission‑critical environments. Current research directions include the application of machine learning and causal inference for root cause analysis, multi‑criteria optimization for remediation decision-making, and the emerging role of large language models and agentic workflows for explainable and proactive self-healing. The overarching aim is to establish self-healing data pipelines as a foundational capability for scalable, reliable, and future-ready data platforms in both industry and academic contexts.

Fields of Study & Subjects

Software System Performance and Reliability Advanced Database Systems and Queries Data Quality and Management

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN