AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing

Read full text Summarise with VIRAT
Authors: Girish Dashmukhe, Yash Vinod Bhat, Manasi Badhe, Vaishnavi Khande, Vishakha Patel
Cited by: 0
References: 2

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Abstract Modern enterprises ingest vast amounts of data from diverse sources, requiring robust ETL (Extract-Transform-Load) pipelines. Traditional pipelines often break due to data inconsistencies or infrastructure faults. This paper proposes a self-healing ETL pipeline architecture built on AWS serverless services (S3, Glue, Lambda, Step Functions, etc.) augmented with AI/ML for automatic error detection and recovery. In our design, raw data lands in Amazon S3, triggers AWS Glue jobs for transformation, and any failures are detected via anomaly detection (e.g. using AWS Glue Data Quality or a SageMaker-trained model). A Step Functions workflow coordinates the ETL tasks and can retry or invoke corrective routines when errors occur. We include the architecture diagram, workflow, methodology, and cost analysis. Experimental results from related work show that self-healing pipelines significantly improve uptime and reduce manual intervention. KEYWORDS Cloud Computing, AWS, ETL Pipeline, Serverless Architecture, Artificial Intelligence, Machine Learning, Data Transformation, Data Quality, Automation, Data Pipeline.

Fields of Study & Subjects

Cloud Computing and Resource Management Data Quality and Management Software System Performance and Reliability

Publication details

Journal / source
INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
Volume
10
Issue
03
Pages
1–9
Publication date
25 March 2026
Language
EN