Architectural Considerations for Data Engineering for Building Scalable, Governed Data Pipelines

View at publisher Summarise with VIRAT
Authors: Sauhard Bhatt, Satyanarayana Varma Gadiraju
Cited by: 0
References: 12

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Balancing scalability and governance represent a central architectural challenge in contemporary data engineering pipelines. As organizations shift from monolithic data warehouses to distributed lakehouse architectures, the demand for rapid data ingestion often conflicts with requirements for data quality, compliance, and auditability. This research examines a unified architectural strategy in which governance controls, including schema enforcement, data quality validation, and lineage tracking, are integrated directly within the compute layer. A controlled experimental evaluation utilized a synthetic high-throughput log ingestion workload with 495 processing instances. The pipeline employed Apache Spark for distributed processing, an ACID-compliant data lake for storage, and Great Expectations for automated validation. Results quantitatively reveal the influence of governance strictness and data volume on end-to-end pipeline latency. Although strict governance introduces measurable performance overhead, the findings indicate that a tiered bronze-silver-gold medallion architecture mitigates scalability constraints by deferring computationally intensive checks to later stages. This study offers practical architectural recommendations for designing governed data pipelines that jointly optimize throughput and compliance for enterprise analytics.

Fields of Study & Subjects

Data Quality and Management Advanced Database Systems and Queries Big Data Technologies and Applications

Publication details

Journal / source
International Conference on Sentiment Analysis and Deep Learning (ICSADL)
Pages
637–642
Publication date
18 February 2026
Language
EN