Architectural Considerations for Data Engineering for Building Scalable, Governed Data Pipelines
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Balancing scalability and governance represent a central architectural challenge in contemporary data engineering pipelines. As organizations shift from monolithic data warehouses to distributed lakehouse architectures, the demand for rapid data ingestion often conflicts with requirements for data quality, compliance, and auditability. This research examines a unified architectural strategy in which governance controls, including schema enforcement, data quality validation, and lineage tracking, are integrated directly within the compute layer. A controlled experimental evaluation utilized a synthetic high-throughput log ingestion workload with 495 processing instances. The pipeline employed Apache Spark for distributed processing, an ACID-compliant data lake for storage, and Great Expectations for automated validation. Results quantitatively reveal the influence of governance strictness and data volume on end-to-end pipeline latency. Although strict governance introduces measurable performance overhead, the findings indicate that a tiered bronze-silver-gold medallion architecture mitigates scalability constraints by deferring computationally intensive checks to later stages. This study offers practical architectural recommendations for designing governed data pipelines that jointly optimize throughput and compliance for enterprise analytics.
Fields of Study & Subjects
Data Quality and Management
Advanced Database Systems and Queries
Big Data Technologies and Applications
Publication details
- Journal / source
- International Conference on Sentiment Analysis and Deep Learning (ICSADL)
- Pages
- 637–642
- Publication date
- 18 February 2026
- Language
- EN