Trustworthy DataOps for the Medallion Lakehouse

View at publisher
Authors: Aymen Fannouch, Jihane Gharib, Youssef Gahi
Cited by: 1
References: 21

Abstract

Modern data platforms increasingly rely on DataOps to orchestrate large-scale, automated data pipelines. However, most existing approaches prioritise speed and agility while treating trust spanning data quality, compliance, and lineage as a secondary concern, which is particularly problematic in regulated environments. This paper introduces a Trustworthy DataOps framework for a Medallion Lakehouse architecture, operated under a Data Mesh model, where trust becomes a first-class, measurable property of the pipeline. The proposal embeds Trust Data Gates between the Bronze, Silver and Gold layers, governed by a composite Trust Data Score (TDS) that aggregates three dimensions: data quality, policy and regulatory compliance, and lineage/observability. Promotion thresholds and circuit-breaker mechanisms prevent untrusted data from reaching analytical and ML workloads, while quarantine paths ensure controlled recovery from failures. The framework is operationalised through a continuous lifecycle (Discover, Instrument, Enforce, Observe, Improve) and set of trust-aligned KPIs that monitor the health of both data and processes across layers. Together, these mechanisms enable measurable trust, shift-left compliance and improved auditability, turning Medallion Lakehouse data products into explainable, governable and recoverable assets rather than opaque pipeline outputs.

Fields of Study & Subjects

Scientific Computing and Data Management Access Control and Trust Research Data Management Practices

Publication details

Journal / source
2026 6th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)
Publisher
IEEE
Pages
1–9
Publication date
14 May 2026