Trustworthy DataOps for the Medallion Lakehouse
Abstract
Modern data platforms increasingly rely on DataOps to orchestrate large-scale, automated data pipelines. However, most existing approaches prioritise speed and agility while treating trust spanning data quality, compliance, and lineage as a secondary concern, which is particularly problematic in regulated environments. This paper introduces a Trustworthy DataOps framework for a Medallion Lakehouse architecture, operated under a Data Mesh model, where trust becomes a first-class, measurable property of the pipeline. The proposal embeds Trust Data Gates between the Bronze, Silver and Gold layers, governed by a composite Trust Data Score (TDS) that aggregates three dimensions: data quality, policy and regulatory compliance, and lineage/observability. Promotion thresholds and circuit-breaker mechanisms prevent untrusted data from reaching analytical and ML workloads, while quarantine paths ensure controlled recovery from failures. The framework is operationalised through a continuous lifecycle (Discover, Instrument, Enforce, Observe, Improve) and set of trust-aligned KPIs that monitor the health of both data and processes across layers. Together, these mechanisms enable measurable trust, shift-left compliance and improved auditability, turning Medallion Lakehouse data products into explainable, governable and recoverable assets rather than opaque pipeline outputs.
Fields of Study & Subjects
Scientific Computing and Data Management
Access Control and Trust
Research Data Management Practices
Publication details
- Journal / source
- 2026 6th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)
- Publisher
- IEEE
- Pages
- 1–9
- Publication date
- 14 May 2026