DHT-Backed Ancestor-Assisted Merkle Verification for Scalable and Real-Time Log Integrity in Cloud Data Lakes

Cloud data lakes accumulate massive volumes of append-only, high-frequency logs from web servers, gateways, security appliances, and databases. Ensuring their integrity is challenging because existing blockchain-assisted auditing schemes are primarily optimized for static or periodically updated data and provide limited support for continuous ingestion, real-time verification, and auditor access control. These limitations lead to high recomputation overhead, delayed tamper detection, and the lack of reusable proofs required for scalable log publishing.This paper introduces a verifiable log publishing framework based on a newly proposed ancestor-assisted Merkle Tree with stride-based checkpoints for efficient verification of encrypted logs. Each log record is canonically formatted, AEAD-encrypted with metadata binding, and producer-attested prior to storage. Parent-node records and ancestor certificates are distributed via a private Distributed Hash Table (DHT) to enable proof reuse, while only an epoch-level root-of-roots is anchored on a public blockchain to provide tamper-evident integrity guarantees. The framework further supports auditor authorization using certification-authority-issued tokens and incorporates micro-root commitments for near real-time verification. Experimental results demonstrate substantial performance improvements: single-record verification is significantly faster than the strongest baseline, verification throughput is markedly higher, commit latency is greatly reduced, and on-chain storage remains constant across all evaluated epoch sizes. These results confirm that the proposed architecture enables scalable, low-latency, and compliance-aware log integrity verification suitable for modern cloud data lake environments.