Ensuring Deterministic Outputs in Distributed Build Systems Through Runtime-Based Verification

Distributed build systems are used for building large codebases, which generate many files. These can be compiled more efficiently in parallel. Builds on single machines cannot scale to larger software projects. Spreading compilation across multiple machines can improve performance. However, achieving this consistency and reproducibility for different builds poses an important engineering challenge, as customary approaches require predicting output files from tool commands and heuristics. Such prediction mechanisms do not scale well, nor do they support custom toolchains or compiling behavior. A runtime-based approach solves the issue of determinism in distributed build systems by recording real file operations during task execution using filesystem call tracing. Size-based verification protocols ensure a complete synchronization of the output before the dependent tasks are invoked. This article is independent of the tools being used to prevent problems with partial file reads and lowers the risk of race conditions between producer and consumer tasks from breaking the build. Systematic techniques to recover from dependency failures due to filesystem propagation delays at low performance overhead have been developed. To adapt these best-effort optimization techniques into correct execution models, distributed builds rely on observation and verification protocols.