Block-Aware Distributed Data Pipelines for Out-of-Core Tabular Machine Learning

Modern Machine Learning (ML) pipelines are often bottle-necked by prolonged data stalls, particularly in out-of-core settings where data volumes exceed main memory capacity. In such cases, preprocessing and batch-wise data loading impose substantial I/O overhead, resulting in underutilization of accelerators and, consequently, limited scalability. In this work, we present a block-aware distributed data pipeline system for out-of-core tabular ML composed of two complementary components. The first extends the Mneme preprocessing framework to distributed environments through an MPI-based, multi-level parallel execution model. Python multithreading is also employed, as the preprocessing workload is dominated by disk I/O and native backend execution, both of which release the Global Interpreter Lock (GIL). The second is a cache-aware PyTorch DataLoader that reduces I/O operations by retrieving data batches from a dedicated application-level in-memory cache and applying preprocessing operators on the fly. Experimental results show that the proposed system enforces block-level locality, reducing memory I/O amplification from 44.9× the dataset size to 3.7×, and translates these reductions into up to 9.3× higher data-loading throughput and significant end-to-end training time savings on a large-scale tabular dataset stored in raw-text CSV format.