Cross-Layer Caching for High-Performance Computing: Experimental Analysis of Alluxio, Ceph Tiering, and Hybrid Designs Using HACC-IO Traces

High-performance computing (HPC) applications exert substantial pressure on distributed storage systems, where backend latency, metadata overheads, and small-object amplification often limit I/O throughput. Caching is therefore critical to achieving scalable performance; however, the relative effectiveness of inbuilt distributed caching (IDC) such as Ceph tiering and external FUSE-based caching (EFC) such as Alluxio remains underexplored under realistic scientific workloads. This study presents a comprehensive evaluation of three caching architectures—Ceph writeback/readonly tiering (IDC), Alluxio Tier-1 selective caching (EFC), and a Hybrid multi-tier design combining EFC with Ceph SSD caching—using 100 GB of real HACC-IO cosmology traces. Results reveal two dominant asymmetries. First, EFC-only achieves the highest Tier-1 read locality, reaching 59.5% T1 hits and requiring nearly 324 backend HDD block reads, compared to only 459 T1 hits and 341 promotions in the Hybrid setup. Second, for write-intensiveworkloads, HybridASYNC_THROUGH+ writeback decisively outperforms all other configurations, reducing backend HDD activity by more than an order of magnitude. In read workloads, Hybrid narrows the performance gap with EFC by serving 332 read misses from SSD rather than HDD, lowering HDD fetches from 324 blocks (EFC) to only 9 blocks (Hybrid) with a 35× reduction in backend disk I/O. Overall, the findings demonstrate that while EFC maximizes pure read locality and IDC excels in write absorption, the Hybrid design delivers the most balanced and robust performance across mixed HPC workloads. By combining Alluxio’s selective promotion with Ceph’s SSD writeback tier, Hybrid caching achieves near-EFC read performance and superior write throughput while dramatically reducing HDD load. These insights provide a quantitative foundation for designing effective multi-tier caching strategies in modern distributed HPC storage systems.