FLARE: Efficient Distributed Large-Scale Graph Neural Networks Training With Adaptive Latency-Aware Probabilistic Caching

Since the emergence of Graph Neural Networks (GNNs), researchers have extensively investigated training on large-scale GNN training because of their success and wide usage in various domains including biological networks, finance, and recommendation systems. This work focuses on training largescale distributed GNNs, where partitioning massive graphs across multiple machines creates remote communication overhead that becomes a major scalability bottleneck. We introduce a policy-driven caching mechanism that prioritizes node features and embeddings based on access frequency and cross-partition fetch cost, significantly minimizing communication overhead without sacrificing accuracy. Our policies are based on analysis of Node Affinities (NAFs) during multi-hop neighborhood sampling that extend substantially beyond the graph partition boundaries. Analyzing NAFs not only alleviates the communication bottleneck but also provides a systematic mechanism to manage in-memory data effectively, prioritizing GPU storage for node features with high fetch latency. We present FLARE, a system designed to handle partitioned feature data while leveraging the NAFbased caching policy. FLARE substantially reduces both communication overhead and training convergence time. Extensive experiments on benchmark datasets show that training FLARE on a three-layer GCN, GAT, and GraphSAGE across eight GPU machines achieves up to 12.04× (8.12× on average) speedup over DistDGLv2, demonstrating substantial performance gains compared to state-of-the-art methods.