RackWeave: Hierarchical Gradient Exchange for Distributed AI

In this paper, we present RackWeave, a rack-scale hierarchical gradient exchange framework for distributed deep learning systems. The architecture we propose proposes a different model update flow by co-designing software and hardware components across network interfaces, memory hierarchies, and multi-core CPUs. RackWeave implements features including finegrained gradient chunking, NUMA-aware buffer management, and hierarchical reduction improve throughput. Experimental results on various vision workloads show improvements in performance and cost over existing parameter server and collective communication baselines, with maintained training accuracy.