Redundant Hierarchical Ring All-Reduce in Hypercubes

Distributed training of large language models demands efficient gradient synchronization strategies. Existing All Reduce algorithms often fail to fully leverage underlying topological properties, resulting in low edge utilization and unbalanced load in hypercube networks. This paper proposes a Redundant Hierarchical Ring All-Reduce (RHRA) algorithm tailored for hypercube topologies. The algorithm first designs a hierarchical Ring All-Reduce structure with high edge utilization by exploiting the hypercube's inherent high parallelism and symmetry, proving that the optimal number of hierarchical layers is n 2. It then introduces controlled redundancy, dynamically adjusting transmission strategies based on the relationship between the value of bandwidth B and the value of data volume D to reduce communication hops and transmission latency while ensuring reliability. Experiments demonstrate that our algorithm achieves the minimum number of gradient synchronization hops under varying bandwidth constraints, with end-to-end transmission time significantly outperforming existing schemes. Further reliability simulations verify its effectiveness in balancing redundancy to prevent resource waste while maintaining high reliability.