Enhancing Remote Direct Memory Access Network With Loss-Tolerant Flow Control for Distributed AI Clusters

Current AI training clusters widely utilize RoCEv2 to enhance the communication efficiency of interconnect networks across machines. RoCEv2 relies on priority flow control (PFC) to ensure a lossless network. However, PFC brings specific side effects, such as head-of-line blocking, congestion spreading, and deadlock. Numerous studies have been proposed to eliminate the side effects. Unlike traditional high-performance computing applications, distributed machine learning (DML) is bounded-loss tolerant. In light of this observation, this paper proposes a loss-tolerant flow control (LTFC). LTFC does not rely on PFC to ensure a lossless environment, but to control the packet loss ratio within the tolerance threshold. Compared to the traditional trigger condition, LTFC reduces the likelihood that the PFC will be triggered. Additionally, we replace RoCEv2's default Go-back-N mechanism with a non-retransmission mechanism to eliminate retransmission delay. Further, we adapt LTFC to high network load scenarios through loss-aware buffer management. We demonstrate the bounded-loss tolerance feature of DML on our testbed and evaluate the performance of LTFC in simulations. Simulation results show that LTFC reduces the average flow completion time (FCT) slowdown by up to 27.2% and tail FCT slowdown by up to 23.8%.