Optimizing Intra-Layer Parallel Communication for LLM Training on Systems with Fully-Connected Mesh GPU Topology

Large language model (LLM) training relies on intra-layer parallelism, such as tensor and sequence parallelism, to meet memory constraints. However, this strategy introduces frequent collective communication, which becomes a major performance bottleneck. Systems equipped with high-bandwidth switch-based interconnects can alleviate this cost at the hardware level, but many large-scale GPU clusters still employ full-mesh interconnects with more limited bandwidth. In such systems, existing communication–computation overlap techniques, which generally rely on topology-agnostic algorithms, fail to fully exploit available bandwidth and can even degrade performance when applied to lightweight computations. To address this limitation, we propose AG-WGRAD overlap, a method that hides communication latency by rescheduling specific computations. The key idea is to defer weight-gradient calculations that have no downstream dependencies, and execute them concurrently with All-Gather communications. This design avoids the inefficiencies of partitioned collectives, and achieves effective overlap without requiring modifications to communication kernels. In LLM training workloads on full-mesh GPU systems, our approach reduces backward time by up to 1.79%, and when combined with existing techniques, delivers up to 4.12% reduction in total forward and backward time. The approach scales to multi-node intra-layer parallelism, where it reduces iteration time by up to 3.77% for Llama-34B training and mitigates performance drop from limited inter-node bandwidth. In addition, it remains effective in 3D-parallel training configurations, achieving 1.47% end-to-end time reduction for GPT-3 175B.