Optimizing Distributed Deep Learning Training via Adaptive Gradient Compression and Topology-Aware Aggregation

Distributed training of deep neural networks is crucial for tackling complex machine learning tasks with massive datasets. However, communication overheads often become a significant bottleneck, hindering scalability. This paper explores a novel approach combining adaptive gradient compression with topology-aware aggregation to mitigate these communication costs. Our adaptive compression dynamically adjusts compression ratios based on gradient importance, minimizing information loss. Topology-aware aggregation leverages network characteristics to optimize data transfer paths, reducing latency. We present a theoretical analysis of the proposed method and demonstrate its effectiveness through experimental evaluations on benchmark datasets, showing significant improvements in training time and scalability compared to existing approaches. The approach is particularly relevant in applications such as recommender systems, where large embedding tables require efficient distributed learning techniques, as highlighted in recent work on representation learning by Gaggi.