From PowerSGD to PowerSGD+: Low-Rank Gradient Compression For Distributed Optimization With Convergence Guarantees
Abstract
Low-rank gradient compression methods, such as PowerSGD, have gained attention in communication-efficient distributed optimization. However, the convergence guarantees of PowerSGD remain unclear, particularly in stochastic settings. In this paper, we show that PowerSGD does not always converge to the optimal solution and provide a clear counter-example to support this finding. To address this, we introduce PowerSGD+, which periodically updates the projection subspace via singular value decomposition, ensuring that it remains aligned with the optimal subspace. We prove that PowerSGD+ converges under standard assumptions and validate its effectiveness through empirical evaluation on large language model tasks.
Fields of Study & Subjects
Iterative Learning Control Systems
Advanced Vision and Imaging
Advanced Data Compression Techniques
Publication details
- Journal / source
- ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Pages
- 921–925
- Publication date
- 21 April 2026
- Language
- EN