High Performance Singular Value Decomposition on GPU Architectures

With the advancement of GPU architecture, matrix computation engines such as NVIDIA Tensor Cores now support double-precision (FP64) General matrix multiplications (GEMMs) with the same efficiency as single-precision (FP32) GEMMs. However, the adoption of this enhanced FP64 capability remains limited, primarily restricted to applications that involve multiple FP64 BLAS3 operations. Singular Value Decomposition (SVD), a fundamental decomposition in numerical linear algebra with numerous applications, can greatly benefit from exploiting this hardware feature. In this article, for FP32 SVD, we propose a novel algorithm, FP64 precision eigenvalue decomposition (EVD) based SVD, specifically designed to leverage the latest GPU architectural features. We provide a theoretical analysis demonstrating the feasibility of our approach on emerging GPU architectures and evaluate it from both accuracy and performance perspectives. Moreover, for FP64 SVD, we introduce a double-blocking band reduction technique combined with a GPU-based bulge chasing algorithm to further accelerate the overall SVD process. Experimental results show that, for FP32 SVD, our EVD-based SVD implementation achieves higher numerical accuracy and delivers speedups of up to 6.1× on H100 and 5.0× on A100 over the state-of-the-art cuSOLVER SVD solver. In the case of FP64 SVD, our method also achieves 4.9× and 4.8× speedups on H100 and A100, respectively. These results highlight the potential of our approach as a highly efficient and accurate solution for SVD on modern GPU platforms.