SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training
Abstract
Training large language models (LLMs) efficiently while preserving model quality poses significant challenges, particularly with subbyte precision supported by state-of-the-art GPUs. Current mixed-precision training approaches either apply uniform precision to all GEMM operations or rely on heuristic-based methods that fail to generalize during training, leading to suboptimal convergence and instability.
Fields of Study & Subjects
Numerical Methods and Algorithms
Parallel Computing and Optimization Techniques
Advanced Neural Network Applications
Publication details
- Journal / source
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)
- Pages
- 1815–1831
- Publication date
- 10 March 2026