New Pretraining

Abstract Language models have demonstrated exceptional capabilities in semantic understanding, long-document reasoning, and sequence modeling tasks. However, such approaches often impose high demands on GPU memory and computational resources.(e.g., consumer-grade or edge hardware).This work investigates how to retain long‑context abilities while dramatically reducing memory consumption through a combination of orthogonality-preserving optimization, low‑rank bottleneck factorization, and lightweight sparsification (via pruning and knowledge distillation). We propose a knowledge distillation framework for long-context language models that combines low-rank weight update projection, orthogonal gradient updates, and L₁-based lightweight pruning, without introducing any additional trainable parameters. This approach forms a regularized momentum update strategy, enabling stable and controllable model compression under low-memory consumer-grade hardware while maintaining performance. The experiment adopted a hybrid architecture model with a parameter scale on the order of hundreds of millions as the teacher model. Knowledge distillation with a sequence length of 1024 was conducted under three different maximum context length configurations (32K/16K/1024): Maximum context length: 1024, training sequence length: 1024; Maximum context length: 16K, training sequence length: 1024; Maximum context length: 32K, training sequence length: 1024. The core reproducible experiments used a context length of 1024. Results showed that the model training loss could steadily decrease from approximately 2.01 to 1.71, while sparsity was maintained at around 2.3%. Further analysis indicated that, even when the training sequence length was limited by resource constraints, models with long-context architectures could still achieve stable convergence and exhibit significantly superior performance. Conclusions Our findings demonstrate that, under extreme hardware constraints, the combination of orthogonal-preserving optimization, low-rank factorization, and lightweight pruning effectively preserves long-context capabilities. This approach offers a feasible solution for deploying powerful language models on edge devices and resource-constrained platforms, while retaining essential language understanding abilities. Keywords: Orthogonality Optimization, Low-Rank Approximation, Sparsification.