Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity

Large Language Models (LLMs) demand extensive computational resources for fine-tuning, posing significant challenges for widespread customization. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA mitigate this, their reliance on fixed ranks often leads to inefficient resource allocation and performance bottlenecks due to the heterogeneous importance of different model layers. To address this, we propose Layer-wise Adaptive Sparse Fine-tuning (LASF), a novel approach that significantly enhances LLM fine-tuning efficiency. LASF integrates Dynamic Rank Allocation, which adaptively determines each layer's optimal LoRA rank based on its real-time contribution, with Sparsity Regularization, which prunes redundant parameters within these low-rank matrices. A Lightweight Evaluation Module ensures minimal overhead for this adaptive process. Extensive experiments demonstrate LASF's superior efficiency-performance balance. It consistently achieves lower task loss while substantially reducing average memory usage, training latency, and energy consumption compared to stateof-the-art PEFT methods and full fine-tuning. Ablation studies confirm the additive benefits of dynamic ranks and sparsity, and human evaluations validate that LASF maintains or improves the subjective quality of generated text. These findings establish LASF as a highly practical and robust solution for efficient LLM customization, enabling sustainable and high-quality adaptation across diverse domains.