BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes

Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes.However, the link between fine-tuning different bias terms (i.e., b q , b k , and b v in the query, key, or value projections) and downstream performance remains largely unclear to date.In this paper, we investigate the link between fine-tuning b q , b k , and b v with the performance of the downstream task.Our key finding is that directly fine-tuning b v generally leads to higher downstream performance in low-data regimes, in comparison to b q and b k .We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoderonly architectures up to 6.7B parameters (including bias-free LLMs).Our results provide strong evidence for the effectiveness of directly fine-tuning b v across various downstream tasks.