A Comparative Analysis of Learning-Rate Selection for Fine-Tuning Transformer Language Models

Learning rate is a critical hyperparameter in the fine-tuning of large language models, strongly influencing convergence, training stability, and generalization performance. However, learning-rate selection is often based on heuristics, and systematic comparisons across different model architectures remain limited. This work presents a controlled empirical study of learning-rate sensitivity in three transformer-based language models of varying scale: GPT-2, OPT-350M, and EleutherAI/Pythia-160M. All models were fine-tuned on the same public-domain dataset, Alice’s Adventures in Wonderland , under identical experimental conditions. Four learning rates (1e-05, 5e-05, 1e-04, and 5e-04) were evaluated over three training epochs while keeping all other hyperparameters fixed. Model performance was assessed using epoch-wise validation loss and final perplexity, enabling direct comparison of convergence behavior and training stability. The results indicate that moderate learning rates lead to stable convergence and improved generalization, whereas excessively large learning rates cause optimization instability, reflected by increasing validation loss and sharp rises in perplexity. OPT-350M achieves the lowest perplexity and exhibits greater robustness at lower learning rates, while GPT-2 and Pythia-160M show higher sensitivity to learning-rate selection. Overall, the study highlights the importance of careful learning-rate tuning and provides practical guidance for stable fine-tuning of transformer-based language models.