Teaching Small LLMS to Reason Using Reinforcement Learning

This manuscript presents a reinforcement learning (RL)-based framework to enhance the mathematical reasoning capabilities of large language models (LLMs) using Group Relative Policy Optimization (GRPO). Our study focuses on two parameter-efficient variants of the Qwen2.5 model family: Qwen2.5-0.5B and Qwen2.5-3B, which are fine-tuned on the GSM8K dataset, a benchmark for middle school mathematics problems. To guide the models toward producing accurate, wellstructured, and interpretable solutions, we design a custom reward function that promotes correctness, coherent reasoning steps, and consistent output formatting. Experimental results show that the proposed RL framework significantly enhances the reasoning performance of both models. Specifically, Qwen2.5-0.5B achieves an accuracy improvement from 0.08% to 27.64%, outperforming much larger models such as GPT-3 (175B, 25%) and LLaMA-1 (65B, 20%), while Qwen2.5-3B improves from 55.1% to 74.6%, surpassing models like Minerva (540B), Falcon-180B, and LLaMA-2-70B. Overall, this work demonstrates that reinforcement learning, when combined with carefully designed reward functions and efficient adaptation techniques, can unlock strong reasoning performance in lightweight LLMs, making them viable for deployment in real-world, resource-constrained environments.