TAPO: Dynamic Teacher and Perturbed Answer Injection for Policy Optimization

Reinforcement learning (RL) has emerged as a powerful paradigm for aligning large language models (LLMs) with human preferences, particularly in reasoning-intensive tasks. However, existing RL methods such as Group Relative Policy Optimization (GRPO) often suffer from group collapse—where all sampled completions are either correct or incorrect—leading to vanishing gradient signals and ineffective alignment. Furthermore, many approaches focus solely on final answer correctness, overlooking the alignment of intermediate reasoning steps and penalizing output length in a rigid manner. To address these issues, we propose \textbf{TAPO}, a novel RL framework for improving reasoning alignment in LLMs via enhanced optimization signals. TAPO introduces three key techniques: (1) \textit{Dynamic Teacher Injection (DTI)} selectively injects high-quality or adversarial completions into training groups, restoring learning signal in collapsed scenarios; (2) \textit{Perturbed Answer Injection (PAI)} introduces partially correct completions to enable finer-grained contrastive supervision between answer-level and reasoning-level alignment; and (3) \textit{InfoLen-Aware Reward Shaping} penalizes responses based on semantic redundancy and overlength, encouraging outputs that are both informative and concise. Extensive experiments across diverse reasoning benchmarks demonstrate that TAPO achieves superior alignment with desired reasoning behavior, significantly outperforming GRPO baselines. Ablation studies further confirm the individual impact of each technique on improving signal quality and alignment fidelity.