CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts

Training expert LLMs in domains with scarce fine-grained annotated data is admittedly challenging, often relying on multiple-choice questions (MCQs).However, standard outcomebased reinforcement learning (RL) on MCQs is risky.While outcome-based RL may improve accuracy, it frequently compromises the reasoning process, yielding internally inconsistent rationales that diverge from the final predictions.Existing solutions to supervise the reasoning process, such as large-scale Process Reward Models (PRMs), are prohibitively expensive.To address this, we propose CLARITY, a costeffective RL framework that enhances reasoning quality using a small, general-purpose LLM only.CLARITY integrates a consistency-aware reward mechanism with a 2-stage refine-thenmonitor training pipeline to enhance reasoning consistency, and a dynamic data reformulation strategy to better exploit annotated data available.Experiments demonstrate that CLARITY can improve the consistency of responses by 16.5% over standard outcome-based RL, and bring an improvement of 7.5% in final accuracy.Human evaluations further confirm substantial gains in factual correctness and reasoning coherence, leading to more trustworthy model outputs.Thus, CLARITY offers a generalizable solution that enables smaller models to effectively guide expert LLM training by monitoring reasoning consistency. 1