Prototype Entropy Alignment: Reinforcing Structured Uncertainty in LLM Reasoning

Recent research reveals that a minority of high-entropy tokens significantly influences the reasoning quality of large language models (LLMs). Inspired by this, we propose Prototype Entropy Alignment (PEA), a reinforcement learning framework that models effective reasoning not as a single path, but as a collection of learnable "entropy signatures." PEA identifies these signatures by clustering the uncertainty patterns of expert trajectories into a dynamic set of prototypes. It then rewards the model for aligning its own reasoning process with these evolving targets, creating a self-improving loop. Instead of replacing traditional outcome-based rewards, PEA provides a complementary, process-oriented signal. Our experiments show this synergy is crucial: PEA substantially boosts performance on creative and general reasoning tasks, and when combined with outcome rewards, achieves state-of-the-art results on structured tasks like mathematics. By rewarding alignment with diverse and evolving reasoning structures, PEA offers a robust, verifier-free pathway to enhance the depth and adaptability of LLM reasoning.