Compression rate as reward for unsupervised policy optimization: eliciting reasoning in LLMs

Recent advancements in Reinforcement Learning (RL) have significantly improved the reasoning capabilities of Large Language Models (LLMs). However, these methods often rely on external verifiers or human-labeled data, limiting their practicality. Concurrently, a line of research has re-established the deep connection between language modeling and lossless compression, showing that a model’s compression capability is a powerful unsupervised signal. In this work, we bridge these two domains by proposing Compression Rate Policy Optimization (CRPO)—a novel, fully unsupervised RL framework for improving LLM reasoning. CRPO uses a composite reward signal derived from the theoretical lossless compression of generated outputs, combining the mean of log-probabilities (i.e., the negative compression rate, encouraging high confidence) with their standard deviation (explicitly rewarding mixed exploratory and high-confidence steps). Experiments demonstrate that CRPO can match supervised methods with only the confidence reward and surpass them when incorporating the exploratory reward, highlighting the benefits of balancing confidence and exploration. This work establishes the statistics of a trajectory’s log-probabilities as a new class of intrinsic reward signals, showing better compression can elicit complex reasoning without external supervision.