Recursive Self-Optimizing Reinforcement Learning via Max‑Capability Verification

We present a formal framework and algorithm for recursive self-optimizing reinforcement learning (RSORL), in which both the policy (generator) and the reward evaluation mechanism (verifier) are learned endogenously and co‑evolve. The core architectural innovation is a max‑capability selector that, at each stage, selects the model from a growing population that best serves as the verifier; the verifier assigns rewards to candidate trajectories, and is itself optimized via a meta‑objective that measures the verifier’s downstream impact on future policy improvement. Unlike ordinary RL or RLHF, RSORL dispenses with a hand‑crafted reward function and instead defines reward through the strongest available internal verifier, while guarding against collapse through explicit meta‑training and multi‑path verification. We formalize the framework, provide algorithms (policy update via group‑relative policy optimization inspired by GRPO, verifier meta‑training), and prove two central results: (1) conditional monotone improvement in closed, fully verifiable MDPs under mild assumptions, and (2) an unavoidable discriminative‑closure limitation when all evaluation signals are internal to the modeled family. We discuss implications for scientific‑scale reasoning systems and directions to mitigate the closure limitation (external grounding, multi‑modal anchoring, occasional external audits).