Weak-to-Strong Honesty Alignment via Group-Relative Policy Optimization

Despite recent progress in aligning large language models (LLMs) with human intent, achieving honest alignment — models’ ability to express epistemic uncertainty and refuse to answer when lacking knowledge — remains an unsolved challenge, especially under low-resource supervision. In this paper, we introduce GREAT, a fundamentally novel alignment paradigm that integrates group-level reinforcement alignment with weak-to-strong epistemic supervision, explicitly targeting the honesty objective. Unlike prior methods that rely on costly human annotation or fragile pairwise preference modeling, our framework leverages a lightweight “honest head” trained with Group Relative Policy Optimization (GRPO) to identify high-fidelity uncertainty-aware responses from beam outputs, which are then used to bootstrap a large-scale pseudo-labeled corpus. Crucially, our design integrates (1) intra-group epistemic consistency constraints, (2) a dual-head actor-critic architecture, and (3) soft ensemble decoding to resolve the dilemma between cautious refusal and over-conservatism. Experiments across diverse domains — including opendomain QA and a sensitive military medical equipment testbed — demonstrate that GREAT not only outperforms prior alignment methods under limited data, but also generalizes robustly to unseen domains. Our framework introduces a new paradigm in LLM alignment by operationalizing epistemic humility as a trainable signal, offering a pathway toward safe and transparent deployment of LLMs in high-stakes settings. Extensive experiments show GREAT significantly boosts honest alignment in large models even with limited labeled data. Our code is available at https://github.com/cynicalight/Weak-to-Strong-Honesty-Alignment-via-Group-Relative-Policy-Optimization.