Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with Constraints

Large language models with search capabilities frequently exhibit miscalibrated confidence, producing incorrect answers with high certainty.We present Deliberative Searcher, a reasoning-primary framework that integrates search operations into chain-of-thought generation while maintaining explicit confidence calibration.Our method employs constrained reinforcement learning with adaptive Lagrangian multipliers to jointly optimize correctness and reliability.Experiments across five benchmarks demonstrate substantial improvements: our 7B model reduces average false-certain rates from 54% in baselines to 2%, while our 72B variant achieves competitive accuracy with closedsource models and reduces false-certain rates to 9%.The well-calibrated confidence scores also enable more efficient test-time compute: instead of standard majority voting, we use confidence-weighted aggregation and match the performance of 16-sample majority voting with only 4 samples, a 4× reduction in inference compute.These results establish calibrated confidence as a foundation for both trustworthy outputs and adaptive test-time compute, demonstrating the value of the proposed constrained RL framework in search-augmented language models.