Efficient Reinforcement Learning from Human Feedback via Bayesian preference inference
Abstract
Learning from human preferences is essential for aligning machine learning models with subjective judgments, but collecting preference data is costly. We study a hybrid framework that combines the scalability of Reinforcement Learning from Human Feedback (RLHF), which trains neural reward models from pairwise comparisons, with the sample efficiency of Preferential Bayesian Optimization. The method integrates Laplace-based Bayesian uncertainty estimation to guide informative preference queries. On high-dimensional Rosenbrock optimization, the approach successfully converges in problems with up to 50 dimensions. In large language model (LLM) fine-tuning, it improves reward-model accuracy by a value within 6 − − 14 % under limited annotation budgets.
Fields of Study & Subjects
Reinforcement Learning in Robotics
Explainable Artificial Intelligence (XAI)
Behavioral and Psychological Studies
Publication details
- Journal / source
- IFAC Journal of Systems and Control
- Publisher
- Elsevier BV
- Volume
- 35
- Pages
- 100398–100398
- Publication date
- 1 March 2026
- Language
- EN