Utility Estimation for Reinforcement Learning from Human Feedback: Integrating Large Language Models and Choice-Based Conjoint Analysis

Large language models (LLMs) have demonstrated their potential as tools for proposing business strategies based on their vast knowledge. However, because they are trained on general text data from the Internet, they face the problem of "strategic hallucination," offering advice that may conflict with the specific preferences of market consumers. This occurs because, while AI possesses linguistic fluency, it does not internalize the empirical data (Ground Truth) that forms the basis of decision-making. To address this issue, this study proposes the "Utility-Grounded RLHF" framework, which combines Choice-Based Conjoint (CBC) analysis—a methodology from marketing— with Reinforcement Learning from Human Feedback (RLHF). This study proposes a model that derives utility vectors for each attribute through choice-based conjoint analysis and builds a novel utility-grounded reward model that accepts these vectors as the conditional inputs. Subsequently, the generative model is optimized using the Proximal Policy Optimization (PPO) algorithm to internalize the evaluation criteria of the reward model. By proposing a grounding method that directly uses empirical consumer data (CBC) as the input for the reward model, this study aims to fundamentally secure reward reliability. From a practical perspective, the significance of this study lies in presenting a concrete methodology for reflecting company-owned data or market research findings directly into an LLM, thereby enabling the construction of a “trustworthy AI consultant who understands our customers” rather than merely an “AI that speaks well.”