Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior

Although large language models (LLMs) are increasingly trained using human feedback for safety and alignment with human values, alignment decisions systematically overlook human social diversity. Researchers have advocated for sociotechnical approaches integrating pluralistic values into model training. This study examines how operationalizing these approaches shapes LLM behavior through a systematic evaluation of demographic variation and design parameters within the alignment pipeline. We built an inclusive data collection process, obtaining alignment datasets from US and German participants (N = 1,095 participants, 27,375 ratings) who rated LLM responses across five dimensions: toxicity, emotional awareness, sensitivity, stereotypical bias, and helpfulness. We fine-tuned multiple Large Language Models and Large Reasoning Models using value preferences from different social groups while varying key parameters, including rating scale, disagreement handling, and preference optimization method. Results show systematic demographic effects: male participants rated responses as 18% less toxic than female participants; conservative and Black participants reported 27.9% and 58% higher emotional awareness ratings than liberal and White participants, respectively. Models finetuned on group-specific preferences exhibited distinct behaviors. Technical decisions had even greater impact on toxicity reduction: preserving rater disagreement was 1.6 times more effective than majority vote aggregation; 5-point scales outperformed binary formats by 1.23 times; Direct Preference Optimization (DPO) consistently outperformed Group-based Relative Preference Optimization (GRPO) on both toxicity and emotional awareness optimization. Meta-analysis confirms the robustness of these effects, raising a critical question: How should alignment processes balance expert-driven and user-driven signals to ensure both safety and fair representation?