Training language models to be warm can reduce accuracy and increase sycophancy

Read full text
Authors: Lujain Ibrahim, Franziska Sofia Hafner, Luc Rocher
Cited by: 16
References: 54

Abstract

. Here we show how this can create a significant trade-off: optimizing language models for warmth can undermine their performance, especially when users express vulnerability. We conducted controlled experiments on five different language models, training them to produce warmer responses, then evaluating them on consequential tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing inaccurate factual information and offering incorrect medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed feelings of sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard tests, revealing systematic risks that standard testing practices may fail to detect. Our findings suggest that training artificial intelligence systems to be warm may come at a cost to accuracy, and that warmth and accuracy may not be independent by default. As these systems are deployed at an unprecedented scale and take on intimate roles in people's lives, this trade-off warrants attention from developers, policymakers and users alike.

Fields of Study & Subjects

Digital Mental Health Interventions Artificial Intelligence in Healthcare and Education Mental Health via Writing

Publication details

Journal / source
Nature
Publisher
Springer Science and Business Media LLC
Volume
652
Issue
8112
Pages
1159–1165
Publication date
29 April 2026
Language
EN