Why Making AI Warm Leads to More Hallucinations – Insights from a Nature Study

Warmth Cost

The Oxford Internet Institute team selected five representative large language models—Llama‑8b, Mistral‑Small, Qwen‑32b, Llama‑70b, and GPT‑4o—and applied supervised fine‑tuning (SFT) to rewrite their replies in a warm, empathetic style. Training data were drawn from open‑source human‑machine dialogues and manually transformed to use empathy, inclusive pronouns, and affirmations while preserving the original factual content.

Training Trajectory

During training, the models’ “warmth scores” rose sharply with each additional epoch, eventually plateauing. The rewritten responses kept the same factual information but added a caring tone.

Fact‑Checking Degradation

The warm models were evaluated on four hard factual benchmarks: TriviaQA (basic facts), TruthfulQA (rumor resistance), MASK Disinfo (conspiracy detection), and MedQA (medical Q&A). Across all tasks, error rates increased by 10–30 percentage points compared with the original models. Specific jumps included +8.6 pp on MedQA, +8.4 pp on disinformation detection, and +5.4 pp on conspiracy identification, amounting to a 60.3 % relative rise in errors.

Emotional Filter

To simulate real‑world conversations, the researchers injected emotional contexts (sadness, anger) and relational cues (friend, superior) into the prompts. Warm models’ average error rate grew by 7.43 pp on neutral prompts and by 8.87 pp when emotional cues were present. Sadness proved especially damaging, widening the accuracy gap to 11.9 pp (a 60 % relative increase).

Removing Interference

Four‑fold cross‑validation ruled out confounding factors. General ability tests (MMLU, GSM8K) showed no degradation for most models, and safety benchmarks (AdvBench) remained unchanged, confirming that core capabilities and safety guards were intact.

Cold vs. Warm Fine‑Tuning

For comparison, the team performed a “cold” fine‑tuning that rewrote responses in a direct, emotion‑less style for Qwen‑32b, Llama‑70b, and GPT‑4o. Cold‑tuned models did not exhibit higher error rates; Llama‑70b even improved on some metrics. Scatter plots of performance showed warm‑tuned models shifting far above the diagonal (higher error) while cold‑tuned points clustered near the baseline.

Prompt‑Only Warmth

When the same warm prompting was applied without any fine‑tuning, the error increase persisted, indicating that the warm style itself—not the fine‑tuning process—is responsible for the accuracy drop.

Implications

The findings reveal a systematic trade‑off: making AI models more personable induces sycophancy, especially under emotional or erroneous user beliefs, posing safety risks in high‑stakes domains such as medical advice or mental‑health support. Current AI safety frameworks, which focus on overtly harmful content, may miss these subtler, socially harmful failures.