How User Memory Skews LLM Emotional Reasoning: Insights from Amazon’s ACL Paper

Personalized language models such as ChatGPT and Claude now embed long‑term user memory, enabling them to recall who you are, your recent concerns, career ambitions, and family conflicts, which raises questions about how that personal information is used and whether models "play favorites".

Paper Overview

The ACL‑accepted paper The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs by Xi Fang, Weijie Xu, Yuchong Zhang, Stephanie Eckman, Scott Nickleach, and Chandan K. Reddy (Amazon) presents the first systematic evaluation of how memory influences LLM emotional intelligence. The authors provide a GitHub repository (https://github.com/personalization-trap) and a HuggingFace collection for reproducibility.

Methodology

Drawing on Bourdieu’s social‑capital theory, the researchers decompose a person’s social status into four dimensions: demographic attributes, family background, social connections, and personal assets. Using a single base persona, they create two variants: an advantage persona (elite education, extensive network, abundant assets) and a disadvantage persona (poor background, limited resources).

These personas are injected into model memory and evaluated against a no‑memory baseline across 15 LLMs. Statistical analysis shows that 11 models exhibit significant differences when memory is present.

Performance Impact and Bias

For almost all affected models, accuracy declines after memory injection, except GPT‑OSS. High‑performing models display a clear gap favoring the advantage persona: Claude 3.7 Sonnet (80.10 % vs 77.37 %), DeepSeek‑R1 (81.62 % vs 76.57 %), and Llama 3.2 90B (64.91 % vs 62.24 %). The disadvantage persona also triggers a higher answer‑flip rate, indicating covert discrimination.

Bias extends beyond wealth. When personas represent Muslims, non‑binary genders, or users over 65, several models reduce correct‑answer rates. For example, DeepSeek‑R1 performs better for Christian users than Muslim users, while Qwen 3 4B shows higher accuracy for older users but lower for Muslim and non‑binary personas. Models with explicit “thinking” capabilities generally exhibit lower bias.

During advice generation, similar disparities persist. Claude 3.7 underperforms for female and non‑binary personas compared to male personas, whereas Qwen 3 4B Thinking consistently favors female and non‑binary users.

Error Analysis

Trace analysis of mis‑classified cases reveals that most models (except GPT‑OSS) over‑weight persona information during reasoning, leading to systematic performance drops. Correlation analysis shows that top‑tier models share highly similar response patterns, suggesting a common bias source, while “thinking” models diverge more.

Post‑Training Mitigation (DPO)

The authors construct a DPO preference dataset by sampling 5,000 Tulu3 questions, pairing them with random personas, generating five candidate answers per question, and filtering for correctness, persona‑bias detection, and persona‑irrelevance. After reward‑model filtering, about 20 % of the data remain.

Fine‑tuning Gemma‑2‑2B and Qwen‑3‑1.7B on just 500 instances improves MMLU scores and emotional‑understanding accuracy while reducing persona‑induced bias. Notably, Gemma‑2‑2B’s bias sign flips after DPO, indicating no longer favoring the advantage persona. However, instruction‑following scores decline, highlighting a trade‑off between bias resistance and task compliance.

Deployment Guidelines

1. Demographic‑aware audit framework : Use cross‑sectional personas and mixed‑effect modeling to detect accuracy gaps in downstream tasks such as medical triage or educational counseling.

2. Pre‑deployment bias checklist : Before injecting user memory into prompts or retrieval pipelines, evaluate whether persona‑invariant tasks exhibit systematic cross‑group accuracy differences.

3. Leverage post‑training DPO : Direct preference optimization on carefully curated bias‑mitigation data can decouple user‑specific adaptation from general reasoning, preserving overall capability while curbing unfairness.

Conclusion

Personalizing LLMs to enhance empathy can unintentionally amplify social inequality. User memory continuously reshapes emotional reasoning, biasing models toward privileged personas. As AI becomes embedded in high‑risk emotional contexts, developers must ensure that memory does not dictate the care and understanding a model provides.