How RLVER Boosts a 7B LLM to Match Top Commercial Models in Emotional Dialogue

Background

Open‑domain multi‑turn dialogue lacks a single correct answer, making it hard to optimize large language models (LLMs) for emotional intelligence (EQ) with static data or expensive human annotations.

Three RL Challenges for Emotional Dialogue

Environment dilemma : Build a realistic, diverse interaction space that allows free model roll‑outs.

Reward dilemma : Convert subjective user satisfaction into a stable, optimizable long‑term reward.

Training dilemma : Achieve stable, efficient online RL training for LLMs over many dialogue turns.

RLVER Framework

RLVER (Reinforcement Learning with Verifiable Emotion Rewards) proposes a unified "environment + reward" user simulator that simultaneously serves as the interaction environment and the reward generator, directly addressing the three challenges.

Simulator as Environment

The simulator creates a living dialogue world populated with diverse user personas, backgrounds, and latent needs. Each simulated user interacts with the model, updates its emotional state in real time, and produces personalized replies, providing an endless, realistic training loop while preventing reward hacking.

Simulator as Reward Model

Based on the SAGE framework, the simulator explicitly models the user's emotional trajectory after each turn. The accumulated "mood score" at the end of a conversation becomes the reward signal, which drives PPO or GRPO optimization.

Global Reward Optimization

Instead of per‑turn feedback, RLVER optimizes the entire dialogue’s emotional trajectory, using only the final mood score to encourage long‑horizon strategies that keep user sentiment rising.

Key Experimental Results

After RLVER training, the Qwen2.5‑7B model’s Sentient‑Benchmark score rose from 13.3 to 79.2, matching top commercial models such as GPT‑4o and Gemini 2.5 Pro. The model retained its general capabilities (math, coding) without catastrophic forgetting.

Deep Insights

Think‑then‑say vs. Reactive Models

RLVER introduces an explicit "think‑then‑say" prompt template that forces the model to perform emotion analysis and strategic reasoning before responding. Two pathways emerge:

Think‑based models develop deep understanding, superior problem insight, and precise empathetic expression, acting like a "soulful confidant".

Reactive models skip the reasoning step, producing faster but more action‑oriented replies that compensate for weaker empathy.

PPO vs. GRPO

Experiments show GRPO yields steadier, balanced capability growth, while PPO pushes specific dimensions (e.g., empathy depth) to higher peaks.

Environment Difficulty

Two simulator variants were tested:

Vanilla : Open, positive feedback, easy for early‑stage exploration.

Challenging : Subtle, strict feedback, higher realism but low tolerance, which hampers early learning.

Results indicate a gentle "coach" is more effective for early model development, while a stricter environment benefits later fine‑tuning.

Robustness of Reasoning

In the challenging environment, models with explicit reasoning chains remain robust (score drop modest), whereas non‑reasoning models collapse (score ≈ 19.8), demonstrating that internal reasoning mitigates sparse‑reward instability.

Practical Takeaways

When applying RL to open‑ended tasks like emotional dialogue, design training environments with a growth curve rather than excessive difficulty, and equip models with explicit reasoning to improve stability and performance.

References

Paper: https://arxiv.org/abs/2507.03112

Code: https://github.com/Tencent/digitalhuman/tree/main/RLVER

Model hub: https://huggingface.co/RLVER

Code example

来源：量子位
本文
约2600字
，建议阅读
9
分钟
本文介绍了
7B模型。