EchoChat: Modeling Empathetic Speech Dialogue with Perception, Theory of Mind, and Unified Training

From Voice to Response: The Steps of Empathetic Dialogue

In daily conversation, an appropriate response relies on a chain of judgments: what acoustic cues reveal the speaker's emotion and its intensity? Given the dialogue context, what beliefs, intentions, and psychological needs might the speaker hold? How should the response's content, tone, and emotional expression be chosen? Merely recognizing an emotion category is insufficient; the same label "sad" can manifest as sobbing or quiet melancholy, each demanding a different response. EchoChat explicitly splits the empathy process into three interconnected stages:

Perception: Identify emotions and related acoustic cues from speech to ground subsequent reasoning.

Theory of Mind Reasoning: Combine emotional cues from the voice with the dialogue context to infer the speaker's beliefs, intentions, and latent psychological needs, and formulate a suitable empathy strategy.

Speech Response Generation: Produce a supportive spoken response whose content, prosody, and emotional expression match the user's inferred needs.

However, this multi-stage reasoning faces a "one error cascades" challenge: a mistake in an early stage can propagate and lead to an inappropriate final response. EchoChat tackles this through a unified optimization framework that provides targeted feedback at each stage.

Addressing Error Propagation with Unified Training

Acoustic-Anchored Attention (AAA)

Speech carries information far beyond transcribed text: intonation, rhythm, pauses, and loudness all provide clues to emotion, especially when verbal content and vocal delivery diverge. During supervised fine-tuning (SFT), EchoChat introduces Acoustic-Anchored Attention (AAA) to strengthen the link between the model's understanding process and the raw acoustic information. The mechanism directly asks: when the model analyzes the user's emotion and state, are those judgments anchored in acoustic evidence? For empathetic speech dialogue, fully utilizing such evidence is a critical foundation for later mental-state inference and response generation.

Step-Decomposed Credit Assignment (SDCA)

The stages of empathetic reasoning are not independent. When a response is inadequate, the fault may lie in misperceived emotion, correct emotion but misjudged need, or reasonable understanding but poor response choice. If training feedback only evaluates the final reply, it cannot precisely distinguish these cases. In the reinforcement learning (RL) phase, EchoChat introduces Step-Decomposed Credit Assignment (SDCA) . It supplies stage-specific training signals and employs a gating mechanism that constrains the reward of later stages by the performance of earlier ones, preventing the model from receiving credit for superficially plausible reasoning or responses. This enables the training to pinpoint why a reply is unsuitable and which step should be improved.

Dataset and Benchmark: EchoDialogue-400K and EchoEval

To advance empathetic capabilities of speech large models, the team constructed EchoDialogue-400K , a dataset covering diverse language styles, varied dialogue scenarios, and fine-grained emotional expressions. Alongside, they introduced EchoEval , an evaluation benchmark featuring human-recorded speech and annotations by psychology experts, assessing models across multiple empathy dimensions. This multi-dimensional evaluation turns "empathetic ability" into a research problem that can be further decomposed, analyzed, and improved.

Experimental Results

Experiments show that EchoChat achieves leading performance on all three dimensions: emotion perception, mental-state reasoning, and response alignment. These results support the paper's core thesis: decomposing empathy into interlinked cognitive steps helps the model form better-grounded understanding and generate responses more closely aligned with the user's state and needs. The multi-dimensional evaluation also allows finer scrutiny of a single "I understand you": did the model capture subtle acoustic cues? Is the inferred mental state reasonable? Is the final response consistent with the preceding understanding? Together these aspects form the basis for measuring empathy capability.

Conclusion: Grounding "I Understand You" in Evidence

A fitting response begins with listening to the voice and depends on understanding context and needs. EchoChat integrates emotion perception, Theory of Mind reasoning, and speech response generation into a unified training framework, enabling joint optimization of the complex multi-stage empathy process. When an AI says "I understand you," we expect it to hear the emotion in the words, grasp the unspoken needs, and respond just right.