EchoChat: From Voice Perception to Empathetic Response in Speech Dialogue

Introduction

When a voice says "It's fine, you go ahead," the literal words may convey reassurance, but acoustic cues—lowered pitch, hesitation, a sigh—can reveal a different emotional state. As speech large language models improve at conversation and comfort, a key question arises: can they move beyond merely "sounding warm" to truly understanding subtle emotional nuances and responding appropriately? EchoChat, accepted at NeurIPS 2026, addresses this by modeling empathetic speech dialogue as a structured cognitive process comprising perception, Theory of Mind (ToM) reasoning, and response generation, all optimized within a unified framework.

Three-Stage Empathetic Dialogue Modeling

In daily interaction, an appropriate response stems from a chain of judgments: what acoustic features does the speaker's voice exhibit, what emotion and intensity do they reflect? Given the conversational context, what beliefs, intentions, and psychological needs might the speaker hold? How should the response's content, tone, and emotional expression be chosen? Mere emotion classification is insufficient; the same "sad" label covers both sobbing and quiet melancholy, each demanding a different response. EchoChat explicitly decomposes empathy into three interconnected stages:

Perception: Recognize emotion and associated acoustic cues from speech, providing evidence for subsequent reasoning.

Theory of Mind Reasoning: Combine vocal emotional cues with dialogue context to infer the speaker's beliefs, intentions, and latent psychological needs, and formulate a suitable empathy strategy.

Speech Response Generation: Produce a supportive spoken response with appropriate content, tone, and emotional expression aligned with the user's needs.

Challenges: Error Propagation Across Stages

This multi-stage reasoning faces a "one error cascades" challenge: a mistake in an early stage can distort later judgments and ultimately yield an inappropriate response. EchoChat tackles this from two angles: better utilization of acoustic information and multi-stage optimization.

Solution 1: Acoustic-Anchored Attention (AAA)

Speech carries far more information than its transcription. Prosody, rhythm, pauses, and intensity all provide clues to emotion, especially when verbal content contradicts vocal delivery. During supervised fine-tuning (SFT), EchoChat introduces Acoustic-Anchored Attention (AAA) to strengthen the link between the model's understanding process and the acoustic information. The core question is direct: when the model analyzes the user's emotion and state, are those judgments grounded in the voice? For empathetic speech dialogue, fully leveraging such grounding is a critical foundation for subsequent psychological-state inference and response generation.

Solution 2: Step-Decomposed Credit Assignment (SDCA)

The stages of empathetic reasoning are not independent. When a reply is inadequate, the fault may lie in misperceived emotion, correct emotion but misunderstood need, or reasonable understanding but poor response choice. If training feedback only evaluates the final reply, it cannot precisely localize the error. In the reinforcement learning (RL) phase, EchoChat introduces Step-Decomposed Credit Assignment (SDCA). It provides targeted training feedback for each stage and uses a gating mechanism so that later-stage rewards are conditioned on earlier-stage performance, preventing the model from receiving reward based on superficially plausible reasoning or responses. This enables the training to focus on why a reply is unsuitable and which step should be improved first.

Dataset and Evaluation Benchmark

To advance speech LLMs' empathetic capabilities, the team constructed EchoDialogue-400K , a dataset covering diverse language styles, dialogue scenarios, and fine-grained emotional expressions. Simultaneously, they proposed EchoEval , an evaluation benchmark built from human-recorded speech and annotated by psychology experts, assessing empathetic ability across multiple dimensions. The paper is available at https://arxiv.org/html/2610.04826v1.

Experimental Results

Experiments show EchoChat achieves leading performance in emotion perception, psychological-state reasoning, and response alignment. These results support the core thesis: decomposing empathy into interlinked cognitive steps helps the model form better-grounded understanding and generate responses more aligned with the user's state and needs. Multi-dimensional evaluation also allows deeper scrutiny of a single "I understand": did the model capture subtle acoustic cues? Is the inferred psychological state reasonable? Is the final response consistent with the preceding understanding? Together these stages constitute the basis for measuring empathetic capability.

Conclusion

A fitting response begins with listening to the voice and depends on understanding context and needs. EchoChat integrates emotion perception, Theory of Mind reasoning, and speech response generation into a unified training framework, enabling coordinated optimization of the complex multi-stage empathy process. When AI says "I understand," we expect it to hear the emotion in the words, grasp the unspoken needs, and respond just right.