The Next Breakthrough for Speech LLMs: Turning Your Voice Model into a Prosody‑Aware Text Model

Speech large language models (LLMs) often suffer a severe "modality gap" when the same model that excels at text interaction is adapted for voice, leading to notable drops in logical reasoning and basic correctness.

Two waves of industry attempts have tried to close this gap. The first introduced the Thinker‑Talker architecture, buffering speech through a textual Thinker before synthesis. The second focused on output alignment via knowledge distillation and representation alignment. Even with millions of hours of speech data, models such as Qwen2.5‑Omni still lose more than 15% performance on complex math tasks.

CUHK researchers therefore shifted the focus to the input side, presenting the paper "Minimizing Modality Gap from the Input Side: Your Speech LLM can be a Prosody‑Aware Text LLM" and the TextPro‑SLM architecture. By treating the speech LLM as a text LLM that is aware of prosody, they achieve the industry’s lowest modality gap using only ~1,000 hours of speech data for 3B‑ and 7B‑parameter models.

TextPro‑SLM decouples semantic and prosodic information at the encoder. A modified Whisper‑large‑v3 model, called WhisperPro, adds a decoder and a reconstruction loss so that the encoder outputs both clean text tokens and a compact prosody embedding (capturing emotion, accent, age, timbre, etc.). The LLM then receives these two streams via two injection strategies:

Global Prepending : compress the entire prosody embedding into a single vector and place it at the beginning of the token sequence, acting as an "emotion tag".

Interleaving : intersperse compressed prosody vectors among the text tokens at a 5:1 ratio, preserving fine‑grained emotional cues.

Because the input matches the LLM’s preferred modality, training requires only about 1,000 hours of audio for knowledge distillation and prosody learning, a stark contrast to commercial models that consume millions of hours.

Experimental results show TextPro‑SLM‑7B reduces the average modality gap to 0.7%, far below Qwen2.5‑Omni (3.1%) and SALAD (7.1%). On the VoxEval math benchmark, the gap drops from a baseline 17.5% to 1.8%. Prosody‑related tasks also see state‑of‑the‑art performance, with the interleaving injection further raising the ceiling.

Beyond the numbers, the work suggests that clever feature decoupling at the input stage can be more effective than brute‑force multimodal fusion, offering a practical path for startups and developers to build high‑quality speech agents with modest data budgets.