Lychee-FD Achieves Breakthrough in Full‑Duplex Speech Modeling, Wins ACL 2026 Outstanding Paper

Problem

Full‑duplex speech conversation requires a model to listen, understand, generate speech, and control turn‑taking continuously, while preserving semantic reasoning efficiency.

Scientific Insight

The paper *Hierarchical Acoustic‑Semantic Modeling: Modality Separation and Semantic Coherence for Full‑Duplex SLMs* identifies two root causes of performance loss in native full‑duplex models:

When acoustic and semantic sub‑tasks share deep‑layer parameters, gradient conflict arises: shallow layers cooperate, but deeper layers diverge, leading to “intelligence loss”.

High‑frequency acoustic signals drown sparse textual supervision, causing semantic dilution.

Architectural Innovation

Lychee‑FD adopts a hierarchical acoustic‑semantic framework:

Shallow layers retain a shared backbone for joint low‑level representation learning.

Deep layers are split into three dedicated channels—semantic, acoustic, and dialogue‑control—eliminating deep‑layer parameter sharing.

A dense semantic alignment channel preserves internal semantic cues during speech generation.

Engineering Implementation

Using vLLM as the inference backbone, a real‑time parallel multi‑stream engine dispatches intermediate representations to the three channels, each with its own KV‑cache. This design achieves a 2.96× speedup and reduces GPU memory usage by 23% compared with the baseline single‑stream engine.

A “control‑head early‑exit” strategy generates control tokens (e.g., start/stop, interruption) ahead of full speech/text output, providing a fast path for rapid response.

Experimental Results

On the Spoken QA benchmark, Lychee‑FD improves accuracy by 7.4%.

On FullDuplexBench 1.5, it gains 28.5%.

Across three full‑duplex benchmarks, it leads on 10 evaluation metrics, establishing state‑of‑the‑art performance.

Demos

Lychee‑FD powers a digital‑human demo (Soul‑LiveAct) where a virtual avatar continuously listens, speaks, and synchronizes facial expressions in real time.

It also drives the physical robot “Shennie”, which exhibits the same native full‑duplex interaction.

Open‑Source Release

Model, code, paper, and project site are publicly released:

Repository: https://github.com/HITsz-TMG/Lychee-FD

Paper: https://arxiv.org/abs/2607.06540

Project site: https://hitsz-tmg.github.io/Lychee-FD