How Does Alibaba’s Tmall Genie Achieve Full‑Duplex Natural Dialogue?

What Is Full‑Duplex Natural Dialogue?

Full‑duplex natural dialogue enables a voice assistant to listen and respond continuously without requiring the user to wait for a prompt, making interactions feel more natural and conversational.

For example, imagine Sun Wukong using the golden staff with Tmall Genie: in a traditional turn‑by‑turn exchange he must wake the staff, issue a command, wait for a response, and repeat; with full‑duplex dialogue he can simply say “make it bigger” repeatedly without re‑waking or waiting.

Technical Solution

The system consists of several modules:

Device side : handles listening and speaking, determines when to capture audio and when to output speech.

ASR (Automatic Speech Recognition) : converts user speech to text and extracts acoustic features.

NLU (Natural Language Understanding) : interprets the text and transforms it into machine‑readable intents.

TTS (Text‑to‑Speech) : synthesizes spoken responses from text.

DM (Dialogue Management) : uses NLU results and conversation context to invoke services and fulfill user requests.

Human‑Machine Interaction Recognition : determines whether the captured audio belongs to the user speaking to the device.

Device‑Side Interaction Management

When a user activates natural dialogue, the server sends a recording command to the device. The device enters a listening state, starts capturing audio when speech is detected, and stops when speech ends, reporting both the total dialogue duration and the user's speaking duration to the server.

If the server’s response does not contain a new recording command, the device exits the natural‑dialogue state. The command also includes a maximum recording length; if no speech is detected within that window, the device exits automatically.

Device‑Side Playback Management

During interaction, the system must decide whether to resume or stop ongoing playback. Three playback types are defined: playback that should resume after interruption, playback that should not resume, and prompt tones. The current and upcoming playback types determine whether the playback state is saved.

Duration Management

Continuous listening would overload cloud processing, so the system limits dialogue duration using a sliding‑window timer: after each user interaction the timer resets, allowing flexible yet bounded interaction periods.

Human‑Machine Interaction Recognition

The system uses acoustic features extracted by ASR as input to a deep‑learning model that decides whether the speech originates from the user speaking to the device. If the speech is identified as unrelated, the assistant remains silent.

Conversation Flow

NLU retains conversation history and uses it to interpret follow‑up requests, enabling natural multi‑turn interactions such as “What’s the weather today?” followed by “Tomorrow?” without repeating the full query.

Conclusion

Full‑duplex natural dialogue is a comprehensive engineering effort that spans device‑side signal processing, cloud‑side ASR, NLU, TTS, and dialogue management, requiring close collaboration across multiple teams. After launch, the feature received widespread user praise and higher engagement compared to other functionalities.

Acknowledgments

The team thanks all contributors, some of whom even gave up their holiday plans to deliver the end‑to‑end solution within a month.