End-to-End Entity Extraction for Tmall Genie: Speech2Slot Model and Unsupervised Pre‑Training

The session, hosted by DataFunSummit, features Alibaba’s algorithm expert Zhou Xiaohuan who shares innovations and practices of entity extraction in the Tmall Genie entertainment playback assistant.

Tmall Genie, an intelligent speaker, relies heavily on content‑on‑demand; when a user says, for example, “Tmall Genie, I want to hear Liu Dehua’s *Wang Qing Shui*,” the system must recognize the intent and extract the relevant entities (artist, song).

The traditional pipeline processes the request through ASR → NLU (entity extraction) → IR (content retrieval). This approach suffers from exposure bias (NLU trained on perfect transcripts but tested on ASR output) and mismatched training objectives (different error impacts across transcript positions), leading to error propagation.

To overcome these issues, an end‑to‑end speech‑semantic understanding model is proposed, handling both classification (domain/intent) and generation (entity extraction) tasks directly from raw audio.

The core of the generation task is the Speech2Slot model. Its input is the phoneme posterior matrix (ASR’s intermediate output), and its output is a slot phoneme sequence. The architecture consists of three parts: (1) a Speech Encoder that builds a memory matrix via a Transformer; (2) a Knowledge Encoder pretrained on all entity names in the content library; and (3) a Bright Layer that attends the Knowledge Encoder’s outputs to locate entity‑containing audio slices and predict the corresponding phoneme.

Content‑library information is incorporated by pretraining the Knowledge Encoder on the full entity list and using a cold‑fusion fine‑tuning strategy; during inference a Trie built from the library constrains beam‑search outputs to valid entity names.

Because labeled speech data are scarce, an unsupervised pre‑training method called Phoneme‑BERT is introduced. It masks random frames of the phoneme posterior matrix and predicts the original distribution, differing from text‑based BERT in input/output format, token semantics, and information density. Silent frames are filtered to reduce noise.

The workflow first pretrains the Speech Encoder with Phoneme‑BERT on large unlabeled corpora, then fine‑tunes the entire Speech2Slot model on a small labeled set. This pre‑training scheme can also benefit other end‑to‑end speech‑semantic models.

Experiments show that both the Speech2Slot model and classification models achieve higher accuracy and lower error rates after unsupervised pre‑training, on public English datasets and on Tmall Genie’s own data.

The presentation concludes with thanks and a reminder to like, share, and follow the DataFun community.