Innovations and Practices of Entity Extraction in Tmall Genie Voice Assistant

This talk, delivered by Alibaba’s algorithm expert Zhou Xiaohuan, introduces the business background of Tmall Genie’s content‑on‑demand service and the challenges of traditional pipeline‑based voice entity extraction, which suffers from exposure bias, mismatched training objectives, and error propagation across ASR, NLU, and IR modules.

The speaker then proposes an end‑to‑end semantic understanding model that jointly learns classification (domain/intent) and generation (entity extraction) tasks directly from raw audio, eliminating intermediate information loss.

Two generation‑style entity extraction architectures are described: a streaming model that inserts special markers (e.g., "$" for singer, "#" for song title) into the output token sequence, and an encoder‑decoder translation model that directly maps acoustic features to entity names.

To address the difficulty of modeling a massive, unstructured set of content entity names, the Speech2Slot model is introduced. It consists of three components: a Speech encoder (Transformer‑based memory matrix), a Knowledge encoder (pre‑trained on the entire content library using a language‑model objective), and a Bright layer that attends the knowledge representations to locate and classify entity audio slices.

The model incorporates content‑library information via cold‑fusion fine‑tuning of the Knowledge encoder and a Trie‑based constraint during beam search, ensuring that generated slots correspond to existing entities.

Unsupervised speech‑semantic pre‑training is then discussed. Because labeled data are scarce, a Phoneme‑BERT approach masks random frames of the phoneme‑posterior matrix and predicts the original distribution, learning semantic representations without textual transcripts. Differences from text‑based BERT (input/output format, token meaning, and information density) are explained, and a segment‑wise masking strategy is used to mitigate the low information per frame.

Experimental results show that both phoneme‑based and text‑based pre‑training consistently improve downstream entity extraction and classification accuracy on internal Tmall Genie data and public English speech datasets. Visualizations of performance gains are provided.

In summary, the end‑to‑end Speech2Slot architecture, enriched with knowledge‑encoded content libraries and strengthened by phoneme‑level unsupervised pre‑training, delivers robust entity extraction for voice‑driven content playback in Tmall Genie.