How Alibaba’s Smart Dialogue Platform Turns Search into AI‑Powered Conversations

Alibaba’s Shenma Search is evolving from a traditional web search engine into an intelligent dialogue platform, building a comprehensive AI‑driven information service that supports Tmall Genie speakers and personal voice assistants.

Terminology Alignment

TaskBot Engine : The core processing unit is a “skill”, defined as a structured (query + content) vertical‑scenario task such as real‑time queries, tool or control tasks.

QABot Engine : Includes KG‑QA, QAPair, and DeepQA engines. KG‑QA focuses on encyclopedia‑style precise answers using a full‑web knowledge graph; QAPair handles large‑scale QA pair production and consumption; DeepQA is a multi‑stage system based on URL indexing, classification, focus‑word extraction, and summarization.

ChatBot Engine : Provides both retrieval‑based and generation‑based chit‑chat capabilities.

Content System

Web search and intelligent dialogue share data, algorithms, and architecture. Leveraging this synergy, Alibaba’s AI platforms (Google, etc.) can quickly launch AI products for B2B/B2C services.

Industry Skill Library : Phase 1 structured 100+ vertical industries (entertainment, travel, finance, etc.); Phase 2 refined query structures and multi‑turn dialogues, deployed to Tmall Genie.

Full‑Web Knowledge Graph : Provides knowledge cards, entity recommendations, and precise QA.

QA Library : Includes community QA (≈1 B docs), UPGC production (student‑driven “Knight Squad”), high‑quality curated QA, and “egg‑white” responses for partial answers.

Core Library : Operates via combined operations and mining to ensure content quality.

Example queries and their corresponding libraries:

“How many points does the Rockets lead by?” → Skill Library “Who invented basketball?” → Knowledge Graph “Will Harden enter the Hall of Fame?” → QA Library “Let’s talk about the NBA.” → ChatBot Library

According to a recent Stone Temple study, Google Assistant answers 68 % of user questions with 90.6 % accuracy, while Microsoft Cortana answers 56.5 % with 81.9 % accuracy, Apple Siri 21.7 % with 62.2 % accuracy, and Amazon Alexa 20.7 % with 87 % accuracy.

Architecture System

The engine handles data construction and computation, while the platform provides a closed‑loop solution for production, multi‑tenant consumption, operations, and demand management. The system is fully decoupled from search and supports traffic from Tmall Genie and other services.

ShenJiang Platform

Skill Open Platform : Offers both content and capability openness. Provides a BotFramework for external developers to build skills and an OpenAPI for direct skill consumption.

Skill Production Platform : Enables internal R&D to produce built‑in skills, supporting multi‑scenario deployment (no‑screen, mobile, large‑screen) and C++ dynamic libraries for ranking and NLG strategies.

Statistics Analysis Platform : Multi‑dimensional metrics, reporting, and efficiency analysis for production and consumption.

Operation Management Platform : Manages content operations (real‑time interventions) and application operations (CRUD and training of skills).

TaskBot Engine

Offline Computing : Transforms external material into internal data such as entity dictionaries, classification models, intent‑slot plugins, NLG templates, DM scripts, and ranking plugins.

Content Management : Versioned management of data per application/skill, ensuring stateless, portable, and rollback‑able assets.

Scheduling : Handles data dispatch, environment management (iteration, validation, pre‑release, production), and service management (traffic‑based sharding, scaling).

Online Engine : The SDS engine processes user queries using a DM (Dialog Manager) as the control core, NLU for understanding, US for recall and ranking, and NLG for response generation. Reported trigger rates of 97‑98 % and accuracy above 95 % for various skills on Tmall Genie.

The DM maintains dialog context, manages multi‑turn flows, and isolates domain‑specific logic from the core engine. The NLU has two designs: one returning structured Domain/Intent/Slot for BotFramework developers, and another providing multi‑dimensional N‑Best results for dialog products, enabling disambiguation of entities like “Li Bai”.

QABot Engine

Focuses on large‑scale retrieval‑based QA, divided into KG‑QA, Baike‑QA, Deep‑QA, and Pair‑QA. KG‑QA and Baike‑QA offer high precision but limited coverage; Deep‑QA handles unstructured data with broader coverage but more noise; Pair‑QA leverages social production to boost productivity.

Key components:

Question Understanding : Uses Alibaba’s search NLP stack for semantic expansion, weighting, entity recognition, and correction; classifies questions (e.g., chat, person, organization, time).

Information Retrieval : Combines inverted‑index text search and vector‑based semantic search, selecting the appropriate method per corpus and scenario.

Answer Generation : Applies ranking models (CNN, DSSM, GBDT) for Pair‑QA and deep models (Bi‑LSTM, summarization, cross‑validation) for Deep‑QA.

Corpus Construction : End‑to‑end pipeline for oral QA data, including open‑question mining, scenario mining, social answer production, and high‑quality answer extraction.

Graph Engine

The knowledge graph is the core infrastructure of Shenma Search, powering knowledge cards, entity recommendations, and precise QA. Specialized skills (recipes, poetry, Three Kingdoms, world records) are delivered to Tmall Genie, while continuous innovation in knowledge extraction and reasoning enriches the graph.

Conclusion

In the past year, the intelligent dialogue team completed a technology upgrade from search to AI‑driven dialogue, establishing a robust AI + information‑service architecture, algorithms, operations, and content system that now powers multiple Alibaba services.