Deep Optimization of the 58 Yellow Pages Smart Chat Assistant for Enhanced User Experience and Business Opportunity Conversion

Background : At the 2021 World Robot Conference in Beijing, Han Wei, head of the Intelligent Q&A Department of 58.com TEG AI Lab, delivered a talk titled "Deep Optimization of the Yellow Pages Merchant Smart Chat Assistant User Experience". The talk was recorded and is presented here.

The Yellow Pages platform connects users with service merchants (cleaning, moving, repairs, etc.) via text chat. Merchant response delays caused lost opportunities. In 2020, a "micro‑chat" operation model introduced human agents to handle chats, but the goal shifted to an AI‑driven assistant that first engages users, converts opportunities when possible, and falls back to human agents otherwise, achieving a 95% human‑level conversion rate and saving dozens of客服 staff.

Dialogue Smoothness Metric : The team defined "dialogue smoothness" as a key C‑side user experience indicator. A round is considered smooth if the robot does not produce erroneous answers. Metrics include single‑round smoothness, whole‑conversation smoothness (all rounds smooth), and smoothness up to the 5th round (74% initially, later improved to ~90%).

QABot Improvements : The QABot’s ability to answer correctly and retrieve relevant answers was enhanced using several models: FastText (n‑gram features, hierarchical softmax), DSSM (query‑doc embedding), LSTM‑DSSM (adds sequential modeling), and BERT‑based pre‑training. Later, a custom SPTM model (BERT‑like architecture with LSTM encoder and shared‑parameter Transformer) was built to reduce inference latency while maintaining performance.

Retrieval‑Based QA : To boost recall, a retrieval‑augmented QA pipeline was added. Offline, a representation model encodes labeled queries into vectors and builds a Faiss index. Online, user queries are encoded, top‑K nearest vectors are retrieved, and ranking strategies (thresholds, scores) select the final answer.

TaskBot Enhancements : After QABot handles factual queries, TaskBot (task‑oriented dialogue) guides users toward business opportunities. The dialogue system comprises NLU, DST, DP, and NLG, with DST and DP merged into a Dialogue Management (DM) module. Two DM versions were explored: a finite‑state‑machine (FSM) node‑based design and a reinforcement‑learning (RL) based manager that selects optimal actions to maximize reward.

Business‑Side Optimizations : The team built a multi‑level opportunity quality evaluation system, using voice analysis (Lingxi platform) and a classification model to flag low‑quality (advertising) leads. High‑quality leads were enriched with additional slots (contact, location, time, service details). A "need‑exploration" card was introduced to gauge urgency, improving high‑quality lead identification (95% precision, 77% recall).

Slot Extraction Architecture : Given ~50 slot types across categories, a hybrid approach was adopted: entity dictionary matching for high‑frequency slots, model prediction for long‑tail and ambiguous cases, and rule‑based extraction for critical slots like phone numbers. Models evaluated included BiLSTM+CRF, IDCNN+CRF, and BERT; IDCNN+CRF was chosen for core slots, while ALBert‑Tiny+GlobalPointer was selected for richer slots due to its superior F1 gains.

Results and Outlook : After iterative optimization, the smart chat assistant achieved a 95% robot‑to‑human conversion rate, saved dozens of客服 positions, reached 94% dialogue smoothness, and attained 95% accuracy in high‑quality lead extraction. The product, branded "Micro‑Chat Butler", is commercialized with thousands of B‑side merchants generating monthly revenue of several hundred thousand yuan. Future work includes end‑to‑end dialogue models, multi‑objective optimization for C‑side experience, and further enhancements to B‑side lead quality and transaction closure.