Design and Real-Time Optimization of Xianyu E‑commerce Search System

Search is the core traffic entry for e‑commerce platforms, driving transactions, intent convergence, and activity placement. A stable, efficient, and scalable search system is essential for platform survival. This article describes how to build a complete product search system and presents the timeliness optimizations applied to Xianyu’s second‑hand marketplace.

Search Engine Basics : Modern open‑source engines such as Lucene and Elasticsearch (or Alibaba Cloud OpenSearch based on Ha3) provide core capabilities. Key concepts include tokenization, inverted index, forward index, and various index types (text, spatial, vector, numeric). Search actions involve tokenizing the query, matching documents via the inverted index, and ranking results.

Online Service Flow : The request passes through eight steps – from request access (parameter validation, load balancing, security interception) to application layer (business logic), sorting access layer (intent prediction, request assembly, engine recall, fine‑ranking model scoring, re‑ranking rules, external mixing), and finally back to the client. Each module is described with its responsibilities.

Offline Modules : Include full‑index building (Fullindex), batch incremental index (IncIndex), and real‑time index (RtIndex) generation. These modules handle data dumping, cleaning, and index construction.

Stability : Xianyu deploys multi‑datacenter active‑active architecture for both application and engine layers to ensure high availability.

Surrounding Systems : Advertising placement, evaluation system, and debugging platform support the core search pipeline.

Timeliness Optimization : Xianyu’s unique characteristics—single‑stock items, frequent changes, and lack of clear hot/cold splits—require strong real‑time processing. Past delays reached up to 8 hours, prompting a dedicated optimization effort.

Searcher Expansion : Increased the number of searcher columns from 16 to 24, boosting real‑time incremental processing capacity by 50%.

Engine Architecture Governance : Separated query engine from detail engine to balance CPU/memory resources and reduce incremental pressure.

Incremental Grading : Developed a plugin to classify field updates by urgency, routing only time‑critical changes through the real‑time path.

Auxiliary Table Write Diffusion : Mitigated exponential growth of incremental messages caused by joins with auxiliary tables by converting high‑frequency fields to online auxiliary tables.

After these optimizations, Xianyu achieved near‑zero search latency, effectively realizing real‑time search.