Semantic Retrieval and Product Ranking in JD E‑commerce Search

JD.com’s search engine consists of three core stages: query understanding, recall, and ranking. Traditional inverted indexing cannot retrieve semantically related items, prompting the adoption of vector‑based retrieval and deep ranking models.

Vector Retrieval – Queries and SKUs are embedded into a shared high‑dimensional space using a dual‑tower model (DPSR). Similar items have smaller distances, enabling semantic matching beyond exact keyword overlap. Approximate nearest‑neighbor methods such as PQ are employed to handle billions of products efficiently.

The model computes a relevance score as f(q,s)=G(Q(q),S(s)) , where Q(q)∈R^{d×m} and S(s)∈R^{d×m} are the query and SKU embeddings, and G is a scoring function (e.g., inner product).

Two‑Tower Architecture – Both query and SKU towers contain separate feature encoders; the query tower incorporates multi‑head projections to capture diverse semantics (e.g., “apple” as fruit or phone). An attention loss optimizes the multi‑head outputs.

Negative Sampling – Positive samples are derived from billions of click logs. Negatives are generated from random and batch sampling, with a tunable ratio to balance popularity and relevance.

Training Optimizations – Custom C++ tokenizers are loaded as TensorFlow operators for consistency, data compression reduces storage, and distributed embedding sharding alleviates bandwidth bottlenecks. Model serving combines vector retrieval and TensorFlow inference in a single service, supporting CPU/GPU hybrid deployment.

Product Ranking – A twin‑network model processes user‑query‑item triples, using shared parameters and ReLU activations to output a relevance score. Features include numeric metrics, textual embeddings, user behavior histories, and pre‑trained ID embeddings via graph‑based methods.

Personalization is enhanced by attention mechanisms that adapt user embeddings based on the current query and candidate items, while timeliness improvements incorporate hourly click and purchase signals, online score calibration, and faster model update cycles.

Overall, the semantic retrieval and ranking solutions have significantly increased conversion rates, reduced query rewrite frequency by ~10%, and improved the freshness and relevance of search results in JD’s production environment.