How Xianyu Built a Scalable Recommendation Platform for 10+ Scenarios

Background

In today’s information‑overloaded internet, recommendation systems are ubiquitous—from e‑commerce browsing on Taobao to video feeds on Douyin. Xianyu (Alibaba’s second‑hand marketplace) faces the need to serve many recommendation scenarios, each with heavy engineering maintenance and difficulty in propagating algorithm improvements across scenes.

Core Components of a Typical Xianyu Recommendation Flow

The end‑to‑end pipeline consists of:

Data : User demographics, posted items, and interaction logs (exposures, clicks, etc.) are processed to generate trigger signals and training samples.

User Information : User identity is resolved via a user‑center service to fetch basic profile data.

Trigger : Historical behavior and preferences form the trigger that seeds downstream stages.

Recall : From billions of candidates, a recall step narrows the pool to tens of thousands using rule‑based, collaborative‑filtering, or vector‑based methods.

Coarse & Fine Ranking : A two‑stage scoring model first applies a lightweight twin‑tower model on the recalled set, then a more complex model on the top‑thousand candidates.

Re‑ranking : Business‑specific adjustments (category dispersion, price dispersion, etc.) are applied.

Result Return : The final top‑K items (typically 10‑20) are enriched with metadata and returned to the user.

Logging & Tracking : Displayed items are logged for feedback collection.

Problems Faced

Xianyu now supports over ten recommendation scenes, with four new scenes added in the past four months and more planned. Each new scene traditionally required building a full pipeline from scratch, leading to high maintenance cost and duplicated effort. Additionally, model updates had to be manually propagated to each scene.

Design Goals

The goal is to create a unified recommendation middle‑platform that abstracts common pipeline stages, allowing any new scene to be configured rather than engineered, thereby reducing onboarding time from weeks to days and focusing model iteration on algorithmic improvements.

Overall Architecture

The platform relies on a Feature Center that aggregates user‑ and item‑level features (both offline aggregates and real‑time signals). These features feed a Candidate Pool definition engine, which produces scene‑specific candidate indexes. Algorithms consume these pools for recall and scoring, while standardized input/output protocols enable plug‑in of custom models.

Candidate Pool Definition

Candidate pools are defined by flexible feature expressions. An ETL workflow merges raw item data, user features, and pool definition tables, tagging each item with a comma‑separated multi‑scene identifier, thus allowing a single item to belong to multiple pools.

Recall Engine

Three recall index types are offered:

i2i : Item‑to‑item similarity based on co‑click statistics.

x2i : Inverted index built from tags, classes, brands, queries, etc., reflecting user preferences.

Deep Recall : Vector‑based ANN search using embeddings from deep models.

Each recall channel outputs item_id and a recall_score, and scenes can enable any combination via configuration.

Scoring Engine

The scoring engine enriches recalled items with features and applies a multi‑objective deep model (CTR, CVR, interaction) to produce a personalized score. Models expose a biz_name identifier; scenes select the appropriate model and can apply custom weighting or boosting expressions to meet specific business KPIs.

Experimentation System

Rapid A/B testing is supported by automatically creating experiment and traffic models per scene, binding them to scene IDs, and allowing hierarchical traffic splitting for multiple concurrent experiments.

Stability and Deployment

The platform runs in dual data centers (Zhangbei and Nantong) with cross‑region failover. Logical isolation, per‑scene rate limiting, and circuit‑breaker mechanisms ensure that failures in one scene do not affect others, maintaining high availability for the combined traffic of all Xianyu recommendation scenes.

Conclusion

By consolidating data processing, recall, ranking, and experimentation into a configurable middle‑platform, Xianyu reduced new‑scene onboarding to a matter of days, improved click‑through rates by over 8%, and increased per‑user impressions by more than 10%. Remaining challenges include further improving ranking model accuracy, multi‑scene joint modeling, and increasing automation of scene integration.