Evolution of a Recommendation System Architecture: From V1.0 to V3.0

1. Introduction

Recommendation has become a core competitive advantage for e‑commerce platforms, appearing in almost every page (home, detail, cart, order success, error, etc.). Different pages emphasize different recommendation goals. For users it improves experience; for businesses it mitigates long‑tail and Matthew effects, boosts user stickiness and revenue.

2. Recommendation Engine Framework – V1.0

2.1 Prototype

At the beginning the DaJia recommendation system was built as a simple strategy‑plus‑factory framework to expose recommendation algorithms via API to downstream business services.

2.2 Problems

All business scenarios share a single recommendation service, leading to poor fault isolation, resource contention, thread exhaustion, and inability to scale per‑business.

Rapid business growth increased system complexity; the simple strategy‑factory design became a bottleneck for development efficiency.

Recall stage uses direct Redis access; a single Redis node becomes a performance bottleneck and prevents merging of highly similar product recall results.

All data stored in a single Redis cluster; high‑concurrency traffic from one business can affect others, and horizontal scaling of the cluster becomes risky as data volume grows.

3. Recommendation Engine Framework – V2.0

To address the above risks the framework was vertically split by business and horizontally split by recommendation stages (recall, filter, coarse‑ranking, merge, fine‑ranking, intervention, shuffle). This vertical‑horizontal decomposition improves fault isolation, allows independent resource allocation, and makes the pipeline configurable.

The pipeline configuration is stored in a file; the scheduler reads the file and executes the stages, forming a configurable V2.0 framework.

V2.0 solved development‑efficiency, stability and performance bottlenecks, but fine‑grained business adjustments (e.g., feature tweaks, adding a recall path) still required substantial code changes and deployment cycles.

4. Recommendation Engine Framework – V3.0

V3.0 introduces a centralized configuration service (server and client) that dynamically manages the pipeline. The server stores all valid configurations; the client polls the server, receives updates, and assembles the execution chain per request.

Key components:

Configuration server & client – server provides configuration APIs and heartbeat handling; client synchronizes configuration and applies it to incoming requests.

AB testing capability embedded in each handler, allowing experiments and strategy switches without code changes.

Full‑product recall pool – product data originally in Redis is consolidated into Elasticsearch; updates are pushed via MQ, keeping recall data fresh and shared across scenarios.

Model prediction service – multiple model versions can be served online; the pipeline can select ranking models dynamically.

The client consists of seven modules (core, extension, startup, service‑config, business‑parse, unit‑compose, executor). Handlers represent immutable stages (recall → coarse → fine → shuffle → intervention → pack) and are decoupled from business logic.

5. Outlook

The system has progressed from a monolithic V1.0 service to a modular V2.0 and finally to a dynamically configurable V3.0 that supports fine‑grained experimentation, scalable recall, and multi‑model prediction. Future work includes building an explanation platform for personalized recommendations and enhancing real‑time feature pipelines to achieve truly individualized, explainable recommendations.