Unlocking Scalable Private‑Domain Recommendations with a “4+N” Architecture

1. Background and Challenges

Business Background

Because private‑domain recommendation scenarios are numerous and fragmented, rebuilding model pipelines for each new request is costly; a generic marketing recommendation model framework is needed to improve model deployment efficiency.

Technical Challenges

Four main challenges:

Scenario complexity: heterogeneous recommendation items, long‑tail items, and the need for a unified feature framework.

Pipeline complexity: many systems must be integrated.

Model complexity: deep neural network models are harder to develop and deploy than traditional machine‑learning models.

Collaboration complexity: multi‑party coordination is required.

2. Architecture Goals

The desired recommendation model framework should be systematic, standardized, and automated, supporting rapid rollout and iteration of lightweight marketing recommendation algorithms.

Target Architecture

Build a “systematic, standardized, automated” recommendation model framework that supports quick launch and iteration of lightweight marketing recommendation algorithms.

Capability Decomposition

Fast‑launch capability: quickly build baseline models and go live.

Scenario‑reuse capability: the same framework serves multiple scenarios with minimal code changes.

Fast‑iteration capability: allow feature addition/removal and model updates without a full release cycle.

3. “4+N” Recommendation Model Architecture

Overall Design

Close the model development loop by adding a feature‑log feedback step to the traditional five‑step pipeline (sample preparation, feature engineering, model training, model deployment, model serving).

The “4+N” idea standardizes sample, feature, model, and pipeline so that one set solves N scenarios.

Engineering Pipeline

The recommendation pipeline consists of three major processes:

Recall: direct recall from business systems or “search engine + recall system”. Core systems: recommendation platform, operation platform, search engine, recall system.

Ranking: data preparation on a big‑data platform, feature processing and model development on a model‑dev platform, model deployment on a deployment platform. Core systems: big‑data, model‑dev, model‑deployment.

Re‑ranking: traffic control and shuffling rules configured on the operation platform.

4. Implementation and Business Impact

One Set of Features

Construct a generic feature set for private‑domain scenarios, including user, item, and user‑to‑item features. User features are relatively static profiles; item features contain basic info and traffic metrics; u2i features capture historical interactions.

One Set of Samples

Generate training samples by joining click‑through logs with feature logs, requiring standardized logging of item_id, item_type, session_id, and consistent feature logs.

One Set of Models

Use deep learning models (wide&deep baseline, later deepFM, AFM, DIN) for ranking. Feature extraction is automated via full‑table statistics and binning, with SQL‑generated configuration files.

Model Debugging and Distributed Training

Provide notebook‑based single‑node debugging and Yarn‑based distributed training using the same codebase, encapsulating data processing and evaluation so engineers focus on network architecture.

One Pipeline

Unified integration on the operation platform connects external systems, enabling engineers to concentrate on recommendation logic.

Model Deployment

Deploy models via a C++‑based prediction service; model‑dev and deployment platforms are linked, and mock testing is supported.

Recommendation Pipeline

Feature view configuration, recall configuration, model invocation, feature query, feature‑log feedback, and AB testing are integrated on the recommendation platform built on Ant Financial’s SOFA architecture.

Recall Pipeline

Introduce search‑engine capability, creating a chain: business system → search engine → recall platform → recommendation platform → business system, providing multi‑path real‑time recall.

N Scenarios

The “4+N” architecture has been deployed in multiple private‑domain scenarios, cutting model rollout time from two weeks to two days.

5. Future Outlook

Future work includes building a “two‑graph‑one‑network” private‑domain recommendation capability, leveraging knowledge graphs for long‑term user value, social graphs for community‑based recommendations, and multi‑objective optimization networks to balance short‑ and long‑term metrics.