Can Recommendation Ranking Models Follow the LLM Scaling Law? Introducing GPSD for Scalable Transformers

Scaling Dilemma in Recommendation Systems

Industrial recommendation pipelines use a two‑stage recall + ranking architecture, where the ranking stage (CTR/CVR prediction) determines the final items shown to users. Although Transformers have excelled in NLP/CV, simply increasing the parameter count of discriminative ranking models often worsens performance due to severe over‑fitting.

Two over‑fitting patterns are identified:

One‑epoch over‑fitting : validation metrics drop sharply at the start of each epoch because high‑dimensional sparse ID features (users, items) are memorized.

Within‑one‑epoch over‑fitting : during the first epoch, training loss continues to fall while validation performance stalls, caused by highly similar samples and distribution shift between training and online data.

GPSD: Generative Pretraining for Scalable Discriminative Recommendation

Inspired by the robustness of generative language models, the authors propose GPSD , a two‑stage training paradigm:

Generative pretraining stage : a generative task (similar to LLM pretraining) optimizes the sparse item embeddings, providing stable gradients via random negative sampling.

Discriminative fine‑tuning stage : after the embeddings are well‑trained, a traditional discriminative task (e.g., CTR prediction) fine‑tunes only the dense Transformer parameters.

The framework uses a standard Transformer with several enhancements: Pre‑Normalization, RMSNorm, RoPE positional encoding, SwiLU activation, causal masking, sampling softmax to reduce vocabulary‑size cost, shared negative samples, BFloat16 training, AdamW optimizer, linear warm‑up followed by cosine decay.

Sparse Transfer & Sparse Freeze (ST&SF) Strategy

Because item embeddings dominate recommendation models, GPSD adopts a Sparse Transfer & Sparse Freeze policy:

Transfer the pretrained item embeddings to the discriminative model and freeze them during fine‑tuning.

Train only the dense Transformer parameters, allowing the model to focus on complex feature interactions for the downstream task.

Experimental Validation

Over‑fitting Re‑examination

Using a notation LuHvAw (depth = u, width = v, heads = w), experiments on a CTR task show that discriminative models exhibit a large gap between training AUC and validation AUC regardless of scale, and larger models (e.g., L4H256A4) perform worse on validation than smaller ones (L4H128A4).

In contrast, generative models maintain a small, stable gap between training and validation loss, and larger generative models consistently outperform smaller ones, attributed to random negative sampling that stabilizes sparse parameter learning.

Scaling Law on a 5‑billion‑sample E‑commerce Dataset

On a real‑world dataset with 5 billion samples, sparse parameters are scaled from 125 M to 4 B (32×) and dense parameters from 13 K to 327 M (25 000×). Performance improves monotonically with model size and fits a power‑law curve similar to language models. Extrapolation predicts an empirical AUC ceiling of 0.7097 and a loss floor of 0.3695 for the CTR‑XL dataset.

Cross‑Architecture Generality

GPSD is applied to two newer recommendation backbones:

HSTU : a Transformer‑like sequential model that replaces the original Transformer layer.

Wukong : a non‑sequential model that requires fixed‑length input padding.

Four model scales are tested; both architectures gain significant performance and scalability improvements after applying GPSD, confirming the method’s cross‑architecture advantage.

Results on Public Benchmarks

Experiments on public datasets (Taobao, Amazon, etc.) compare baseline models (DeepFM, DIN, DIEN, DMIN, DMR) with and without the ST&SF strategy. All baselines see performance lifts ranging from 2.36 % to 10.03 %, and the GPSD‑enhanced Transformer surpasses all baselines, demonstrating robust over‑fitting mitigation.

Production Deployment in AliExpress “You May Also Like”

The GPSD framework is integrated into the AliExpress homepage recommendation pipeline. The upgraded model (L4H64A4) yields a 7.03 % increase in GMV, a 50 % boost in throughput, 30 % reduction in resource usage, and a 10 ms latency drop, while supporting KV‑Cache for low‑latency serving.

Future Outlook

The authors plan to explore even larger models and investigate knowledge transfer from open‑source LLMs to further advance recommendation scalability.