How Generative Pretraining Overcomes Discriminative Model Bottlenecks in Ad Ranking

Background

Current ad ranking models rely on a discriminative paradigm and suffer two core bottlenecks: (1) loss‑y compression of long‑sequence information, and (2) diminishing returns from parameter scaling, as research [1‑3] shows that larger discriminative models can degrade performance. This contrasts with large language models (LLMs) that benefit from simultaneous data and parameter scaling.

To address these issues, the authors outline two generative‑based technical paths that are emerging in the industry: (1) generative pre‑training plus discriminative application, and (2) end‑to‑end LLM‑style ranking. Path 1 efficiently uses whole‑site data to alleviate sparsity and over‑fitting, integrates smoothly with mainstream DLRM architectures, and is well‑suited for extracting value from ultra‑long user sequences.

Model Solution (UserLLM)

UserLLM follows a three‑stage progressive adaptation pipeline: Generative Pretrain → Discriminative SFT → CTR Fusion . The pipeline first pre‑trains on massive, whole‑site click sequences to learn universal user‑item collaborative signals, then fine‑tunes on ad‑domain data to align with downstream CTR tasks.

Stage 1: Global Generative Pretrain

Training Data – User click sequences from all site scenarios (search, recommendation, ads) are aggregated chronologically, truncated or split to a maximum length, and each item is represented by its ID plus attribute features (category, brand, etc.). Only the most recent sequence per user is kept for inference.

Model Architecture – An Item Encoder embeds raw item features into dense vectors via learnable embeddings and a non‑linear projection to a fixed‑size Item Token. A User Encoder based on the LLaMA Transformer decoder processes the Item Token sequence with causal attention, ensuring each position only sees historical behavior. Multiple Transformer blocks produce a final user context vector.

Training Objective – Self‑supervised next‑token prediction (causal mask) optimized with sampled softmax. Because the item vocabulary reaches billions, the RecIS framework [9] provides exposure‑frequency‑weighted negative sampling and a shared negative pool, while a reject_mask excludes a user’s historical clicks.

Stage 2: Advertising‑Domain Discriminative SFT

Training Data – Real ad exposure logs (positive and negative clicks) are appended as candidate tokens to the end of the T‑1 day click sequence, forming a mixed behavior sequence that aligns pre‑training and downstream distributions.

Model Structure – The Item and User encoders remain unchanged; an Adapter module is inserted after each attention and feed‑forward layer to retain pre‑training knowledge while allowing task‑specific adaptation. A custom mask combines causal masking with a Candidate Mask that prevents exposure‑level leakage between ads on the same day.

Prediction Head – All candidate tokens share a two‑layer MLP that maps the User Encoder’s hidden state at each candidate position to a click probability, trained with binary cross‑entropy. Only candidate ad positions contribute to the loss; historical sequence positions participate in context interaction but do not affect the loss.

Training Strategy – Adapter projection layers are initialized to zero, creating an identity mapping that keeps the model output identical to the pre‑trained state at the start of fine‑tuning, ensuring stable, gradual adaptation.

Stage 3: Downstream Model Application

Pretrained Representations – After SFT, item vectors (from the Item Encoder) and user vectors (the final hidden state of the User Encoder) are hashed to IDs for downstream consumption.

Ranking Model Integration – Various strategies are explored: (a) Pretrain EMB – using vectors as collaborative incremental features; (b) SimScore – computing user‑item similarity scores; (c) Pretrain Sequence Attention (PSA) – sequence‑level attention over pretrained vectors; (d) Pretrain Sequence Score (PSS) – per‑position similarity features; (e) Generative Pretrain EMB (GPE) – replacing traditional ID embeddings with frozen pretrained vectors and letting the downstream model learn to adapt.

Parameterization Trick – Instead of concatenating dense pretrained features (which inflates storage), the vectors are stored as model‑internal embedding parameters, enabling ID‑based lookup without extra I/O. A linear dimensionality‑reduction layer compresses high‑dimensional vectors to low‑dimensional ones without loss of performance.

Incremental Scheduling

A cascading schedule is designed: weekly pre‑training on new whole‑site data, monthly SFT on fresh ad exposure data, and daily downstream representation updates using a “new‑representation + old‑dense” warm‑start strategy.

Experimental Results

Evaluation Metrics

A hierarchical evaluation uses whole‑site HitRate@K for generalization, Proxy Task AUC for rapid validation of ad‑domain representations, and downstream CTR performance as the final benchmark.

Offline Metrics

Comparing a two‑stage “pretrain → CTR” pipeline with the three‑stage pipeline (including SFT) on an online DLRM baseline shows that SFT reduces overall HitRate but the drop on ad items is much smaller (‑6.3% vs ‑15.1%). Proxy Task AUC improves by +3.02 pts, and downstream CTR gains confirm that discriminative fine‑tuning aligns the generic collaborative signal with the ad domain.

CTR Fusion Strategy Comparison

Multiple downstream ranking architectures were tested with the various pretrained‑representation strategies; results (shown in the accompanying tables) demonstrate consistent improvements when integrating UserLLM vectors.

Scaling Law Verification

Models were scaled along three axes—token dimension (embedding size), number of layers, and network width. HitRate and downstream AUC continuously increased, with token‑dimension expansion yielding the largest gains, confirming that the generative‑pretrained architecture continues to benefit from additional compute.

Conclusion and Outlook

The three‑stage “generative Pretrain → discriminative SFT → CTR Fusion” paradigm transforms ultra‑long behavior sequences into high‑dimensional representations, overcoming the compression and scaling bottlenecks of pure discriminative models. Future work will expand behavior signals beyond clicks, enrich user‑side interest pools and item‑side multi‑channel embeddings, and simplify the pipeline to a two‑stage “mixed pretrain‑adaptation” approach for broader scenario reuse.