Scaling Up iQIYI’s Ad CVR Model: 10× Parameter Growth and GPU Inference Deployment

Background

In recent years, iQIYI’s advertising CVR models evolved from sparse‑feature lightweight MLPs to much larger dense networks with richer feature interactions and longer user‑behavior sequences. Over the past two years the model’s parameter count grew tenfold, and dense compute rose from 5‑8 MFLOPs to 60‑70 MFLOPs, pushing the existing TensorFlow + CPU stack to its cost and capacity limits.

Why PyTorch + GPU?

Matrix‑heavy workloads (large matrices, long sequences, deep nets) map naturally to GPU‑accelerated batch‑matmul, layer‑norm and vector reductions.

PyTorch offers a dynamic‑graph experience, mature CUDA kernels, AMP/BF16 support, attention modules, operator fusion and export tooling, shortening the path from research to production.

Upgrade Goals

The aim was not merely to replace the framework but to restore sustainable iteration space for the ad‑ranking pipeline. TorchRec [1] was chosen as the migration target, and a GPU‑enabled online inference chain was built to support larger dense networks, more complex feature interactions, and longer user‑behavior sequences.

Model Migration and Effect Alignment

TensorFlow and TorchRec differ in training mode, parameter defaults, and pipeline construction, requiring extensive adaptation of the existing TensorFlow code. Key differences include:

Training mode: TorchRec uses synchronous all‑reduce updates; TensorFlow uses asynchronous sparse updates via parameter servers.

Parameter handling: TorchRec needs fine‑grained control of dense and sparse parameters; TensorFlow relies on existing online defaults.

Efficiency: TorchRec requires manual pipeline tuning to hide I/O overhead; TensorFlow Estimator provides optimized data‑reading APIs for CPU‑centric workloads.

Initial offline AUC showed a ~1 pp gap between TorchRec and the production TensorFlow model. Using AI tools and TorchRec source code, the team identified several root causes and applied the following optimizations:

Initialize dense layers with Glorot uniform and sparse embeddings with uniform unit scaling to match TensorFlow’s initialization.

Correct default aggregation for multi‑value sparse features from sum to mean to align expected behavior.

Redesign loss and optimizer settings for TorchRec’s synchronous all‑reduce: use mean reduction, split optimizers for sparse and dense parameters, and apply AdamW with a scaled base learning rate to compensate for larger global batch sizes.

Ensure consistent GPU initialization across workers by decoupling static model configuration (embedding tables, pooling methods) from dynamic request‑time metadata (sequence lengths, offsets).

After these steps, the TorchRec offline AUC matched the TensorFlow baseline, eliminating the effectiveness gap and enabling further scaling.

Dense Parameter Modeling (Scaling‑Up Core)

The upgraded CVR architecture consists of three layers: feature input, feature interaction, and multi‑task prediction. The interaction layer adopts the Query Decoding and Query Boosting ideas from HyFormer [2].

Feature input: Two groups – sequence features (user behavior sequences) and non‑sequence features (user, item, context). Sequence features are represented as S=[s₁,s₂,…,s_T] after embedding and concatenation; non‑sequence features are embedded and concatenated as well.

Tokenization: Non‑sequence features are split into N independent heads (multi‑head tokenization) to preserve semantic diversity with minimal extra parameters.

Feature interaction: Queries are constructed by concatenating each non‑sequence token with a pooled representation of a behavior sequence, passed through a small FFN to produce a global query token. Cross‑attention between queries and sequence keys/values retrieves relevant historical signals. Attention temperature scaling prevents softmax collapse, ensuring smooth information flow.

Token Mixer: After cross‑attention, the resulting query tokens are merged with the original non‑sequence tokens (total N+S tokens) and processed by a TokenMixer‑Large‑style module that performs split → mixing → revert steps, achieving full‑token interaction while preserving token semantics.

Offline experiments showed that increasing the interaction layer from 1 to 2 stacked layers improved AUC by over 0.1 pp, indicating clear future scaling potential.

Prediction Layer

The prediction head uses a PLE + MLP structure: three single‑layer expert networks followed by a two‑layer MLP with non‑linear activations. Core hyper‑parameters include token_dim=128, interaction‑layer depth = 1, non‑sequence token count = 13, sequence decode token count = 3, and sequence length = 50. This configuration yields an offline AUC gain of 0.4 pp over the baseline.

Model Export Challenges

TensorFlow’s SavedModel format handles static graphs well, but exporting PyTorch/TorchRec models is difficult due to dynamic sparse features (offsets, variable sequence lengths). The solution decouples static model assets (embedding weights, feature‑to‑embedding mappings, pooling methods) from request‑time metadata, representing the latter as explicit tensor inputs. This eliminates reliance on Python objects or fixed shapes, ensuring robust online inference.

Inference Performance Optimizations

Initial GPU inference retained many PyTorch paths, resulting in lower‑than‑expected throughput and latency. Detailed profiling identified bottlenecks in CPU‑GPU data movement, sparse lookups, variable‑length handling, dense computation, and request merging. The following optimizations were applied:

Data movement: Batch concatenation on CPU, then a single transfer to GPU followed by in‑GPU expansion; reuse GPU buffers to cut redundant copies.

Sparse lookup merging: Consolidate multiple embedding lookups, feature alignments, and variable‑length handling into fewer GPU kernels, reducing kernel launch overhead.

Dense computation: Use TensorRT for graph optimization, operator fusion, and BF16 inference; dynamic profiling supports varying batch sizes and sequence lengths.

Kernel scheduling: Reuse execution contexts and buffers per batch bucket; capture stable kernel launch sequences with CUDA Graphs to replay without CPU‑side launch overhead.

Request merging (Triton Server): Implement a “User + Item” mixed batch where a single user feature is paired with multiple item features; defer user‑feature repetition until just before concatenation to minimize memory use.

Automated tuning workflow: Adopt test‑driven development (TDD/SDD) and a develop/debug/test/profiling/monitor agent pipeline, cutting manual tuning time by 50%.

Business Impact

CVR model: revenue + 3.42 %, RPM + 2.81 %, clicks + 1.91 %, conversions + 7.22 %, deep‑revenue + 2.28 %.

DCVR model: revenue + 1.60 %, RPM + 1.21 %, clicks + 1.01 %, conversions + 0.25 %, deep‑revenue + 2.37 %.

Inference efficiency: machine cost reduced by 15 %; timeout rate dropped from 0.15 % to 0.03 %.

Future Roadmap

Model architecture: Leverage freed GPU capacity to enlarge dense parameters, incorporate longer sequence modeling, lightweight multi‑head self‑attention, and deeper multi‑task learning.

GPU inference: Pursue further operator fusion, INT8 quantization, dynamic batching, embedding caching, and multi‑model co‑location with elastic resource scheduling.

Platformization: Codify the migration artifacts (model schema, export toolchain, performance‑tuning methodology, Triton configuration) into a reusable TorchRec training‑inference framework, turning future model onboarding into a configuration‑driven process.

References

[1] Ivchenko D, Van Der Staay D, Taylor C, et al. TorchRec: a PyTorch domain library for recommendation systems. In: Proceedings of the 16th ACM Conference on Recommender Systems, 2022: 482‑483.

[2] Huang Y, Jin J, Hong S, et al. HyFormer: revisiting the roles of sequence modeling and feature interaction in CTR prediction. arXiv preprint arXiv:2601.12681, 2026.

[3] Zhu J, Fan Z, Zhu X, et al. RankMixer: scaling up ranking models in industrial recommenders. In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025: 6309‑6316.

[4] Jiang Y, Zhu J, Han X, et al. TokenMixer‑Large: scaling up large ranking models in industrial recommenders. arXiv preprint arXiv:2602.06563, 2026.