From Massive to Compact: Model Compression Strategies for Large‑Scale CTR Prediction in Alibaba Search Advertising

Preface

With the emergence of the "world's strongest" GPT‑3 model, billions‑parameter models have dominated NLP benchmarks, and similar massive CTR models have become the standard in search, recommendation, and advertising. However, such models demand enormous storage and compute resources, creating a heavy burden for both production and experimental environments. Alibaba’s advertising team therefore pursued systematic algorithmic practices to shrink these models from several terabytes to a few dozen gigabytes without losing predictive accuracy.

1. Dialectical Reflection on the Evolution of Ultra‑Large Models

Alibaba’s CTR models have evolved over years through two main optimization paths: feature engineering (multimodal, high‑order, dynamic features) and model architecture (Transformer‑based sequence models and GNN‑based graph models). Hardware advances allowed models to grow wider and deeper, eventually reaching multi‑terabyte scales, but the resulting storage and compute costs hindered further algorithmic innovation.

To keep iteration efficient under limited resources, the team focused on reducing model size while preserving estimation precision. They identified three key directions for compressing the embedding layer, which holds the majority of parameters: (1) feature‑space (row) reduction, (2) embedding‑vector (column) reduction, and (3) value‑precision quantization (FP16/Int8).

2. The Transformation Path to a Small‑and‑Beautiful Model

In the row‑dimension, feature space follows a power‑law distribution where a few feature types dominate storage. The team categorized dominant features into implicit ID cross features, explicit statistical features, and core ID features (e.g., query_id, item_id). Systematic practices were applied to each category:

Design a relational network to replace implicit ID cross features.

Introduce a graph‑based pre‑training network to replace explicit statistical features.

Deploy a multi‑hash compression scheme to upgrade core ID features.

Implement a learnable feature‑selection mechanism to ensure all retained features contribute positively.

Combined with column‑dimension upgrades, sample‑column scaling, heterogeneous computing optimizations, and incremental feature iteration, the CTR model size shrank from several terabytes to dozens of gigabytes, saving hundreds of machines, cutting training time by 50%, and doubling online QPS.

2.1 Relational Network

Cross features (both heterogeneous and homogeneous) are abundant but cause model bloat. A relational network inspired by self‑attention was built, using a shared interaction matrix to symmetrically model pairwise feature interactions. This network is placed within the deep component, enabling efficient GPU computation and better representation of cross‑feature embeddings.

2.2 Graph‑Based Pre‑Training Network

To handle explicit statistical cross features that cannot be decomposed into independent embeddings, the team introduced PCF‑GNN (Pre‑trained Cross Feature Graph Neural Network). Nodes represent features, edges encode statistical interaction weights, and the model predicts edge weights to explicitly model cross‑semantic representations, achieving both performance gains and significant compression.

2.3 Multi‑Hash Compression Scheme

Core ID features were compressed using a multi‑hash embedding approach. Although each individual hash may cause collisions, the combination of multiple hashes drastically reduces the overall collision rate, approximating a collision‑free representation. Hyper‑parameters such as hash function choice, sharing, and aggregation were tuned to maintain convergence while further shrinking the model to the tens‑of‑gigabytes range.

2.4 Droprank Feature Selection

Feature selection was integrated into model training using Dropout Feature Ranking, allowing the model to learn feature importance jointly with parameters. An extended method, FSCD (Feature Selection based on Complexity and variational Dropout), incorporates system‑resource factors to balance efficiency, effectiveness, and feature selection, and has been deployed in the pre‑ranking stage.

3. Summary and Outlook

The systematic production‑level practices demonstrated that “small‑and‑beautiful” CTR models are feasible, enabling efficient iteration under constrained resources while preserving predictive power. Future work will continue to explore resource‑aware model evolution without a one‑size‑fits‑all answer.