How SORT Transforms Precision Ranking with a Transformer‑Based Architecture

Background

Industrial ranking models have been limited by fragmented deep networks that cannot fully exploit modern compute. The paper “A Systematically Optimized Ranking Transformer for Industrial‑scale Recommenders” proposes SORT, a Transformer‑based ranking model that reconstructs the underlying paradigm to scale with compute, parameters and data.

Data Reconstruction – Request‑Centric Paradigm

SORT replaces exposure‑centric sample organization with a request‑centric approach. One request aggregates N candidate items, sharing user‑side features, eliminating redundant data for billions of parameters. This enables all three feature types—user behavior sequences, user profiles, and candidate item features—to be processed jointly.

From exposure to request aggregation : consolidates samples per request, removing duplicate user features.

Multi‑type behavior fusion : adds a “behavior type” tag to align heterogeneous interactions by timestamp.

Model Reconstruction

SORT keeps the Transformer backbone but introduces several optimizations to resolve the tension between scale and efficiency.

Sparse Computation

Local Attention & Query Pruning : spatial Local Attention reduces complexity from O(L²) to O(mL) (m = window size), achieving a 6.97 % FLOPs drop while improving CTR‑AUC by 0.10 % and CTCVR‑AUC by 0.06 %.

Query Pruning : three strategies (Linear, Exp, Pooling) progressively shrink the query dimension across layers, cutting FLOPs from 43 G to 24 G and raising CTR‑AUC +0.21 % and CTCVR‑AUC +0.04 %.

Attention Mechanism Optimization

Special Tokens : learnable tokens mark feature boundaries, acting as attention sinks; experiments show CTR‑AUC +0.33 % and CTCVR‑AUC +0.20 % without extra FLOPs.

Attention Gate & QK Norm : dynamic gating and RMSNorm stabilize training, delivering an additional CTR‑AUC gain of 0.12 %.

Model Capacity Expansion

Adopting a DeepSeek‑style Mixture‑of‑Experts (MoE) FFN decouples parameter count from compute. Parameter count grows from 18 M to 83 M while FLOPs stay constant; CTR‑AUC improves +0.19 % and CTCVR‑AUC +0.06 %.

System Reconstruction – High‑Performance Training & Inference

Built on the RecIS framework, SORT introduces:

Sparse module optimization : multi‑process group communication hides I/O latency.

Dense module optimization : custom sparse attention kernels, torch.compile, mixed‑precision and gradient accumulation raise MFU from 13 % to 22 % and speed up training 2.44×.

Inference engine : torch.export + AOTInductor, KV‑Cache, FP16/BF16, multi‑stream execution cut latency 24.4 % and boost throughput 16.7 %.

Experimental Results

On four core domains, SORT delivers >5 % lifts in GMV, buyer count and order volume. Offline metrics show CTR‑AUC gains up to 0.33 % and CTCVR‑AUC up to 0.21 % across variants. Scaling‑law studies confirm steady improvements with larger data, model size, and sequence length up to 4096.

Conclusion & Outlook

SORT demonstrates that a Transformer‑native ranking architecture can break the capacity ceiling of traditional models, delivering both accuracy and efficiency gains. Future work will explore richer tokenizers, multi‑step inference, reinforcement‑learning‑based long‑term value optimization, and deeper hardware‑software co‑design.