Engineering Exploration of Generative Recall: Leveraging Large Models for Next‑Gen Recommendation Retrieval

Background

Traditional recommendation recall relies on item IDs (x2i) which have strong memorization but weak generalization, especially for cold‑start items, and vector‑based recall suffers from limited expressive power. The two‑stage "recall + ranking" pipeline also incurs precision loss due to mismatched stages.

Overall Solution

We introduce a hierarchical Semantic ID and a generative recall model called ASGR (AliExpress Semantic‑ID‑based Generative Recommendation) . User behavior sequences are encoded as token streams and a seq2seq task directly generates candidate semantic IDs, aiming to break the bottleneck of the recall stage, improve cold‑start handling, and open a new growth path for the system.

Inference Pipeline Architecture

The model layer adopts an Encoder‑Decoder architecture to ingest heterogeneous features. The recall layer replaces the traditional "recall + coarse ranking" paradigm with an independent generative recall service that runs in parallel with the existing natural recall (x2i && aux be) and merges results before ranking. This decoupling allows independent optimization of the generative path without affecting the stable natural pipeline, reduces end‑to‑end latency, and leaves headroom for larger models.

Phase I Progress

Model size grew from 4 M (avg rt 62 ms, 140 qps single‑node limit) to 1.2 B (avg rt 105 ms, 24 qps single‑node limit), a 300× parameter increase while keeping real‑time recall latency.

4.1 Superman Optimizer – Enabling SDPA

We replaced the original QKV computation (QKᵀ → scaling → mask → softmax → dropout → weighted sum) with a custom SDPA‑based attention layer. The optimizer rewrites the model’s t5‑attention sub‑module at runtime, enabling true SDPA execution in the online inference path.

4.2 Custom Beam Search ("new(牛) beam search")

Because each position only has 256 candidate tokens (total vocab = n × 256), we prune intermediate results and rank only the 256 truly valid tokens instead of the full n × 256 space. This custom beam search reduces redundant computation and improves latency by 35 % while increasing single‑node peak throughput by 40 % compared with the default torch.generate implementation.

4.3 StaticCache + Compile

When model size reached 0.5‑0.7 B, earlier optimizations no longer met latency targets. We applied torch.compile to fuse operators, achieving noticeable offline gains, but online performance suffered due to dynamic shape changes that forced fallback to CPU kernels, reducing the expected benefits.

4.4 Tree‑Based Attention

By organizing beam candidates in a Trie and applying a tree‑aware attention mask, shared prefixes store a single KV cache entry, cutting prefix‑related memory waste from ~75 % to near zero. The mask ensures tokens only attend to ancestors on their own branch, preserving independence while dramatically reducing memory usage. With beam width = 100, throughput doubled; with width = 50, gains were marginal.

4.5 Sliding‑Window Attention

Given an average input length of 600 tokens, applying a sliding window of size ω reduces computation to O(L·ω) and lowers latency by ~5 % with a 10 % throughput increase.

4.6 Quantization

8‑bit/4‑bit quantization (e.g., Absmax) compresses weights to reduce VRAM but does not guarantee speedup; in our tests it actually degraded inference performance, though it alleviated memory pressure.

4.7 ONNX + Optimum

Exporting the model to ONNX enables heterogeneous execution providers. Constant folding, redundant node elimination, and operator fusion (e.g., Conv + Add) reduce graph size. Using optimization level O4 with mixed‑precision cuts Cast nodes from 233 to 8, improving latency and memory usage.

4.8 TensorRT‑LLM Integration

TensorRT‑LLM provides packed tensor handling, MQA/GQA support, and a unified TrtLlmAttention kernel that fuses QKᵀ, softmax, and weighted sum, eliminating redundant memory traffic. For short sequences we keep standard MHA; for long sequences we switch to FlashAttention. KV cache is managed via a paged system with priority‑based LRU eviction, enabling high‑concurrency, variable‑length batching.

4.9 GPU Resource Sharing (NVIDIA MPS)

Enabling Multi‑Process Service shares a single CUDA context among inference workers, reducing context‑switch overhead. This yields a 15 % increase in single‑node peak throughput and a 10 % latency reduction for 0.2 B/0.7 B/1.2 B models, stabilizing response times under high load.

Future Outlook

The next step is to replace traditional inner‑product recall with the generative paradigm across the entire recommendation stack, unifying recall and ranking objectives to build a more efficient, intelligent next‑generation system.