Addressing the “Sandglass” Bottleneck in Residual Quantization Semantic Identifiers for Generative Search and Recommendation

This article presents a study selected for EMNLP 2024 that investigates the “sandglass” bottleneck in Residual Quantization Semantic Identifiers (RQ‑SID) used in generative search/recommendation systems. The bottleneck manifests as an over‑concentration of middle‑codebook tokens, leading to path sparsity and a long‑tail distribution that degrades performance in e‑commerce scenarios.

Background : Generative search/recommendation increasingly relies on numeric identifiers (e.g., DSI, NCI, TIGER, GDR, GenRet) for efficiency and generalization. TIGER, which employs residual quantization (RQ) to generate semantic identifiers (SID), has shown strong potential in product‑centric e‑commerce but suffers from the sandglass effect.

Task Definition : Given a user profile (age, gender, membership) and historical interactions, the system must predict the most likely purchased item for a query such as “XX mouse”. This exemplifies any SID‑based task.

RQ‑VAE SID Generation : The method follows the TIGER pipeline—training a dual‑tower model on billions of query‑item logs, extracting item embeddings, and applying RQ to produce multi‑layer SIDs.

Sandglass Phenomenon : Visualizations of three‑layer codebooks reveal a massive concentration of tokens in the second layer, confirmed by low entropy, high Gini coefficient, and large standard deviation. The phenomenon originates from residual quantization’s tendency to allocate most residuals to a few middle‑layer tokens, creating sparse paths and a long‑tail token distribution.

Analysis : Experiments compare uniform vs. non‑uniform input distributions, showing that non‑uniform (long‑tail) data exacerbate the sandglass effect. Theoretical discussion links the effect to the diminishing residual magnitude across layers.

Practical Impact : Splitting test sets by head vs. tail second‑layer tokens shows significant performance gaps—head tokens yield higher accuracy, while tail tokens suffer. This gap persists across models (LLaMA2, Baichuan2, Qwen1.5) and various RQ configurations.

Solution Strategies : Two approaches are proposed: (1) heuristically remove the second layer entirely, sacrificing capacity; (2) adaptively prune top‑K tokens in the second layer (top@K strategy) to obtain a variable‑length SID while preserving overall distribution. Experiments on LLaMA demonstrate that adaptive token removal (e.g., top@400) improves metrics and reduces computational cost, outperforming the baseline and the naive layer removal.

Conclusion : The study systematically uncovers the sandglass bottleneck in RQ‑SID, validates its detrimental effect through extensive ablations, and offers effective mitigation techniques, with adaptive token pruning achieving the best results.

Future Work : Plans include optimizing SID generation with temporal and statistical features, unifying sparse (SID) and dense representations for LLMs, and achieving loss‑less end‑to‑end search pipelines.