Skip to content
Tencent's LatentRec Shifts LLM Reasoning to Training for 18x Faster Recommendations

Tencent's LatentRec Shifts LLM Reasoning to Training for 18x Faster Recommendations

Tencent's LatentRec framework moves chain-of-thought reasoning from inference to training in generative recommendation, achieving 18.3x latency reduction and 477.8% cold-start Hit@1 improvement in production system TGR-Reason, accepted at CIKM 2026.

PaperAgent
PaperAgent
PaperAgent

Background: The "Thinking Tax" in LLM-Based Recommendation

Generative recommendation using LLMs (SID-based) encodes items into hierarchical Semantic IDs (e.g., "12-45-7-233") and generates them autoregressively. To improve quality, Chain-of-Thought (CoT) distillation was introduced: a large model writes reasoning traces for each user history, which are then distilled into a smaller model. At inference, the model first outputs dozens to hundreds of reasoning tokens before the SID. This adds significant latency and compute cost — unacceptable for online recommendation where users won't wait.

LatentRec: Three Components to Move Reasoning to Training

LatentRec asks: can training-time supervision replace inference-time reasoning? The framework adds three lightweight components to a frozen LLM backbone (Qwen3-1.7B) with LoRA.

1. Continuous Thinking Module (CTM)

CTM adds only 2.5M parameters (<0.2% of 1.7B LLM). During training, it performs K iterative latent reasoning steps on the LLM's hidden states. Each step passes the current hidden state through the Soft Token Selector (STS) to produce a new input embedding, which is fed back into the same LLM. This acts as a "small brain" that helps the model think deeper during training.

2. Soft Token Selector (STS)

STS solves a critical scale mismatch: on Qwen3-1.7B, hidden states and input embeddings differ by ~14x in magnitude. Directly feeding raw hidden states back caused training divergence in the first epoch (triggering auto-stop). STS maintains 64 learnable soft tokens initialized from the model's own vocabulary embeddings. At each step, it computes attention weights over these tokens based on the current hidden state and produces a weighted combination as the next input. This re-expresses the state in the model's familiar embedding space, stabilizing the recurrent loop.

3. Progressive Reasoning Loss (PRL)

PRL supervises every latent step. Each intermediate hidden state passes through the shared LM head to predict the first token of the target SID (the coarsest routing signal), using cross-entropy loss. Default training uses two steps: step 1 with temperature 3.5 (looser), step 2 with temperature 1.0 (tighter). Only the first SID token is supervised because deeper tokens would require teacher-forcing the prefix, locking the latent rollout trajectory and conflicting with autoregressive decoding at inference.

Key insight: the latent modules (CTM+STS) are discarded at deployment (K=0). However, the step-wise supervision updates the LLM's LoRA parameters during training, so the reasoning benefit remains in the model weights without extra inference cost.

Experimental Results: Faster and More Accurate

Evaluated on Amazon Beauty, Toys, Sports datasets against same-backbone CoT SFT baseline.

Recommendation Quality (HR@10 at K=0)

Beauty: 0.0625 → 0.0656 (+5.0%)

Toys: 0.0639 → 0.0668 (+4.5%)

Sports: 0.0297 → 0.0309 (+4.0%)

Inference Latency (Beauty, H20 GPU, batch=1)

CoT SFT generating 128 reasoning tokens: 10.80 seconds per sample. LatentRec K=0: 0.59 seconds — 18.3x speedup while improving HR@10.

Ablation Study (Beauty, from CoT SFT baseline 0.0625)

Continue LoRA only: 0.0629 (+0.64%)

+ Latent module without PRL: 0.0633 (+0.64%)

+ PRL: 0.0656 (+3.63%)

PRL is the dominant contributor. Additional inference-time latent steps (K>0) gave inconsistent gains: Beauty +0.6%, Toys -0.6%, Sports -0.6%. Training depth also matters: K=0 with 2 training steps outperformed 16 steps. Conclusion: multi-step latent computation's value lies primarily in training, not inference.

Production Deployment: TGR-Reason

LatentRec is integrated into Tencent's TGR-Reason, part of the TGR generative recommendation stack deployed on TRS platform (https://trs.tencent.com) across 10+ scenarios (news, video, web novels).

Engineering Trick: Pre-computed "Think" Tokens

Since K=0 works, TGR-Reason pushes pre-computation further. A "Think" model (trained with LatentRec) runs offline to generate candidate SIDs for each user, stored as "reason tokens" in a database. At online request time, the system reads recent tokens and injects them as auxiliary signals into the generator — no on-the-fly LLM reasoning needed. Asynchronous background updates keep tokens fresh; cache misses silently fall back to the original path.

Online Metrics (vs. production baseline TGR-GenRec)

Cold-start users offline Hit@1: 4.51% → 26.06% ( +477.8% relative )

Online A/B test: effective consumption rate +1.75%, new user exposure conversion rate +13.09% (both statistically significant)

These gains translate to substantial business value at Tencent's scale.

Conclusion

LatentRec demonstrates that recommendation systems don't need textual reasoning at inference. By shifting the heavy thinking to training via short latent steps and progressive supervision, the model internalizes the reasoning ability. Deployment then skips the extra steps, retaining accuracy while achieving massive speedups. The principle — "think more during training, think less during serving" — applies broadly to AI systems.

Paper: https://arxiv.org/abs/2611.07866v1 (CIKM 2026)<br/>TGR technical report:

https://arxiv.org/abs/2609.00986
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

LLMrecommendation systemsChain-of-ThoughtGenerative RecommendationCIKM 2026LatentRecTGR-Reasontraining-time reasoning
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.