Accelerating Xiaohongshu ‘Ask’ Inference: Slim Vision Tokens, Focused MoE

1.1 Multimodal Large Model Inference Flow

"Ask" is Xiaohongshu’s intelligent Q&A capability that extracts key information from multiple text‑image notes and generates concise answers. Inference consists of a Prefill stage that encodes all textual and visual inputs once, and a Decode stage that generates tokens sequentially. High‑resolution, multi‑image inputs cause large visual token sequences, stressing both first‑token latency (TTFT) and continuous generation speed.

1.2 PD‑Separated Architecture and Core Metrics

TTFT is mainly affected by input length and Prefill computation.

TPOT (time per output token) is mainly affected by Decode‑stage memory access and MoE expert computation.

2.1 Vision Token Redundancy

After the Vision Encoder and Merger, each high‑resolution image can be represented by thousands of Vision Tokens. In multi‑image scenarios the visual sequence length becomes the primary bottleneck for TTFT.

Deep ViT features already aggregate most visual information into a small set of dominant tokens, allowing redundant tokens to be pruned without harming the model’s understanding.

Experiments show that OCR, document, and table images contain denser information and therefore require a larger token budget, while simpler images can be compressed more aggressively.

2.2 MoE Expert Activation Redundancy

MoE sparsity is defined per token, but online services run in batch mode. As batch size grows, the union of experts activated across tokens expands dramatically (e.g., in Qwen3‑30B‑A3B, average activated experts rise from 8 at batch‑size 1 to 93.4 at batch‑size 128).

Decode is memory‑bound; more activated experts increase weight‑movement, causing TPOT to rise sharply.

The core conflict is the batch‑wise expert set expansion, not the per‑token Top‑K. Directly reducing Top‑K harms model capability, so a similarity‑based dynamic re‑routing is required.

3.1 VisionZip Algorithm Principle

Observation: a few dominant tokens already contain the majority of visual information. VisionZip uses attention scores to keep high‑value tokens and prune redundant ones before the LLM, shortening the Prefill sequence.

In multi‑image cases, tokens from all images compete in a shared importance space; images with higher information density (OCR, documents, tables) automatically receive a larger token budget.

3.2 Engineering Deployment

VisionZip was integrated into the vLLM inference framework for Qwen2.5‑VL, Qwen3‑VL and Qwen3.5. Attention statistics are extracted from a designated ViT layer; each token’s average attention serves as its importance score. All images in a request share a global token budget, and the allocation is performed jointly.

A custom Triton kernel reuses FlashAttention’s Softmax LSE to compute local attention probabilities and importance scores without materialising the full attention matrix, drastically reducing intermediate tensors and memory traffic.

The multimodal sequence handling and mRoPE position‑encoding logic were adapted so that the compressed visual sequence can be fed seamlessly into the existing Prefill, KV‑Cache and Decode pipelines.

3.3 Effect and Benefit

On Qwen3‑VL‑32B with a 30‑image sample, pruning 40 % of vision tokens reduces TTFT from 687 ms to 601 ms (‑13 %) while business‑level quality metrics remain essentially unchanged.

On the RAG‑IGBench multi‑image benchmark, retaining 70 % and 60 % of vision tokens yields Integrated Scores of 38.20 % and 37.48 % respectively, comparable to the baseline 37.61 %. Retaining only 50 % drops the score to 29.06 %, indicating that aggressive compression harms fine‑grained cross‑image alignment.

4.1 SERE (Similarity‑Based Expert Re‑routing) Principle

SERE keeps model weights unchanged and requires no retraining. Offline, a small generic dataset is used to compute layer‑wise expert similarity matrices (Frobenius norm).

During Decode, each batch first aggregates the Top‑K′ primary experts across all tokens. Secondary experts are then re‑routed to the most similar primary expert if the similarity exceeds a threshold; protected key experts remain on their original path.

Router weights are unchanged, and Prefill continues to use the original Top‑K routing.

4.2 Engineering Deployment

Offline calibration collects expert activations on generic text and builds per‑layer similarity matrices; this step runs once and does not depend on downstream task labels.

Online, a high‑performance CUDA kernel merges primary‑expert summarisation, similarity search, and protection‑bit checks, feeding directly into the fused MoE operator and avoiding Python‑CUDA round‑trips.

Key‑expert protection uses a per‑layer 0/1 matrix to mark irreplaceable experts; the kernel blocks routing changes for these experts while allowing others to be re‑routed.

4.3 Effect and Benefit

On real‑world multi‑image load (1000 requests at identical QPS), TPOT drops from 8.5 ms to 7.1 ms (‑16.5 %).

Internal business evaluation shows all aggregated metrics (hallucination, comprehension, safety, formatting, structured generation) remain on par with the baseline.

Public benchmark on Qwen3‑30B‑A3B: baseline top‑8 accuracy 82.24 %; SERE K=2 (ρ=0.5) achieves 80.37 % (97.7 % of original accuracy) while TPOT falls from 44.40 ms to 32.82 ms (‑26.1 %). Fixed Top‑2 would collapse accuracy to 14.30 %.

Kernel‑level optimisation fuses the multi‑step routing pipeline into a single CUDA kernel, yielding ~4.1× speed‑up for the routing operator; the end‑to‑end TPOT gain remains ≈16.5 %.

5.1 Unified Evaluation Metrics

Text: Rouge‑1, Edit Distance

Narrative order: Kendall Score

Image‑text consistency: Alignment Score, CLIP Score

Overall: Integrated Score

5.2 Vision Token Compression Boundary

High compression harms detail‑sensitive tasks such as InfoVQA, complex OCR, and structured table generation, where fine‑grained text and numbers are lost.

General perception remains stable; a default 30‑40 % compression is safe, but the compression rate should be lowered for tasks requiring dense visual detail.

5.3 Bad‑Case Repair for Expert Re‑routing

In structured table generation, SERE K=2 caused minor regressions in later Decode layers. Adding a per‑layer key‑expert protection matrix (16 protected experts per layer) restored the Integrated Score from 80.5 % to 83.9 % (baseline 84.9 %) with only ~0.1 ms extra TPOT.

6.1 Stage Benefits

Vision token compression reduces TTFT by 13 % on a 30‑image sample.

Dynamic expert re‑routing cuts TPOT by 26.1 % on public benchmarks and 16.5 % in production.

All quality dimensions (hallucination, comprehension, safety, formatting, structured generation) stay comparable to the baseline; key‑expert protection adds negligible latency.

6.2 Future Directions

Dynamic selection of vision token compression ratio based on input characteristics.

Dynamic adjustment of expert re‑routing strength during generation.

Joint optimisation of Prefill, KV‑Cache transfer, and Decode.

Extending the techniques to more multimodal models and business scenarios.