PRIME: Joint Pruning & Merging for MoE Expert Compression with 2.23× Throughput Gain
This article presents PRIME, a unified integer linear programming approach that jointly decides per-expert retention, merging, or pruning based on activation frequency and output similarity, reducing routed-expert count by 25% with only ~1% accuracy drop on code/math benchmarks, and demonstrates up to 2.23× throughput improvement in high-concurrency long-output serving via working-set reduction and KV-capacity expansion.
The presentation addresses a fundamental mismatch in Mixture-of-Experts (MoE) deployment: while top-k routing sparsifies computation per token, all routed experts must remain GPU-resident, consuming memory that could otherwise serve KV cache and concurrency. Using Qwen3-30B-A3B (128 routed experts, top-8 activation) as a case study, the authors show that expert weights occupy 56.9 GiB on an 80 GB GPU, leaving only 11.2 GiB for KV cache (~122K tokens).
Redundancy Analysis
Calibration on C4 data reveals two redundancy types across 48 layers:
Low-frequency redundancy : ~24.5% of experts have near-zero activation frequency, making them candidates for direct pruning.
Similarity redundancy : Experts form high-similarity groups; Layer 0's largest group contains 40 experts with pairwise similarity >0.5, suggesting merging preserves information better than deletion.
PRIME: Unified ILP Decision
Existing methods either prune-only (lose information from similar experts) or merge-only (force outliers into clusters, causing representation pollution). PRIME formulates a single integer linear program (ILP) where each expert chooses among retain, merge, or prune. The ILP minimizes total cost: merge cost ≈ frequency × distance, prune cost ≈ frequency × λ. This yields three natural behaviors:
High-frequency experts become aggregation centers (high cost to misplace).
Low-frequency, close-to-center experts merge via frequency-weighted averaging.
Low-frequency, distant experts are pruned when merge cost exceeds prune penalty.
The solver outputs a single group_labels array per layer: non-negative labels indicate merge groups, -1 indicates pruning. This decouples the decision logic from tensor rewriting.
Router Reconstruction
Merging experts breaks the linear router: grouped logits require log-sum-exp over original logits, which is not a linear map. PRIME performs Router Reconstruction during calibration: using collected hidden states and grouped logits, it solves a ridge-regression closed-form to fit a standard K×hidden linear router, preserving a vanilla K-way routing at inference with no backward pass or optimizer.
Layer-Wise Adaptivity
At 25% compression (K=96), layers 6–11 favor pruning while layers 24–35 favor merging; overall split is 73% merge / 27% prune. Ablation confirms both joint decision and frequency weighting contribute independently.
Model Quality
On Qwen3-30B-A3B and Qwen3.5-122B-A10B, 25% expert-count reduction incurs ~1 percentage-point absolute drop on average code/math scores; 50% reduction incurs ~3 points. Compression ratio is defined as 1 - K/N (per-layer routed-expert reduction), not equivalent to parameter, memory, or latency reduction.
Serving Gains: Two Channels
Expert compression does not reduce per-token active FFN FLOPs ( num_experts_per_tok stays 8) nor per-token KV size. Gains come from:
Working-set / memory-access reduction : fewer unique experts touched per decode step at higher concurrency.
KV-capacity expansion : freed GPU memory converts to larger KV pool, enabling longer contexts and higher concurrency.
Quantitative Deployment Results (TP1, A800-80GB, bf16, SGLang)
Expert weights: 56.9 → 29.9 GiB (−27 GiB).
KV capacity: 122K → 417K tokens (3.42×).
Concurrent 16K-context sequences: 7 → 25.
Long-output (512 in / 4096 out), concurrency 128: peak throughput 2.23×. Attribution via free-KV / fixed-KV / TP2 experiments: working-set channel ≈1.32×, KV-capacity channel ≈1.68×, product ≈2.22× (matches 2.23×).
No gain at concurrency 1; working-set gains emerge at C≥4; long-output high-concurrency workloads benefit most.
Measurement Caveats
When the baseline is capacity-constrained, observed differences conflate capacity limits, parallelism changes, and communication overhead. Proper evaluation must report model, hardware, TP/EP, sequence lengths, concurrency, and KV settings, distinguishing throughput-under-same-runnable-conditions from new capacity-enabled concurrency.
moe-compressor: Production Toolchain
To move beyond experiments, the team built moe-compressor, a standardized pipeline with decoupled control/execution planes and three fixed interfaces: CompressionRequest: unified frozen request from CLI, recipe, or Python API. BackendPlan: compiler-style execution plan; algorithms emit decisions, not tensor ops. group_labels: sole intermediate representation (non-negative = group, -1 = prune).
Control plane (config parsing, validation, ILP solve) runs without GPU; execution plane handles calibration forward, statistics collection, tensor rewrite, and saving. New algorithms implement a modifier; new model families implement ModelAdapter, model_ops, and grouper; calibration datasets and eval suites register via schema. Usage example:
{ "model": { "name": "Qwen3-30B-A3B", "path": "..." }, "output_dir": ".../qwen3-30b-a3b-prime-r025", "modifier": { "name": "prime", "compress_ratio": 0.25, "prune_penalty": 0.25 }, "calibration": { "name": "C4", "n_samples": 32 } }Command:
moe-compressor compress --recipe examples/recipes/qwen3-30b-a3b-prime.json --compressed-model-path ./out/qwen3-prime-r025. Output is a standard Hugging Face checkpoint ( config.json with updated num_experts, safetensors shards, tokenizer), directly loadable via from_pretrained and servable with SGLang.
Three Takeaways
Sparse compute ≠ sparse memory : top-k reduces compute but not expert-weight residency; reducing routed-expert count is necessary.
PRIME jointly decides retain/merge/prune : adapts to per-layer redundancy structure; 25% reduction largely preserves code/math ability.
Serving gain is workload-dependent : gains stem from working-set and KV-capacity changes, varying with I/O lengths, concurrency, and deployment config.
Recommended adoption path: profile expert activation frequency and similarity on target workload, start with mild compression ratios, validate quality, then measure working-set, KV capacity, and end-to-end throughput separately rather than relying on a single speedup number.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
