How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture

Why Traditional MoE Hits a Structural Wall

Mixture‑of‑Experts (MoE) has become the dominant path for scaling large language models because it activates few parameters per token while keeping total parameters high. However, each token routed to an expert requires loading a weight matrix of size d × m (with hidden dimension d ≈4096‑7168) and an All‑to‑All communication proportional to d. As models grow, these memory and communication costs scale linearly with the number of activated experts, making further quality gains prohibitively expensive.

Latent MoE’s Core Idea

NVIDIA’s analysis shows that reducing d is the only way to break both latency and throughput bottlenecks, but shrinking d harms representation power. Latent MoE solves this by inserting a learnable projection W↓ that maps the token from the original space ( d) to a smaller latent space ( ℓ, e.g., 1024, a quarter of d), performing routing and expert computation in ℓ, and then projecting the expert output back with W↑. This reduces the expert weight size from d×m to ℓ×m, cutting memory reads and All‑to‑All traffic by roughly four times.

Empirical Evidence

The original LatentMoE paper (NVIDIA) reports that, with the same parameter budget, LatentMoE consistently outperforms standard MoE on all benchmark tasks—including inference, knowledge, and code—both for vanilla Transformer MoE and Hybrid Mamba‑Attention MoE backbones. To match LatentMoE’s accuracy, a conventional MoE would need an extra 350 B parameters, while LatentMoE can be up to 3.5× faster at equal accuracy. The additional linear projections add only about 9 % overhead, far cheaper than the parameter increase required for standard MoE.

Two Teams, Two Practices

NVIDIA’s Nemotron 3 Super (120 B parameters) uses 512 experts with Top‑22 routing and a latent dimension ℓ=1024 (d/4 compression). This yields nearly four times more activated experts than a standard Top‑8 MoE at the same inference cost. Their design also includes Mamba‑2 layers for efficient sequence modeling, a few Transformer attention layers as global anchors, and a shared Multi‑Token Prediction head, achieving 2.2× the throughput of GPT‑OSS‑120B and 7.5× that of Qwen3.5‑122B with comparable accuracy.

Moonshot AI’s Kimi K3 (2.8 T parameters) pushes the scale further with 896 experts and Top‑16 routing, reaching a sparsity ratio of 56:1. To keep training stable at this extreme sparsity, Kimi K3 introduces Quantile Balancing, which derives expert allocation directly from the router’s score quantiles, eliminating heuristic hyper‑parameters. This yields a 2.5× scaling efficiency over Kimi K2.

Common Logic and Future Directions

Both teams share the same underlying logic: compress the expert computation dimension, then reallocate the saved resources to a larger expert pool, expanding the combinatorial knowledge space. Open research questions include making the compression ratio adaptive to token complexity, exploring dynamic Top‑K or multi‑head routing, and leveraging the observed ~7 % zero‑gradient parameters in late‑stage training as a design signal for sparsity‑aware experts.

Combining LatentMoE with linear‑time sequence models (Mamba, KDA) promises practical deployment of “1 M‑token context + sparse activation” as a new industry default.