Kimi K3 Technical Report Reveals Answers to Key Architecture Questions

Kimi K3 Release and Positioning

On the night before the WAIC conference (July 16) Moonshot released Kimi K3, a 2.8 T parameter model that is the largest open‑weight model to date. The full model weights were made publicly available on July 27 together with a 47‑page technical report.

"Kimi K3 ... is consistently ahead of the other open and proprietary models evaluated in our suite."

The report explicitly positions K3 ahead of Claude Fable 5, GPT‑5.6 and claims it surpasses all other open‑source and closed‑source competitors.

LatentMoE: Compressing MoE Communication

LatentMoE replaces the traditional Mixture‑of‑Experts (MoE) routing with a latent‑space routing where the expert width and backbone width are separated. The routing experts operate in a 3584‑dimensional compressed latent space, half the backbone hidden dimension, mirroring the philosophy of MLA’s low‑dimensional KV compression: reduce dimensionality before computation to save communication and memory.

An analogy is given: with 896 expert positions, instead of sending the full dossier to every candidate, a one‑page summary is sent first; only those who can handle the summary receive the full material, saving bandwidth.

K3 scales this to 896 routing experts, activating only 16 per token (56 % sparsity). The report admits that such extreme sparsity introduces two concrete problems.

Activation Explosion and SiTU‑GLU

The routing path involves dimensionality reduction, gated multi‑branch feed‑forward, and expansion, requiring nearly four consecutive matrix multiplications. At 2.8 T parameters this can cause activation values to overflow.

To mitigate this, K3 uses SiTU‑GLU, which applies a soft upper bound (tanh) to both branches of the feed‑forward network. Near the origin the behavior matches SwiGLU, while larger values are clamped to a fixed range, preserving gradients unlike hard clipping.

The report specifies the bounds as β₁ = 4 and β₂ = 25, whose product equals 100, ensuring the final output never exceeds 100 regardless of intermediate magnitude.

Load Imbalance and Quantile Balancing

With 896 experts, traditional fixed‑step probability adjustments cannot keep the selection probabilities stable.

K3 introduces Quantile Balancing, which derives each expert’s bias directly from the quantiles of the current batch. The appendix provides a full derivation and proves that this update is the exact solution to the load‑balancing problem, unlike the incremental adjustments of older methods.

"Each expert’s bias is solved in one step from batch quantiles, offering an exact load‑balancing solution."

Attention Residuals vs. DeepSeek V4 mHC

DeepSeek V4 expands residual paths with manifold‑constrained Hyper‑Connections (mHC). K3 takes a different route: it keeps residual width unchanged and lets each layer use a learnable query vector to attend to all previous layers, weighting them by attention scores.

"Each layer selectively retrieves representations from all preceding layers rather than accumulating them uniformly."

This design, called Attention Residuals, is analogous to a meeting where each participant can directly consult the original remarks of any earlier speaker instead of relying on a single, ever‑growing summary.

The practical implementation is Block Attention Residuals, which splits a 93‑layer network into eight blocks of twelve layers each. Within a block, layer outputs are summed into a representative vector; cross‑block attention is then performed, reducing memory and communication costs. Empirically, N ≈ 8 recovers most of the benefit across model scales.

NoPE Full‑Stack Positional Encoding

K3 removes all RoPE encodings and uses NoPE (no explicit positional encoding) across every MLA layer. The KDA layer retains implicit positional information via token‑wise recursive gating and decay, while MLA layers treat all tokens uniformly.

"No explicit positional encoding is applied to their queries or keys."

This design simplifies length generalization: extending context length does not require adjusting positional parameters. Training progresses from 8 K tokens to 64 K, then to 256 K, and finally to 1 M tokens in four stages. The report stresses that length alone does not guarantee long‑range capability; instead, long, coherent multimodal documents are synthesized to force the model to learn to retrieve information across the entire window.

Open‑Source Release Beyond Weights

Moonshot also open‑sourced the inference kernel FlashKDA (a backend for flash‑linear‑attention), the MoE routing code MoonEP, and the agentic‑RL sandbox AgentENV, which claims a paused sandbox consumes no memory or CPU.

"A paused sandbox consumes no memory or CPU resources."

Additional releases include the MiniTriton compiler written from scratch, and a 48‑hour‑produced RTL implementation nano‑kpu for inference chips. The report argues that exposing these engineering details, such as optimal expert load‑balancing and cross‑device state synchronization, is crucial for reproducibility and reduces the engineering effort for teams attempting to replicate the architecture.

Overall, the technical report positions Kimi K3 as a model that not only matches but often exceeds contemporaries like DeepSeek and Nemotron in efficiency and architectural innovation, explaining why it has been dubbed the “DeepSeek 2.0 moment” in overseas discussions.