How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture

In early May, Moonshot released Kimi K3 with 2.8 T parameters, and a few days later Alibaba launched Qwen 3.8‑Max‑Preview at 2.4 T. Both models claim performance just behind Claude Fable 5 and GPT‑5.6‑Sol, and both are announced as open‑source.

The first technical hurdle is that dense models tie every parameter to every token, making a 2‑T model infeasible. Mixture‑of‑Experts (MoE) breaks this coupling: only a few experts are activated per token, turning parameter count and compute into independent axes. Google’s Switch Transformers (Fedus et al., 2021) demonstrated 1.6 T parameters with MoE, while DeepMind’s scaling‑law study (Clark et al., 2022) provided the theoretical backing. DeepSeek’s 2024 DeepSeekMoE further refined expert granularity by adding shared experts for general knowledge, and a follow‑up scaling‑law paper confirmed that finer expert granularity improves performance under a fixed compute budget. Activation ratios have steadily dropped: DeepSeek‑V3 activates ~5.5 % of its 671 B parameters, Kimi K2 3.2 %, Qwen‑3‑Next selects 10 of 512 experts, and K3 activates 16 of 896 experts, allowing total parameters to grow dozens of times while per‑token compute stays almost constant.

The second obstacle is attention, which scales with context length. Modern flagship models now support 1 M token context, creating a “wall” that parameter cheapening alone cannot cross. Before sparse‑attention research (expected mainstream in late 2025), Chinese models progressed from dense Multi‑Head Attention (MHA) to Grouped‑Query Attention (GQA) and then to Memory‑Light Attention (MLA), which compresses KV caches into latent vectors, cutting cache size by 93.3 %. Two engineering routes emerged to break the wall: (1) sparse attention that limits each token’s view of the context (e.g., DeepSeek/GLM‑5.2 DSA, MiniMax M3 MSA, Xiaomi sliding‑window MFA); and (2) linear‑attention replacements. Both K3 and Qwen 3.8 follow the latter, adopting Gated Delta Networks (Yang et al., 2024) that combine linear attention, a delta‑rule memory update, and gated forgetting. Kimi’s KDA is a refined gated version of this architecture.

Training trillion‑parameter MoE models introduces two specific failure modes: routing collapse (tokens crowding a few experts) and loss spikes (unexpected training divergence). DeepSeek’s “Loss‑Free Balancing” paper (Wang et al., 2024) proposes a gradient‑free load‑balancing method that was deployed at scale on the 671 B V3 model. Optimizer stability was further improved with the Muon optimizer (Moonshot AI, 2025) and its scalable variant; K2 added MuonClip and QK‑clip to achieve zero loss spikes over 15.5 T tokens, while K3 introduced per‑head Muon tuning. Precision advances include the first FP8 training at 671 B (278.8 k H800 GPU‑hours) and, for K3, MXFP4 quantization‑aware training that halves bits per parameter. DeepSeek V4 preview also runs experts in FP4 while keeping attention in FP8, effectively doubling parameter purchasing power for the same budget.

Inference at this scale tests engineering depth. Multi‑token prediction (MTP) with speculative decoding, first scaled in V3, is now adopted by many. Because K3’s KDA does not fit traditional prefix‑cache schemes, the authors contributed a custom implementation to the vLLM community and recommend deployment on nodes with at least 64 GPUs. The 2.8 T model can therefore offer competitive token‑price performance, thanks to these engineering optimizations.

In summary, the emergence of 2 T+ Chinese LLMs is the cumulative result of four years of architectural evolution—parameter‑compute decoupling via MoE, attention scaling strategies, and robust training/inference pipelines—not a sudden miracle. Notably, Zhipu’s GLM‑5.2, with only 753 B parameters, outperforms larger rivals, suggesting that the next GLM‑5.3 iteration, if it reaches the 2 T mark, will likely surpass both K3 and Qwen 3.8‑Max.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus et al. (Google) – https://arxiv.org/abs/2101.03961

Unified Scaling Laws for Routed Language Models, Clark et al. (DeepMind) – https://arxiv.org/abs/2202.01169

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture‑of‑Experts Language Models, DeepSeek‑AI – https://arxiv.org/abs/2401.06066

Scaling Laws for Fine‑Grained Mixture of Experts, Krajewski et al. – https://arxiv.org/abs/2402.07871

DeepSeek‑V2: A Strong, Economical, and Efficient Mixture‑of‑Experts Language Model, DeepSeek‑AI – https://arxiv.org/abs/2405.04434

Auxiliary‑Loss‑Free Load Balancing Strategy for Mixture‑of‑Experts, Wang et al. (DeepSeek) – https://arxiv.org/abs/2408.15664

Gated Delta Networks: Improving Mamba2 with Delta Rule, Yang et al. – https://arxiv.org/abs/2412.06464

DeepSeek‑V3 Technical Report, DeepSeek‑AI – https://arxiv.org/abs/2412.19437

Muon is Scalable for LLM Training, Moonshot AI – https://arxiv.org/abs/2502.16982

Kimi Linear: An Expressive, Efficient Attention Architecture, Moonshot AI – https://arxiv.org/abs/2510.26692

https://www.kimi.com/blog/kimi-k3