How a Team Ran the Open‑Source Kimi K3 Model on 80 RTX 5090 GPUs

On the night of July 27 the complete Kimi K3 weights were uploaded to HuggingFace, split into 96 shards of about 17 GB each for a total of 1.56 TB, which can be downloaded in a day.

The new architecture improves scaling efficiency by roughly 2.5× over K2 through three innovations: (1) mixed attention – each block combines three KDA linear‑attention layers with one global Gated‑MLA layer; (2) Attention Residuals – each layer can retrieve representations from earlier layers instead of compressing all history into a single residual stream; (3) Stable LatentMoE – 896 routing experts, 16 activated per token, with routing performed in a halved latent space and balanced by bounded activation and quantile load‑balancing.

Training proceeds in three stages: a cold‑start SFT phase, followed by RL training that creates nine domain‑specific experts (general, Agent, coding) across three inference effort levels, and finally a multi‑teacher on‑policy distillation that returns a single model. The RL agent framework is white‑box and configurable, allowing the assembly of Claude‑Code‑like or Codex‑like harnesses to avoid over‑fitting a single toolchain. Infrastructure components such as the MoonEP expert‑parallel scheme and the AgentENV micro‑VM sandbox (51 million sandboxes created during evaluation) are open‑sourced, and the FlashKDA kernel and flash‑linear‑attention library have been contributed upstream.

Weight storage uses the native MXFP4 format; the 1.56 TB of weights for a 2.8 T‑parameter model averages only 4.5 bits per parameter because quantization‑aware training (QAT) and MXFP8 activations are applied from the SFT stage onward. Consequently, the model is 4‑bit by design, not a post‑hoc compression.

Memory consumption during inference is dominated by the KV cache rather than the weights. Of the 93 layers, only 24 use global MLA attention and require per‑token cache; the remaining 69 layers employ KDA linear attention with a fixed‑size state, dramatically reducing cache growth even for a 1 M‑token context window. The global layers also omit positional encodings (NoPE), so extending context does not require RoPE.

An early deployment on 80 RTX 5090 consumer cards (zero HBM, Ethernet‑linked) achieved the full K3 model at 20 tokens / s per stream without any tuning. The hardware cost is roughly $4.3 k per card, totaling about $346 k, plus chassis, power supplies, switches, and racks brings the total to $400‑500 k. Power draw reaches 46 kW at full load, leading to an estimated annual electricity bill of $40 k.

By contrast, the official SGLang cookbook recommends professional GPUs (H100, H200, B200, B300, AMD MI350X) in clusters of 64+ cards. An 8‑card B200 node cannot even hold the 1.56 TB weight; an 8‑card H200 node would need two such machines, each costing several million dollars, and the recommended production deployment would require four times that number of cards, pushing total cost toward $3 million.

The API pricing is $0.30 per million input‑cache‑hit tokens, $3 per miss, and $15 per output token. The $400‑500 k hardware investment could instead purchase roughly 23 billion output tokens, enough for a small team to consume 50 million output tokens per day for over a year, eliminating hardware, electricity, and maintenance concerns.

Overall, deploying a 3‑trillion‑parameter model like K3 locally is only justified for a tiny minority of teams; most users are better off using Kimi’s token‑based API plan.