Deploying Kimi K3 Locally: Why You Need Up to $30 Million in Infrastructure

Model size and quantization

K3 has 2.8 T parameters. Weights are stored in MXFP4 format (4‑bit with block‑level scaling) and activations in MXFP8, resulting in ~0.53 bytes per parameter, ≈1.5 TB. The released weight files consist of 96 safetensors shards, total 1.56 TB.

2.8T 参数 × 0.53 字节 ≈ 1.5TB

Memory requirements

Inference needs additional memory for runtime, activations, KV cache and concurrency. Using the vLLM budget factor of 1.18 (derived from GLM‑5.2), the minimum GPU memory is: 1.56TB × 1.18 ≈ 1.84TB KDA linear attention reduces KV‑cache pressure; a KDA prefix‑cache implementation has been contributed to vLLM.

Deployment feasibility

An 8‑GPU NVIDIA H200 server provides 1128 GB VRAM, insufficient to hold the required 1.84 TB, so single‑node deployment is impossible.

Multi‑node inference uses tensor and expert parallelism. GPU counts are chosen as powers of two (16, 32, 64) to match common sharding strategies.

Cost configurations

Two‑node solution (2 × 8 × H200, 16 GPUs, 2256 GB VRAM). After loading weights, ~700 GB remains for KV cache and runtime. Suitable for 128 KB context with careful concurrency. Estimated hardware cost ≈ 7.3 M–8.0 M CNY; high‑speed network (200 Gbps RDMA, optics, cables) adds ≈ 0.3 M–1.0 M CNY. Total ≈ 7.3 M–8.0 M CNY.

Four‑node solution (4 × 8 × H200, 32 GPUs, 4512 GB VRAM). Provides ~3 TB free memory for long context and moderate concurrency. Estimated total cost ≈ 14.5 M–15.5 M CNY.

Eight‑node solution (8 × 8 × H200, 64 GPUs). Officially recommended super‑node. Estimated total cost ≈ 29 M–30 M CNY (hardware ≈ 28 M–30 M CNY, network ≈ 1 M–2 M CNY).

Power and infrastructure

Each 8‑GPU H200 server consumes about 10.2 kW; eight servers exceed 80 kW, requiring upgraded racks, PDUs and cooling, though these are minor compared to hardware acquisition.

Performance benchmarks

On Terminal Bench 2.1, K3 scores 88.3, comparable to GPT‑5.6 Sol. It leads Program Bench and SWE Marathon, surpassing GLM‑5.2. In agent tasks, K3 achieves 91.2 on BrowseComp, ranking first among open‑source models.